Redis connection leak

[peterkrol]

Describe the bug

We have noticed what seems to be a Redis connection 'leak' of some sort with the quarkus-redis-cache extension. There seems to be a scenario where the valueLoader.apply(key) never completes and therefore the .onTermination().call(con::close) in RedisCacheImpl.java:472 is never executed. This leads to connections being occupied and never returned to the pool. The image below showcases the active connections of one of our applications. As you can see, all the different instances have an ever increasing amount of connections either untill the instance is powered down because of Redis 'health' issue (at 70 connections) or an decrease in demand.

I have also configured the pool-cleaner-interval and pool-recycle-interval but these don't seem to help. I think the invocations are waiting indefinitely and are therefore not eligible to be cleaned or recycled.

Expected behavior

Connections should be recycled, no matter what happens to the invocation.

Actual behavior

Connections are not closed because .onTermination().call(con::close) in RedisCacheImpl.java:472 is never executed. This results in the connection pool slowly filling up and eventually not being able to acquire connections.

How to Reproduce?

I have created a small reproducer: https://github.com/peterkrol/quarkus-cache-connection-reproducer

Once running, call the GET /cache-test endpoint and check out the /q/health page. The request should complete after 10 seconds and the number of currently active connections should be 3. This reproducer makes little sense, but it does highlight the issue.

Output of uname -a or ver

Darwin OMD243828 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:53:27 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T6041 arm64

Output of java -version

openjdk version "21.0.6" 2025-01-21

Quarkus version or git rev

3.23.2

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.2 (c9616018c7a021c1c39be70fb2843d6f5f9b8a1c)

Additional information

No response

[quarkus-bot]

/cc @Ladicek (redis), @cescoffier (redis), @machi1990 (redis)

[cescoffier]

I had a quick look, and something that puzzles me is that Mutiny does not cancel when a timeout occurs.

@jponge Shouldn't we cancel the Uni subscription when .await().atMost(xxx) does not emit an item in time?

[jponge]

@cescoffier I'm going to have a look at the logic here, but indeed a cancellation on timeout is a good assumption

[jponge]

Confirmed, working on a fix

[jponge]

See the upstream PR in Mutiny.

@peterkrol you might want to check with a local build of Mutiny on that branch, and bump to Mutiny 999-SNAPSHOT in your project. This does fix a bug in Mutiny, but it's worth checking if this also adresses this Quarkus issue.

[peterkrol]

@jponge - Thanks! Going to check out the snapshot version

[jponge]

@peterkrol it's been merged in Mutiny main, so you should do a checkout + mvn install locally. We don't publish snapshots.

[peterkrol]

@jponge - I've pushed changes to the reproducer. An exclusion of mutiny from quarkus-cache and the snapshot dependency of mutiny. The behavior does seem different, now I have 1 or 2 active connections whenever the request completes (and a empty pool queue). So it seems to be better, but still some invocations are not being completed I think.

I will do some debugging, could you check out the reproducer and verify my findings are consistent with yours?

Edit:

Perhaps this difference in behavior is not because of the change, but because of 'debug' vs 'run' mode. When I test the current reproducer in 'run' mode, it retains 1 or 2 connections. Whenever I run in 'debug' mode, the same behavior as v2.9.0 is shown, 3 connections.

Edit 2:

Reproducer with mutiny v2.9.0 retains 3 connections in both 'debug' and 'run' mode, so the change definitely did something

[cescoffier]

The change is necessary. However, there is another thing. You are nesting 2 cache result using Redis. Unfortunately, this is not working at the moment, because the second access is going to reuse the first connection (I can't remember exactly what was going on).

So with the change in Mutiny and removing the second @CacheResult (even if you extend the timeout) you got the proper behavior.

I'm still looking at the phenomena when you have both cache result.