LoadingCache no longer respects maximumSize in 2.6.1

[panghy]

This seems to be pretty serious but we've noticed that since upgrading to 2.6.1 from 2.6.0 that the maximum size of a cache is no longer respected (for a synchronous loading cache). Attached is a graph of us reporting stats from a cache that has a maximum size of 100k elements.

As the graph shows, before 2.6.1 is deployed the cached stayed at a max size of 100k, after 2.6.1, it got into the millions (almost 200M). The size is simply cache.estimatedSize().

The code is as vanilla as it gets:

 Caffeine.newBuilder().
          maximumSize(100000).
          build(key -> {...});

[panghy]

Trying to inspect the commits since 2.6.0 to see what could have been the cause.

[ben-manes]

I don't recall changing anything in the eviction logic and the EvictionTest#evict passes. Can we get a test case showing it failing? Note that by default eviction is asynchronous so a test either needs to call cleanUp(), use executor(Runnable::run), or wait (e.g. using Awaitility).

[panghy]

This is actually in production so it seems odd that it stopped evicting entries after days.

[panghy]

But anyhow, looking into a test case.

[ben-manes]

yeah, even if ForkJoinPool is too busy to do our work, the write buffer should force a clean-up due to back pressure. The maintenance work replays the writes onto the policy and evicts, but the write buffer has a maximum size. When full, it will block on the eviction lock as a fallback to a thrashing or otherwise bad system state. Since there were many writes and your threads are not all blocked, its not obvious.

Can you get a thread dump just to check?

[panghy]

I can see commonPool workers awaiting work unfortunately.

[ben-manes]

I do know many people were burned by the Linux futex bug, which broke locking everywhere. They of course decided not to inform anyone, it went out into many distributions, and has been a quiet nightmare. If you are on an older kernel then you could be effected.

[panghy]

We are on 4.4.0-1050-aws (ubuntu), not sure if we upgraded together with the 2.6.1 rollout (also java 1.8.0_161)

[ben-manes]

• Linux lost wake-up bug
• Gil Tene's warning

Ubuntu 16.04 should be fine, but earlier were broken. What does the AWS version map to?

[panghy]

We might have also switched instance types as part of the roll-out so there are some additional (m4 -> m5) which is a new AWS base image, trying to generate a small test case to just test on the base metal.

[ben-manes]

According to Gil, the issue started being observed on Haswell-based processors. So if you upgraded to that class, you should also get onto on a newish kernel. I am only making wild guesses based on the limited information we have so far, though.

[panghy]

Just rolled a very simple test that triggers loading from a cache of max size 100k in a loop for a million times and printing out estimateSize(). The works both on my mac and on the production machine that's seeing the issue. :(

[panghy]

Just for some reference, these are the top jmap entries:

   1:      16979066      950827696  [The object that's cached, it's both present as a key and in values]
   2:       7559480      665234240  java.util.concurrent.ConcurrentHashMap$TreeNode
   3:       3824275      438191872  [Ljava.lang.Object;
   4:       7770486      435147216  com.github.benmanes.caffeine.cache.PSMS

[panghy]

We found the issue, this happens if the key is not stable (it's hashCode changes and its equality). Sorry for the false alarm but it's interesting to note that I guess the behavior for a size limited cache would be as such if keys are changed underneath the hood.