Oh my Guava! We are moving to Caffeine.

Caching is extremely important! It provides fast response time, enabling effortless performance improvements in certain use cases.
At Outbrain, we have recently moved to Caffeine caching, after having used Guava in-memory caching for many years.

Background

Caffeine library is a rewrite of Guava’s cache that uses a Guava-inspired API that returns CompletableFutures, allowing asynchronous automatic loading of entries into a cache. The library was written by Ben Manes who is the author of ConcurrentLinkedHashMap on which Guava cache is based.

Guava OUT

1. Guava blocks during loading when a key is not present in the cache.
   We wanted to change the API to work asynchronously, but the added complexity made the code difficult to understand and troubleshoot.
   To make Guava non-blocking, we overrode the load and loadAll methods of CacheLoader to make the API return a future.
2. Since we changed the API to return a future, we encountered a problem. Guava is unaware that we use it to store futures, so it stores futures completed with an exception as well. To prevent the exceptions from being stored in the cache, we needed to add more complexity to our code.

Caffeine IN

1. Caffeine is a rewrite of Guava’s cache that uses an API that returns CompletableFutures out of the box, allowing asynchronous automatic loading of entries into a cache.
2. Caffeine removes futures that complete with an exception from the cache.
3. Caffeine uses both a Least Recently Used (LRU) eviction policy and a frequency-based admission policy relying on CountMin sketch. It has a better hit rate than LRU for many workloads.

Guava’s hit-rate benchmark vs Caffein’s

We analyzed service behavior with real traffic under different cache configurations to get an idea of how production services will behave.

The benchmarked cache contains image URLs keyed by UUID and follow an access pattern of most-frequent / least-frequent data, which is our most common use case at Outbrain.

In the benchmarks below, we did not measure and therefore have no interpretation of memory usage. But analyzing hit rates leads to some interesting insights.

1. Cache size: 10k items
Expiration after write: 5 min

Hit rate with 10k items:

Caffeine 28.33 %
Guava 20.95%

2. Cache size: 50k items
Expiration after write: 20 min

Hit rate with 50k items:

Caffeine 56.04 %
Guava 50.01%

3. Cache size: 100k items
Expiration after write: 20 min

Hit rate with 100k items:

Caffeine 70.10 %
Guava 66.77%

4. Cache size: 300k items
Expiration after write: 20 min

Hit rate with 300k items:

Caffeine 87.19 %
Guava 84.85%

Conclusion

We did it! With minor changes to infrastructure code (as Caffeine and Guava API are almost identical), we improved our hit rate and reduced code complexity for critical services in our system.

Honestly, Caffeine smells better than Guava.