ClassLoader leak (Metaspace OOM) caused by static ThreadLocals in DoubleFormat

[QiuYucheng2003]

Describe the bug
I identified a ClassLoader leak issue in io.micrometer.core.instrument.util.DoubleFormat.

The class uses private static final ThreadLocal fields (DECIMAL_OR_NAN, WHOLE_OR_DECIMAL, DECIMAL) to cache NumberFormat instances for performance. However, there is no mechanism to invoke remove() on these ThreadLocals.

In container environments (like Tomcat/Jetty) where threads are pooled and long-lived:

1. The Container Thread holds a reference to the ThreadLocalMap.

2. The Map Entry holds the NumberFormat instance (Value).

3. The NumberFormat instance holds a reference to its Class/ClassLoader (WebAppClassLoader).

4. Since the ThreadLocal is static and never removed, the ClassLoader cannot be garbage collected upon application reload/undeploy.

5. This leads to java.lang.OutOfMemoryError: Metaspace after multiple redeployments.

Environment
Micrometer version: (Based on source analysis of DoubleFormat.java)

Micrometer registry: Any

OS: Any

Java version: Any

Container: Apache Tomcat / Jetty / Spring Boot DevTools (Hot Reload environments)

To Reproduce

1. Deploy a web application using Micrometer to an external Tomcat container (or use Spring Boot with DevTools).

2. Trigger metrics collection that utilizes DoubleFormat (initializing the ThreadLocals on container threads).

3. Undeploy and redeploy the application multiple times.

4. Observation: Monitor the Metaspace usage. Take a Heap Dump and analyze ClassLoader instances. You will see multiple unreachable WebAppClassLoader instances held by ThreadLocalMap$Entry referencing DoubleFormat.

Expected behavior
Utility classes in a library should avoid using static ThreadLocal without a cleanup hook, as it breaks the lifecycle of the ClassLoader in containerized environments. Consider checking if the performance cost of instantiating DecimalFormat on demand is acceptable on modern JDKs, or implement a proper cleanup strategy.

Additional context
Source Location:

// DoubleFormat.java
private static final ThreadLocal<NumberFormat> DECIMAL_OR_NAN = ThreadLocal.withInitial(() -> ...);
private static final ThreadLocal<DecimalFormat> WHOLE_OR_DECIMAL = ThreadLocal.withInitial(() -> ...);
private static final ThreadLocal<DecimalFormat> DECIMAL = ThreadLocal.withInitial(() -> ...);
// remove() is never called.

[shakuzen]

Thank you for the report and all the details. I'm surprised this has never been reported before. DoubleFormat has been like this since Micrometer 1.0.0.

You will see multiple unreachable WebAppClassLoader instances held by ThreadLocalMap$Entry referencing DoubleFormat.

Why is it multiple class loader instances? From the description, I would have expected the class loader for the first application that uses Micrometer to be referenced, but why are other applications' class loader being referenced? Is there a shared thread pool between the different applications and not all threads in the pool are being used by each application so different threads end up with references to different application class loaders?

[QiuYucheng2003]

Yes, your guess is exactly right!

In servlet containers (like Tomcat), worker threads belong to a global shared pool and outlive the applications. These long-lived threads handle requests for multiple different apps. When applications are redeployed or hot-reloaded, their old WebAppClassLoader is discarded by the container, but it remains strongly referenced in the shared threads' ThreadLocalMap, leading to multiple leaked instances over time.

[shakuzen]

Does each app have its own MeterRegistry instance(s)? That is, none are shared between apps?

[QiuYucheng2003]

Yes, exactly. Each app has its own MeterRegistry and they are not shared.

More importantly, the Micrometer library itself is usually bundled within each application (e.g., in WEB-INF/lib). This means each app uses its own WebAppClassLoader to load the DoubleFormat class.

Because Tomcat worker threads are globally shared and outlive the applications, their ThreadLocalMaps end up holding references to the ThreadLocal instances from multiple different WebAppClassLoaders over time (especially after redeployments or hot-reloads). This is why you see multiple leaked ClassLoader instances.

[shakuzen]

Thanks for the details. I have some ideas for solutions but all of them have trade-offs. We'll have to decide which direction to go on this.