The recent ~100 MiB step in benchmark-list apiserver memory (perfdash) turned out to be benign GC headroom, but proving that took a manual dig into Prometheus snapshots because the dashboard only carries one number. Filing this to improve what we monitor so the next step is self-diagnosing.
What happened
Memory stepped 2685 to 2785 MiB and stayed there. It bisects cleanly to kubernetes/kubernetes#139617 (JSON list encoder), which reuses one buffer instead of allocating per item, about 600x less garbage per list. The RSS went up because of that: less allocation means GC fires ~6x less often, so under GOGC=100 the heap sits higher between collections.
Runs: good (alpha.1.110) to bad (alpha.1.127).
The live heap never moved, which is the whole point and is invisible on the current dashboard:
| 8-min avg, good vs bad |
good |
bad |
| allocation rate |
612 MB/s |
105 MB/s |
| GC frequency |
28.5/min |
4.9/min |
| heap_inuse (live) |
1785 MB |
1778 MB |
| working_set (what we chart) |
up ~100 MiB |
|
The gap
For this job perfdash carries a single apiserver memory series, container_memory_working_set_bytes from ResourceUsageSummary, which is essentially RSS. working_set moves with GC pacing, not just real memory use, so on its own it cannot distinguish:
- a real leak or live-heap growth (the thing we want to catch), from
- GC headroom shifting because allocation rate changed (this episode, benign).
It fails the other way too: a real leak can hide under a flat working_set if GC compensates. So every allocation-pattern change risks a false alarm, and real regressions can be masked.
Ideas
We can surface additional metrics that are already scraped:
- live heap (
go_memstats_heap_inuse_bytes), flat here
- GC rate (
rate(go_gc_duration_seconds_count[5m])), dropped 6x
- allocation rate (
rate(go_memstats_alloc_bytes_total[5m])), dropped 6x
Sidenote: the proto variant dashboard also shows a "regression" from an allocation improvement (kubernetes/kubernetes#139350), so this is a recurring pattern of allocation improvements reading as memory regressions on the dashboard.
The recent ~100 MiB step in benchmark-list apiserver memory (perfdash) turned out to be benign GC headroom, but proving that took a manual dig into Prometheus snapshots because the dashboard only carries one number. Filing this to improve what we monitor so the next step is self-diagnosing.
What happened
Memory stepped 2685 to 2785 MiB and stayed there. It bisects cleanly to kubernetes/kubernetes#139617 (JSON list encoder), which reuses one buffer instead of allocating per item, about 600x less garbage per list. The RSS went up because of that: less allocation means GC fires ~6x less often, so under GOGC=100 the heap sits higher between collections.
Runs: good (alpha.1.110) to bad (alpha.1.127).
The live heap never moved, which is the whole point and is invisible on the current dashboard:
The gap
For this job perfdash carries a single apiserver memory series,
container_memory_working_set_bytesfromResourceUsageSummary, which is essentially RSS. working_set moves with GC pacing, not just real memory use, so on its own it cannot distinguish:It fails the other way too: a real leak can hide under a flat working_set if GC compensates. So every allocation-pattern change risks a false alarm, and real regressions can be masked.
Ideas
We can surface additional metrics that are already scraped:
go_memstats_heap_inuse_bytes), flat hererate(go_gc_duration_seconds_count[5m])), dropped 6xrate(go_memstats_alloc_bytes_total[5m])), dropped 6xSidenote: the proto variant dashboard also shows a "regression" from an allocation improvement (kubernetes/kubernetes#139350), so this is a recurring pattern of allocation improvements reading as memory regressions on the dashboard.