In high-density deployment scenarios (many containers starting/mounting layers concurrently), we observed consistently high CPU usage
Based on pprof analysis, a large portion of CPU time is spent in TOC JSON decode and memory/GC work, instead of actual payload read/decompression.
Problem
Under concurrent startup/mount pressure, JSON-based TOC parsing becomes a bottleneck:
- TOC deserialization is CPU-intensive
- allocation pressure increases GC overhead
- mount/start scalability degrades before I/O becomes the limiting factor
This looks like a structural metadata-format bottleneck rather than a workload-tuning issue.
Proposal
Introduce an optional non-JSON TOC encoding path (e.g., FlatBuffers), while keeping JSON as default for compatibility.
High-level idea:
- Keep legacy JSON read/write behavior unchanged by default.
- Add a build-time TOC format option (default
json, optional flatbuffers).
- Encode TOC format/version in footer in a backward-compatible way.
- At runtime, detect footer version/token and route to the matching parser.
This enables incremental adoption and performance validation without breaking existing users/images.
In high-density deployment scenarios (many containers starting/mounting layers concurrently), we observed consistently high CPU usage
Based on pprof analysis, a large portion of CPU time is spent in TOC JSON decode and memory/GC work, instead of actual payload read/decompression.
Problem
Under concurrent startup/mount pressure, JSON-based TOC parsing becomes a bottleneck:
This looks like a structural metadata-format bottleneck rather than a workload-tuning issue.
Proposal
Introduce an optional non-JSON TOC encoding path (e.g., FlatBuffers), while keeping JSON as default for compatibility.
High-level idea:
json, optionalflatbuffers).This enables incremental adoption and performance validation without breaking existing users/images.