containerd-shim-runc-v2 slow memory leak/race #13561

Description

I'm running containerd v2.2.4 on a IoT device with a 2 core arm 32-bit (armv7) cpu that is a bit on the slower side. Each IoT device runs about 18 containers. In this setup I observe a continuous slow memory growth that I initially reported here containerd/ttrpc#236.

Since this problem manifests in the containerd-shim-runc-v2 I'm filing this bug here as well to reach wider audience.

Steps to reproduce the issue

For me the slow memory growth is 100% reproducible using both containerd v1.7 and v2.2 versions.

Since the problem is most likely caused by a race between two gorotinues the CPU model and number of cores is most likely a factor in how frequent the leaks are.

In order to get detailed golang memory metrics I've instrumented the containerd-shim-runc-v2 using the github.com/prometheus/client_golang/prometheus package. Memory growth happens regardless of if this instrumentation is used or not.

Describe the results you received and expected

Since writing containerd/ttrpc#236 I've now verified that with my proposed fix the memory growth goes away when running same setup with and without fix for 7 days by instrumenting all containerd-shim-runc-v2 shim processes to record the go_memstats_heap_alloc_bytes prometheus memory metric.

Without the fix all 18 shim processes consumes up to 15,2 MB after 7 days. With the proposed fix the same number of shims consume 12.6 MB, which is a ~ 3 MB (20%) decrease over 7 days for 18 shim process.

For the record I've also run the containerd-shim-runc-v2 shim without proposed fix for longer time periods (up to 30 days) and observe continuous memory growth during the entire time.

What version of containerd are you using?

2.2.4

Any other relevant information

In the containerd/ttrpc#236 I tried to explain that the memory leak is caused by a race between two goroutines causing the streamHandler sync.Map to grow unbounded over time.

This is the proposed fix I've been testing

diff --git a/vendor/github.com/containerd/ttrpc/server.go b/vendor/github.com/containerd/ttrpc/server.go
index bb71de677..9b9c0bdc7 100644
--- a/vendor/github.com/containerd/ttrpc/server.go
+++ b/vendor/github.com/containerd/ttrpc/server.go
@@ -486,7 +486,9 @@ func (c *serverConn) run(sctx context.Context) {
                                        continue
                                }

-                               streams.Store(id, sh)
+                               if sh != nil {
+                                       streams.Store(id, sh)
+                               }
                                atomic.AddInt32(&active, 1)
                        }
                        // TODO: else we must ignore this for future compat. log this?

Show configuration if it is related to CRI plugin.

No response