1.5.10 causes memory leak in mysql container; regression from 1.4.13

[C-Higgins]

Description

I updated containerd.io on Fedora from 1.4.13 to 1.5.10. After this, all mysql docker containers across multiple projects began using 20+GB of memory where before they used <300MB. I restarted the computer and confirmed the problem persisted. I then rolled back to 1.4.13 and the problem was gone.

Steps to reproduce the issue

1. Run a docker container with linux/amd64 arch with ENV MYSQL_VERSION=5.7.36-0ubuntu0.18.04.1
2. Start mysqld inside the container
3. Observe massive memory usage in 1.5.10

Describe the results you received and expected

No regression

What version of containerd are you using?

1.5.10

Any other relevant information

runc version 1.0.3
commit: v1.0.3-0-gf46b6ba
spec: 1.0.2-dev
go: go1.16.15
libseccomp: 2.5.3

Linux localhost.localdomain 5.16.7-200.fc35.x86_64 #1 SMP PREEMPT Sun Feb 6 19:53:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Show configuration if it is related to CRI plugin.

No response

[ssaaiidd]

I have the same issue

runc version 1.0.3
commit: v1.0.3-0-gf46b6ba
spec: 1.0.2-dev
go: go1.16.15
libseccomp: 2.5.3

I implemented a temp solution using ulimit like this comment. It's worked for me. docker-library/mysql#579 (comment)

[ssaaiidd]

The difference may be due to this PR: #4475

[C-Higgins]

I have the same issue

runc version 1.0.3
commit: v1.0.3-0-gf46b6ba
spec: 1.0.2-dev
go: go1.16.15
libseccomp: 2.5.3

I implemented a temp solution using ulimit like this comment. It's worked for me. docker-library/mysql#579 (comment)

Yes, can confirm that this is a valid workaround. It seems very likely that the increase to infinity nofile size is the problem on some systems.

[thaJeztah]

That's .... interesting. If that's causing an issue, that feels like a bug in systemd 🤔 (as infinity should be equivalent to the old value)

[C-Higgins]

I will do a repro test later this week with 1.5 and 1.6, after modifying the service file, to see if the problem is isolated to just that line

[thaJeztah]

Thanks in advance for that!

Slightly orthogonal; I recall I once opened a ticket (more a "reminder"); to look into other options in this area; that ticket was related to the systemd unit for docker-ce, but the same applies to the one for containerd (which was originally based on what we had in docker); docker/for-linux#73. If anyone arriving here is interested in doing some research into those options (also taking into account if they're supported on all "usual" distros), that would be welcome.

[C-Higgins]

I installed 1.5 and confirmed problem still persisted. I then modified /lib/systemd/system/containerd.service to read LimitNOFILE=1048576. I then restarted the service and the daemon: sudo systemctl restart service containerd sudo systemctl daemon-reload. Problem still persisted. I then restarted the service once more, because I thought the daemon might have cached the service (so in total I restarted service, daemon, service) and the problem was gone. Docker stable repos don't have 1.6 available so I didn't test that.

[fuweid]

I think you should do daemon-reload first and then restart containerd. And it only makes sure that it works for new containers, not for existing ones.

[CryptoRodeo]

I applied the same fix as @C-Higgins and it also got rid of the issue on my machine.
(containerd version 1.5.11 )

[forestuser]

I installed 1.5 and confirmed problem still persisted. I then modified /lib/systemd/system/containerd.service to read LimitNOFILE=1048576. I then restarted the service and the daemon: sudo systemctl restart service containerd sudo systemctl daemon-reload. Problem still persisted. I then restarted the service once more, because I thought the daemon might have cached the service (so in total I restarted service, daemon, service) and the problem was gone. Docker stable repos don't have 1.6 available so I didn't test that.

Not work for me

docker-library/mysql#840

апр 11 08:40:09 tk-kv.localhost kernel: eth0: renamed from vethe040851
апр 11 08:40:09 tk-kv.localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethf1fa9b5: link becomes ready
апр 11 08:40:09 tk-kv.localhost kernel: docker0: port 1(vethf1fa9b5) entered blocking state
апр 11 08:40:09 tk-kv.localhost kernel: docker0: port 1(vethf1fa9b5) entered forwarding state
апр 11 08:40:09 tk-kv.localhost NetworkManager[702]: [1649648409.8858] device (vethf1fa9b5): carrier: link connected
апр 11 08:40:09 tk-kv.localhost NetworkManager[702]: [1649648409.8861] device (docker0): carrier: link connected
апр 11 08:40:09 tk-kv.localhost systemd-resolved[62431]: Failed to determine the local hostname and LLMNR/mDNS names, ignoring: No such device or address
апр 11 08:40:11 tk-kv.localhost avahi-daemon[704]: Joining mDNS multicast group on interface vethf1fa9b5.IPv6 with address fe80::440a:aff:fee3:4278.
апр 11 08:40:11 tk-kv.localhost avahi-daemon[704]: New relevant interface vethf1fa9b5.IPv6 for mDNS.
апр 11 08:40:11 tk-kv.localhost avahi-daemon[704]: Registering new address record for fe80::440a:aff:fee3:4278 on vethf1fa9b5.*.
апр 11 08:40:27 tk-kv.localhost kernel: mysqld invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
апр 11 08:40:27 tk-kv.localhost kernel: CPU: 0 PID: 294221 Comm: mysqld Tainted: P OE 5.16.18-200.fc35.x86_64 #1
апр 11 08:40:27 tk-kv.localhost kernel: Hardware name: Gigabyte Tecohnology Co., Ltd. H61M-DS2/H61M-DS2, BIOS F4 12/21/2011
апр 11 08:40:27 tk-kv.localhost kernel: Call Trace:
апр 11 08:40:27 tk-kv.localhost kernel:
апр 11 08:40:27 tk-kv.localhost kernel: dump_stack_lvl+0x48/0x5e
апр 11 08:40:27 tk-kv.localhost kernel: dump_header+0x4a/0x1fd
апр 11 08:40:27 tk-kv.localhost kernel: oom_kill_process.cold+0xb/0x10
апр 11 08:40:27 tk-kv.localhost kernel: out_of_memory+0x229/0x4d0
апр 11 08:40:27 tk-kv.localhost kernel: mem_cgroup_out_of_memory+0x120/0x140
апр 11 08:40:27 tk-kv.localhost kernel: try_charge_memcg+0x6a6/0x760
апр 11 08:40:27 tk-kv.localhost kernel: ? __alloc_pages+0xd6/0x210
апр 11 08:40:27 tk-kv.localhost kernel: charge_memcg+0x36/0x130
апр 11 08:40:27 tk-kv.localhost kernel: __mem_cgroup_charge+0x29/0x80
апр 11 08:40:27 tk-kv.localhost kernel: __handle_mm_fault+0xb56/0x1470
апр 11 08:40:27 tk-kv.localhost kernel: handle_mm_fault+0xb2/0x280
апр 11 08:40:27 tk-kv.localhost kernel: do_user_addr_fault+0x1ce/0x690
апр 11 08:40:27 tk-kv.localhost kernel: exc_page_fault+0x72/0x170
апр 11 08:40:27 tk-kv.localhost kernel: ? asm_exc_page_fault+0x8/0x30
апр 11 08:40:27 tk-kv.localhost kernel: asm_exc_page_fault+0x1e/0x30
апр 11 08:40:27 tk-kv.localhost kernel: RIP: 0033:0x7f9f975fa2b3
апр 11 08:40:27 tk-kv.localhost kernel: Code: 47 10 f3 0f 7f 44 17 e0 f3 0f 7f 47 20 f3 0f 7f 44 17 d0 f3 0f 7f 47 30 f3 0f 7f 44 17 c0 48 01 fa 48 83 e2 c0 48 39 d1 74 c0 <66> 0f 7f 01 66 0f 7f 41 10 66 0f 7f 41 20 66 0f 7f 41 30>
апр 11 08:40:27 tk-kv.localhost kernel: RSP: 002b:00007ffc90812a38 EFLAGS: 00010202
апр 11 08:40:27 tk-kv.localhost kernel: RAX: 00007f9b97553430 RBX: 000000003ffffff8 RCX: 00007f9c9693b000
апр 11 08:40:27 tk-kv.localhost kernel: RDX: 00007f9f97552f80 RSI: 0000000000000000 RDI: 00007f9b97553430
апр 11 08:40:27 tk-kv.localhost kernel: RBP: 00007ffc90812a80 R08: 0000000000000000 R09: 00007f9b97553060
апр 11 08:40:27 tk-kv.localhost kernel: R10: 0000000000000022 R11: 00007f9b97553420 R12: 00007f9b97553030
апр 11 08:40:27 tk-kv.localhost kernel: R13: 0000564afa5858a8 R14: 0000564afa5858a0 R15: 0000000000000000
апр 11 08:40:27 tk-kv.localhost kernel:
апр 11 08:40:27 tk-kv.localhost kernel: memory: usage 2097152kB, limit 2097152kB, failcnt 223331
апр 11 08:40:27 tk-kv.localhost kernel: swap: usage 2097152kB, limit 2097152kB, failcnt 2
апр 11 08:40:27 tk-kv.localhost kernel: Memory cgroup stats for /system.slice/docker-39dbb45552a10932a86eafc687fbae48d5bec97eb6dac316249cb868a3c33d50.scope:
апр 11 08:40:27 tk-kv.localhost kernel: anon 2137669632
file 0
kernel_stack 49152
pagetables 8548352
percpu 144
sock 0
shmem 0
file_mapped 0
file_dirty 0
file_writeback 876544
swapcached 909312
anon_thp 0
file_thp 0
shmem_thp 0
inactive_anon 2138464256
active_anon 102400
inactive_file 0
active_file 0
unevictable 0
slab_reclaimable 78664
slab_unreclaimable 168320
slab 246984
workingset_refault_anon 0
workingset_refault_file 1047
workingset_activate_anon 0
workingset_activate_file 0
workingset_restore_anon 0
workingset_restore_file 0
workingset_nodereclaim 0
pgfault 1048185
pgmajfault 37
pgrefill 25
pgscan 2092313
pgsteal 525113
pgactivate 45
pgdeactivate 22
pglazyfree 0
pglazyfreed 0
thp_fault_alloc 0
thp_collapse_alloc 0
апр 11 08:40:27 tk-kv.localhost kernel: Tasks state (memory values in pages):
апр 11 08:40:27 tk-kv.localhost kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
апр 11 08:40:27 tk-kv.localhost kernel: [ 294189] 0 294189 967 658 40960 93 0 docker-entrypoi
апр 11 08:40:27 tk-kv.localhost kernel: [ 294220] 0 294220 967 499 40960 93 0 docker-entrypoi
апр 11 08:40:27 tk-kv.localhost kernel: [ 294221] 0 294221 4203297 524054 8491008 524160 0 mysqld
апр 11 08:40:27 tk-kv.localhost kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=docker-39dbb45552a10932a86eafc687fbae48d5bec97eb6dac316249cb868a3c33d50.scope,mems_allowed=0,oom_memcg=/system.slice/docker-39dbb4>
апр 11 08:40:27 tk-kv.localhost kernel: Memory cgroup out of memory: Killed process 294221 (mysqld) total-vm:16813188kB, anon-rss:2087228kB, file-rss:8988kB, shmem-rss:0kB, UID:0 pgtables:8292kB oom_score_adj:0
апр 11 08:40:27 tk-kv.localhost systemd[1]: docker-39dbb45552a10932a86eafc687fbae48d5bec97eb6dac316249cb868a3c33d50.scope: A process of this unit has been killed by the OOM killer.
апр 11 08:40:27 tk-kv.localhost kernel: oom_reaper: reaped process 294221 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
апр 11 08:40:27 tk-kv.localhost systemd[1]: docker-39dbb45552a10932a86eafc687fbae48d5bec97eb6dac316249cb868a3c33d50.scope: Deactivated successfully.

[forestuser]

Hi, its work for me. MYSQL

docker system prune -a
nano /lib/systemd/system/containerd.service
LimitNOFILE=1048576
systemctl daemon-reload
systemctl restart containerd

and not work

gitlab-runner -> docker -> containerd 1.5.11

yarn install
апр 12 08:40:52 debug3.local gitlab-runner[1068]: Checking for jobs... nothing runner=rccAVBWo
апр 12 08:40:54 debug3.local kernel: webpack invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
апр 12 08:40:54 debug3.local kernel: CPU: 15 PID: 3018610 Comm: webpack Not tainted 5.16.18-200.fc35.x86_64 #1
апр 12 08:40:54 debug3.local kernel: Hardware name: Micro-Star International Co., Ltd. MS-7A38/B450M PRO-VDH MAX (MS-7A38), BIOS B.70 06/10/2020

debug3.local kernel: Memory cgroup out of memory: Killed process 3017931 (webpack) total-vm:38547452kB, anon-rss:3423448kB, file-rss:2896kB, shmem-rss:0kB, UID:1000 pgtables:85972kB oom_score_adj:0

[kangclzjc]

Seems there is a bug in Systemd, if you need more than 65535 files, need to change the LimitNOFILE manually to a bigger number, such as 1048576

systemd/systemd#6559

Maybe we need to revert this PR 4475

[jsalgado78]

It works fine on Ubuntu 18.04.6 but it fails on Fedora 35 running same containerd and runc version

go1.16.15
containerd: 1.5.11
runc: 1.0.3

These commands crashes on Fedora 35 because of container use too much memory:
docker run -it --rm mysql:5.7.36
docker run -it --rm mysql:5.5.62

This command works fine on Fedora 35:
docker run -it --rm mysql:8.0.29

It works fine on Fedora 35 running mysql:5.x after setting LimitNOFILE to 1048576 in containerd.service as @kangclzjc said