Skip to content

Hitting Container Kernel Memory Limit causes OOM on Docker Host #1001

@MrSerth

Description

@MrSerth
  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

If a kernel memory limit has been set for a specific container and is hit during program execution, only the container is affected by the kernel Out Of Memory (OOM) killer. Any process within the container might be affected and as a worst-case scenario, the container might quit.

Further, we expected the documentation regarding the --kernel-memory option to be correct. For our case, with a kernel memory limit below the memory limit (both set), it states:

If a container is using an unexpected amount of either type of memory, it runs out of memory without affecting other containers or the host machine. Within this setting, if the kernel memory limit is lower than the user memory limit, running out of kernel memory causes the container to experience an OOM error.

Actual behavior

If the kernel memory limit of the container is hit and more kernel memory is required, the Docker host (!, not the container) experiences a system-wide OOM (even though the host has more than enough free memory available). As a result, arbitrary applications and system services might be killed. In our experience, system-critical services such as the file system driver for ext4 were affected, causing the file system to get corrupted and further services to crash.

In our testings, the documentation proved to be incorrect: The host and other containers were affected by OOM killer.

Showcase of a fork-bomb crash on CentOS 8.1

Shell (shortend, see this repo for full log):

root@production:~# docker run -it --rm --memory=128M --kernel-memory=16M lmmdock/fork-bomb
Traceback (most recent call last):
  File "/bomb.py", line 6, in <module>
OSError: [Errno 12] Cannot allocate memory
    os.fork()
failed to resize tty, using default size
                                        ERRO[0004] error waiting for container: unexpected EOF
root@production:~# docker ps -a
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Syslog (shortend, see this repo for full log):

[Thu May 14 23:17:24 2020] slab_out_of_memory: 72 callbacks suppressed
[Thu May 14 23:17:24 2020] SLUB: Unable to allocate memory on node -1, gfp=0xdc0(GFP_KERNEL|__GFP_ZERO)
[Thu May 14 23:17:24 2020]   cache: signal_cache(632851:3a166877f91031145f6dc491df3c915c57f1439f4b3bcea4c6a37d1885ea27ea), object size: 1056, buffer size: 1088, default order: 3, min order: 0
[Thu May 14 23:17:24 2020]   node 0: slabs: 31, objs: 930, free: 0
[Thu May 14 23:17:24 2020] oom_kill_process: 70 callbacks suppressed
[Thu May 14 23:17:24 2020] python3 invoked oom-killer: gfp_mask=0x0(), order=0, oom_score_adj=0
[Thu May 14 23:17:24 2020] CPU: 25 PID: 3143648 Comm: python3 Not tainted 5.4.0-29-generic #33-Ubuntu
[Thu May 14 23:17:24 2020] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.10.2-1ubuntu1 04/01/2014
[Thu May 14 23:17:24 2020] Mem-Info:
[Thu May 14 23:17:24 2020] active_anon:57988 inactive_anon:147 isolated_anon:0
                            active_file:578569 inactive_file:1353156 isolated_file:0
                            unevictable:4579 dirty:120 writeback:0 unstable:0
                            slab_reclaimable:559978 slab_unreclaimable:198012
                            mapped:44734 shmem:289 pagetables:2081 bounce:0
                            free:30149832 free_pcp:4723 free_cma:0
[Thu May 14 23:17:24 2020] Node 0 active_anon:231952kB inactive_anon:588kB active_file:2314276kB inactive_file:5412624kB unevictable:18316kB isolated(anon):0kB isolated(file):0kB mapped:178936kB dirty:480kB writeback:0kB shmem:1156kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 4096kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Thu May 14 23:17:24 2020] Node 0 DMA free:15908kB min:8kB low:20kB high:32kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Thu May 14 23:17:24 2020] lowmem_reserve[]: 0 2966 128782 128782 128782
[Thu May 14 23:17:24 2020] Node 0 DMA32 free:3061820kB min:1556kB low:4592kB high:7628kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129200kB managed:3063664kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:1820kB local_pcp:0kB free_cma:0kB
[Thu May 14 23:17:24 2020] lowmem_reserve[]: 0 0 125816 125816 125816
[Thu May 14 23:17:24 2020] Node 0 Normal free:117521600kB min:66016kB low:194848kB high:323680kB active_anon:231952kB inactive_anon:588kB active_file:2314276kB inactive_file:5412624kB unevictable:18316kB writepending:480kB present:131072000kB managed:128843628kB mlocked:18316kB kernel_stack:26656kB pagetables:8324kB bounce:0kB free_pcp:17052kB local_pcp:652kB free_cma:0kB
[Thu May 14 23:17:24 2020] lowmem_reserve[]: 0 0 0 0 0
[Thu May 14 23:17:24 2020] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[Thu May 14 23:17:24 2020] Node 0 DMA32: 1*4kB (M) 1*8kB (U) 3*16kB (M) 2*32kB (M) 3*64kB (M) 2*128kB (M) 2*256kB (M) 4*512kB (UM) 3*1024kB (UM) 2*2048kB (UM) 745*4096kB (M) = 3061820kB
[Thu May 14 23:17:24 2020] Node 0 Normal: 237978*4kB (UME) 179547*8kB (UME) 87980*16kB (UME) 56977*32kB (UME) 25673*64kB (UME) 7943*128kB (UME) 1821*256kB (UME) 494*512kB (UME) 157*1024kB (UME) 24*2048kB (UME) 26442*4096kB (M) = 117514464kB
[Thu May 14 23:17:24 2020] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Thu May 14 23:17:24 2020] 1906236 total pagecache pages
[Thu May 14 23:17:24 2020] 0 pages in swap cache
[Thu May 14 23:17:24 2020] Swap cache stats: add 0, delete 0, find 0/0
[Thu May 14 23:17:24 2020] Free swap  = 0kB
[Thu May 14 23:17:24 2020] Total swap = 0kB
[Thu May 14 23:17:24 2020] 33554298 pages RAM
[Thu May 14 23:17:24 2020] 0 pages HighMem/MovableOnly
[Thu May 14 23:17:24 2020] 573498 pages reserved
[Thu May 14 23:17:24 2020] 0 pages cma reserved
[Thu May 14 23:17:24 2020] 0 pages hwpoisoned
[Thu May 14 23:17:24 2020] Tasks state (memory values in pages):
[Thu May 14 23:17:24 2020] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=3a166877f91031145f6dc491df3c915c57f1439f4b3bcea4c6a37d1885ea27ea,mems_allowed=0,global_oom,task_memcg=/system.slice/containerd.service,task=containerd-shim,pid=3143527,uid=0
[Thu May 14 23:17:24 2020] Out of memory: Killed process 3143527 (containerd-shim) total-vm:108572kB, anon-rss:620kB, file-rss:4544kB, shmem-rss:0kB, UID:0 pgtables:72kB oom_score_adj:1
[Thu May 14 23:17:24 2020] oom_reaper: reaped process 3143527 (containerd-shim), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

[Thu May 14 23:17:24 2020] Out of memory: Killed process 2296209 (containerd) total-vm:19365932kB, anon-rss:59336kB, file-rss:34872kB, shmem-rss:0kB, UID:0 pgtables:2608kB oom_score_adj:0
[Thu May 14 23:17:24 2020] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=3a166877f91031145f6dc491df3c915c57f1439f4b3bcea4c6a37d1885ea27ea,mems_allowed=0,global_oom,task_memcg=/system.slice/containerd.service,task=containerd,pid=2296209,uid=0
[Thu May 14 23:17:24 2020] oom_reaper: reaped process 2296209 (containerd), now anon-rss:0kB, file-rss:14180kB, shmem-rss:0kB

Steps to reproduce the behavior

⚠️ Warning: Some system services might be killed by the kernel OOM!
docker run -it --rm --memory=128M --kernel-memory=16M lmmdock/fork-bomb

Source Code / Dockerfile / Log output:

Output of docker version:

Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 23:42:35 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 22:48:33 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 243
  Running: 243
  Paused: 0
  Stopped: 0
 Images: 8
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: default
  userns
 Kernel Version: 5.4.0-29-generic
 Operating System: Ubuntu 20.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.8GiB
 Name: production
 ID: VM63:PMIK:WNWY:WMIX:BGEX:TEIN:5T44:5E5Y:TM7Z:SRC2:ZYVO:HDZI
 Docker Root Dir: /var/lib/docker/231072.231072
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

  • The output above is from a virtualized system within an OpenStack cluster.
  • We tested other systems (physical and other virtualization providers) and the following OS: Ubuntu 16.04 LTS (Kernel 4.4.0), Ubuntu 18.04 LTS (Kernel 5.3.0), Ubuntu 20.04 LTS (Kernel 5.4.0-29), Debian, CentOS 8.1 (Kernel 4.18.0)
    → All showed similar behavior.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions