snapshotter periodically overflows the stack

[mattmoor]

I'm using https://github.com/itsmurugappan/stargz-snapshotter-k3s (cc @itsmurugappan) on my RPi4 and I am periodically seeing the pods crash with what looks like an infinite recursion:

Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: runtime: goroutine stack exceeds 1000000000-byte limit
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: runtime: sp=0x40271003b0 stack=[0x4027100000, 0x4047100000]
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: fatal error: stack overflow
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: runtime stack:
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: runtime.throw(0x10f129e, 0xe)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /snap/go/6746/src/runtime/panic.go:1116 +0x54
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: runtime.newstack()
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /snap/go/6746/src/runtime/stack.go:1067 +0x600
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: runtime.morestack()
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /snap/go/6746/src/runtime/asm_arm64.s:308 +0x70
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: goroutine 2546 [running]:
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: runtime.mallocgc(0x8, 0xecb060, 0x40033e3001, 0x0)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /snap/go/6746/src/runtime/malloc.go:903 +0x8b8 fp=0x40271003b0 sp=0x40271003b0 pc=0x1e148
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: runtime.newobject(0xecb060, 0x4001ffb3e0)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /snap/go/6746/src/runtime/malloc.go:1195 +0x38 fp=0x40271003e0 sp=0x40271003b0 pc=0x1e358
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/estargz.(*Reader).OpenFile(0x4001ffb380, 0x40045356a0, 0x15, 0x37f7d0, 0x4027100648, 0x2)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/estargz/estargz.go:356 +0x3cc fp=0x4027100460 sp=0x40271003e0 pc=0xd9d22c
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/fs/reader.(*reader).cacheWithReader.func1(0x40045356a0, 0x15, 0x40039d5800, 0xda1508)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/fs/reader/reader.go:208 +0x358 fp=0x4027100580 sp=0x4027100460 pc=0xda1428
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/estargz.(*TOCEntry).ForeachChild(0x40039d5900, 0x4027100630)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/estargz/types.go:209 +0x98 fp=0x4027100610 sp=0x4027100580 pc=0xd9e288
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/fs/reader.(*reader).cacheWithReader(0x400062c480, 0x12cd300, 0x4004017f80, 0x4001ffb440, 0x40000c09b0, 0x40039d5900, 0x4001ffb380, 0x1198308, 0x4000286040, 0x1, ...)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/fs/reader/reader.go:176 +0xb8 fp=0x40271006a0 sp=0x4027100610 pc=0xd9f6c8
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/fs/reader.(*reader).cacheWithReader.func1(0x10e4cd5, 0x1, 0x40039d5900, 0xda1501)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/fs/reader/reader.go:192 +0x25c fp=0x40271007c0 sp=0x40271006a0 pc=0xda132c
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/estargz.(*TOCEntry).ForeachChild(0x40039d5900, 0x4027100870)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/estargz/types.go:209 +0x98 fp=0x4027100850 sp=0x40271007c0 pc=0xd9e288
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/fs/reader.(*reader).cacheWithReader(0x400062c480, 0x12cd300, 0x4004017f80, 0x4001ffb440, 0x40000c09b0, 0x40039d5900, 0x4001ffb380, 0x1198308, 0x4000286040, 0x1, ...)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/fs/reader/reader.go:176 +0xb8 fp=0x40271008e0 sp=0x4027100850 pc=0xd9f6c8
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/fs/reader.(*reader).cacheWithReader.func1(0x10e4cd5, 0x1, 0x40039d5900, 0xda1501)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/fs/reader/reader.go:192 +0x25c fp=0x4027100a00 sp=0x40271008e0 pc=0xda132c
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/estargz.(*TOCEntry).ForeachChild(0x40039d5900, 0x4027100ab0)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/estargz/types.go:209 +0x98 fp=0x4027100a90 sp=0x4027100a00 pc=0xd9e288
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/fs/reader.(*reader).cacheWithReader(0x400062c480, 0x12cd300, 0x4004017f80, 0x4001ffb440, 0x40000c09b0, 0x40039d5900, 0x4001ffb380, 0x1198308, 0x4000286040, 0x1, ...)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/fs/reader/reader.go:176 +0xb8 fp=0x4027100b20 sp=0x4027100a90 pc=0xd9f6c8
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/fs/reader.(*reader).cacheWithReader.func1(0x10e4cd5, 0x1, 0x40039d5900, 0xda1501)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/fs/reader/reader.go:192 +0x25c fp=0x4027100c40 sp=0x4027100b20 pc=0xda132c
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/estargz.(*TOCEntry).ForeachChild(0x40039d5900, 0x4027100cf0)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/estargz/types.go:209 +0x98 fp=0x4027100cd0 sp=0x4027100c40 pc=0xd9e288
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/fs/reader.(*reader).cacheWithReader(0x400062c480, 0x12cd300, 0x4004017f80, 0x4001ffb440, 0x40000c09b0, 0x40039d5900, 0x4001ffb380, 0x1198308, 0x4000286040, 0x1, ...)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/fs/reader/reader.go:176 +0xb8 fp=0x4027100d60 sp=0x4027100cd0 pc=0xd9f6c8
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/fs/reader.(*reader).cacheWithReader.func1(0x10e4cd5, 0x1, 0x40039d5900, 0xda1501)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/fs/reader/reader.go:192 +0x25c fp=0x4027100e80 sp=0x4027100d60 pc=0xda132c
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]: github.com/containerd/stargz-snapshotter/estargz.(*TOCEntry).ForeachChild(0x40039d5900, 0x4027100f30)
Dec 21 22:59:08 ubuntu containerd-stargz-grpc[43571]:         /home/ubuntu/stargz-snapshotter/estargz/types.go:209 +0x98 fp=0x4027100f10 sp=0x4027100e80 pc=0xd9e288

I'll poke around a bit to see if I can spot anything, but wanted to report this.

[mattmoor]

This feels like an infinite recursion, but perhaps this should be wrapped in an eg.Go()?

stargz-snapshotter/fs/reader/reader.go

Line 192 in 9a7db93

if err := gr.cacheWithReader(ctx, eg, sem, e, r, filter, opts...); err != nil {

[ktock]

Yes, this seems to be the issue on cacheWithReader. What image are you pulling?

[mattmoor]

I've deployed github.com/mattmoor/mink to a little K3s cluster on my RPi4.

With estargz encoding enabled in ko the pods periodically crash loop, but this seems to have gone away if I disable the estargz encoding.

[mattmoor]

it's hard to narrow down from the stack trace which image it is, but basically all of them crash loop over time. I'm not sure if all of them have problems or if one image is causing problems for everyone else.

[ktock]

It turned out that if the input tar contains "/" entry (root directory itself), infinite reference loop of entries (root -> root) is created and this causes infinite walk of the directory tree. #221 will address this.