nerdbox doesn't work on containerd 2.2.5 and containerd 2.3.x

[ningmingxiao]

may be we should this pr
containerd/containerd@3c0e8a5

use json.Unmarshal now it doesn't work now

func parseStartResponse(response []byte) (client.BootstrapParams, error) {
	var params client.BootstrapParams

	if err := json.Unmarshal(response, &params); err != nil || params.Version < 2 {
		// Use TTRPC for legacy shims
		params.Address = string(response)
		params.Protocol = "ttrpc"
		params.Version = 2
	}

@dmcgowan

[dmcgowan]

We updated to use the bootstrap protocol introduced in 2.3. Haven't tested it with older versions of containerd, most the changes here end up needing newer containerd anyway.

[ningmingxiao]

but I also find it doesn't work on containerd 2.3 latest version now it should be another bug,we should add some ci to test containerd realease version

[ningmingxiao]

use containerd main branch

# ctr run -t --rm --snapshotter erofs --runtime io.containerd.nerdbox.v1 docker.io/library/alpine:latest test03  /bin/sh
ctr: failed to handle lowerdir option: template: :1:3: executing "" at <mount 1>: error calling mount: failed to resolve mount 1: mount type erofs cannot be directly viewed: not implemented

[hsiangkao]

hi,

use containerd main branch

# ctr run -t --rm --snapshotter erofs --runtime io.containerd.nerdbox.v1 docker.io/library/alpine:latest test03  /bin/sh
ctr: failed to handle lowerdir option: template: :1:3: executing "" at <mount 1>: error calling mount: failed to resolve mount 1: mount type erofs cannot be directly viewed: not implemented

could you debug that if you got some chance?

[ningmingxiao]

ok @hsiangkao

[ningmingxiao]

I add some log print mouts
mounts is [{Type:mkfs/ext4 Source:/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/2/rwlayer.img Target: Options:[X-containerd.mkfs.fs=ext4 X-containerd.mkfs.size=67108864 rw loop]}
{Type:erofs Source:/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs Target: Options:[ro loop]}
{Type:format/mkdir/overlay Source:overlay Target: Options:[X-containerd.mkdir.path={{ mount 0 }}/upper:0755 X-containerd.mkdir.path={{ mount 0 }}/work:0755 workdir={{ mount 0 }}/work
upperdir={{ mount 0 }}/upper lowerdir={{ mount 1 }}]}]

on linux it will not return an error but if it is macos it will return an error , why runtime.GOOS == "darwin" should return an error ?

func withReadonlyFS(ctx context.Context, client Client, mounts []mount.Mount, fn func(fs.FS) error) error {
	// Try to avoid mount if possible by using fsview to directly open
	// overlay/erofs/bind mounts without actually mounting them
	if viewFS, err := fsview.FSMounts(mounts); err == nil && viewFS != nil {
		defer viewFS.Close()
		return fn(viewFS)
	} else if !errors.Is(err, errdefs.ErrNotImplemented) || runtime.GOOS == "darwin" {
		return err
	}

If it is not supported we should failback to old behavior.
@hsiangkao @dmcgowan

[hsiangkao]

I add some log print mouts mounts is [{Type:mkfs/ext4 Source:/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/2/rwlayer.img Target: Options:[X-containerd.mkfs.fs=ext4 X-containerd.mkfs.size=67108864 rw loop]} {Type:erofs Source:/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs Target: Options:[ro loop]} {Type:format/mkdir/overlay Source:overlay Target: Options:[X-containerd.mkdir.path={{ mount 0 }}/upper:0755 X-containerd.mkdir.path={{ mount 0 }}/work:0755 workdir={{ mount 0 }}/work upperdir={{ mount 0 }}/upper lowerdir={{ mount 1 }}]}]

on linux it will not return an error but if it is macos it will return an error , why runtime.GOOS == "darwin" should return an error ?

func withReadonlyFS(ctx context.Context, client Client, mounts []mount.Mount, fn func(fs.FS) error) error {
	// Try to avoid mount if possible by using fsview to directly open
	// overlay/erofs/bind mounts without actually mounting them
	if viewFS, err := fsview.FSMounts(mounts); err == nil && viewFS != nil {
		defer viewFS.Close()
		return fn(viewFS)
	} else if !errors.Is(err, errdefs.ErrNotImplemented) || runtime.GOOS == "darwin" {
		return err
	}

If it is not supported we should failback to old behavior. @hsiangkao @dmcgowan

Hi @ningmingxiao, thanks for debugging.
Could you show the call stack too? basically I'm not sure how to mount erofs on mac, I guess that is what runtime.GOOS == "darwin" is used.

[ningmingxiao]

root@nmxdeMac-mini bin # ctr run -t --rm --snapshotter erofs --runtime io.containerd.nerdbox.v1 docker.io/library/alpine:latest test /bin/sh
goroutine 1 [running]:
github.com/containerd/containerd/v2/internal/fsview.stack(...)
/Users/nmx/code/containerd/internal/fsview/mount.go:88
github.com/containerd/containerd/v2/internal/fsview.resolveOverlayValue({0x53fd297a5129, 0x13}, {0x53fd29a16540, 0x2, 0x3})
/Users/nmx/code/containerd/internal/fsview/mount.go:201 +0x54
github.com/containerd/containerd/v2/internal/fsview.openFormatMount({{0x53fd29726540, 0x14}, {0x53fd29702f60, 0x7}, {0x0, 0x0}, {0x53fd2979f600, 0x5, 0x8}}, {0x53fd29a16540, ...})
/Users/nmx/code/containerd/internal/fsview/mount.go:146 +0x20c
github.com/containerd/containerd/v2/internal/fsview.resolveMount({{0x53fd29726540, 0x14}, {0x53fd29702f60, 0x7}, {0x0, 0x0}, {0x53fd2979f600, 0x5, 0x8}}, {0x53fd29a16540, ...})
/Users/nmx/code/containerd/internal/fsview/mount.go:110 +0x320
github.com/containerd/containerd/v2/internal/fsview.FSMounts({0x53fd29a16540?, 0x16d54fafe?, 0x53fd2978a9a0?})
/Users/nmx/code/containerd/internal/fsview/mount.go:83 +0x9c
github.com/containerd/containerd/v2/pkg/oci.withReadonlyFS({0x53fd29afe9e0?, 0x103a7f060?}, {0x53fd29ae4000?, 0x16d54fb47?}, {0x53fd29a16540?, 0x103950d00?, 0x0?}, 0x53fd29e1e590)
/Users/nmx/code/containerd/pkg/oci/spec_opts.go:950 +0x3c
github.com/containerd/containerd/v2/pkg/oci.WithAdditionalGIDs.func1({0x103a7f060, 0x53fd29ae4000}, {0x103a73580, 0x53fd29ab8140}, 0x53fd298ee180, 0x0?)
/Users/nmx/code/containerd/pkg/oci/spec_opts.go:943 +0x2b8
github.com/containerd/containerd/v2/cmd/ctr/commands/run.NewContainer.WithImageConfig.WithImageConfigArgs.func52({0x103a7f060, 0x53fd29ae4000}, {0x103a73580, 0x53fd29ab8140}, 0x53fd298ee180, 0x53fd29ce05a0)
/Users/nmx/code/containerd/pkg/oci/spec_opts.go:424 +0x474
github.com/containerd/containerd/v2/pkg/oci.ApplyOpts({0x103a7f060, 0x53fd29ae4000}, {0x103a73580, 0x53fd29ab8140}, 0x53fd298ee180, 0x53fd29ce05a0, {0x53fd29ca4800, 0x9, 0x16d54fb47?})
/Users/nmx/code/containerd/pkg/oci/spec.go:110 +0x80
github.com/containerd/containerd/v2/cmd/ctr/commands/run.NewContainer.WithSpec.func51({0x103a7f060?, 0x
goroutine 1 [running]:
github.com/containerd/containerd/v2/internal/fsview.stack(...)
/Users/nmx/code/containerd/internal/fsview/mount.go:88
github.com/containerd/containerd/v2/internal/fsview.resolveMount({{0x53fd29702f45, 0x9}, {0x53fd297a1400, 0x4f}, {0x0, 0x0}, {0x53fd297a2d00, 0x4, 0x4}}, {0x53fd29a16540, ...})
/Users/nmx/code/containerd/internal/fsview/mount.go:112 +0x210
github.com/containerd/containerd/v2/internal/fsview.resolveOverlayValue.func4(0x0)
/Users/nmx/code/containerd/internal/fsview/mount.go:237 +0xb0
reflect.Value.call({0x1038e59c0?, 0x53fd298b2a80?, 0x53fd29e1d788?}, {0x10307d799, 0x4}, {0x53fd29e1d9d0, 0x1, 0x102a33dcc?})
/usr/local/go/src/reflect/value.go:586 +0xac8
reflect.Value.Call({0x1038e59c0?, 0x53fd298b2a80?, 0x53fd29e1d858?}, {0x53fd29e1d9d0?, 0x53fd29e1d7d8?, 0x102a2d110?})
/usr/local/go/src/reflect/value.go:369 +0x94
text/template.safeCall({0x1038e59c0?, 0x53fd298b2a80?, 0x0?}, {0x53fd29e1d9d0?, 0x103a8e618?, 0x1038a0c00?})
/usr/local/go/src/text/template/funcs.go:366 +0x60
text/template.(*state).evalCall(0x53fd29739b30, {0x0?, 0x0?, 0x1?}, {0x1038e59c0?, 0x53fd298b2a80?, 0x3?}, 0x0, {0x103a83250, 0x53fd298b2c00}, ...)
/usr/local/go/src/text/template/exec.go:870 +0x7a0
text/template.(*state).evalFunction(0x53fd29739b30, {0x0?, 0x0?, 0x103955100?}, 0x53fd29e1db88?, {0x103a83250, 0x53fd298b2c00}, {0x53fd2978aa00, 0x2, 0x2}, ...)
/usr/local/go/src/text/template/exec.go:676 +0x158
text/template.(*state).evalCommand(0x53fd29739b30, {0x0?, 0x0?, 0x102a13794?}, 0x53fd29e1dc28?, {0x103905ba0?, 0x103b79cc0?, 0x53fd29cde750?})
/usr/local/go/src/text/template/exec.go:563 +0x160
text/template.(*state).evalPipeline(0x53fd29739b30, {0x0?, 0x0?, 0x102a3a650?}, 0x53fd2978f8c0)
/usr/local/go/src/text/template/exec.go:532 +0xf8
text/template.(*state).walk(0x53fd29739b30, {0x0?, 0x0?, 0x30?}, {0x103a83370?, 0x53fd298b2c60})
/usr/local/go/src/text/template/exec.go:267 +0xb0
text/template.(*state).walk(0x53fd29739b30, {0x0?, 0x0?, 0x53fd298b2ab0?}, {0x103a832e0?, 0x53fd298b2bd0?})
/usr/loc
goroutine 1 [running]:
github.com/containerd/containerd/v2/internal/fsview.stack(...)
/Users/nmx/code/containerd/internal/fsview/mount.go:88
github.com/containerd/containerd/v2/internal/fsview.resolveOverlayValue({0x53fd29726561, 0xd}, {0x53fd29a16540, 0x2, 0x3})
/Users/nmx/code/containerd/internal/fsview/mount.go:201 +0x54
github.com/containerd/containerd/v2/internal/fsview.openFormatMount-range1({0x53fd29726561?, 0x53fd2978aae0?})
/Users/nmx/code/containerd/internal/fsview/mount.go:162 +0x64
github.com/containerd/containerd/v2/internal/fsview.openFormatMount.SplitSeq.splitSeq.func2(...)
/usr/local/go/src/strings/iter.go:59
github.com/containerd/containerd/v2/internal/fsview.openFormatMount({{0x53fd29726540, 0x14}, {0x53fd29702f60, 0x7}, {0x0, 0x0}, {0x53fd2979f600, 0x5, 0x8}}, {0x53fd29a16540, ...})
/Users/nmx/code/containerd/internal/fsview/mount.go:161 +0x538
github.com/containerd/containerd/v2/internal/fsview.resolveMount({{0x53fd29726540, 0x14}, {0x53fd29702f60, 0x7}, {0x0, 0x0}, {0x53fd2979f600, 0x5, 0x8}}, {0x53fd29a16540, ...})
/Users/nmx/code/containerd/internal/fsview/mount.go:110 +0x320
github.com/containerd/containerd/v2/internal/fsview.FSMounts({0x53fd29a16540?, 0x16d54fafe?, 0x53fd2978a9a0?})
/Users/nmx/code/containerd/internal/fsview/mount.go:83 +0x9c
github.com/containerd/containerd/v2/pkg/oci.withReadonlyFS({0x53fd29afe9e0?, 0x103a7f060?}, {0x53fd29ae4000?, 0x16d54fb47?}, {0x53fd29a16540?, 0x103950d00?, 0x0?}, 0x53fd29e1e590)
/Users/nmx/code/containerd/pkg/oci/spec_opts.go:950 +0x3c
github.com/containerd/containerd/v2/pkg/oci.WithAdditionalGIDs.func1({0x103a7f060, 0x53fd29ae4000}, {0x103a73580, 0x53fd29ab8140}, 0x53fd298ee180, 0x0?)
/Users/nmx/code/containerd/pkg/oci/spec_opts.go:943 +0x2b8
github.com/containerd/containerd/v2/cmd/ctr/commands/run.NewContainer.WithImageConfig.WithImageConfigArgs.func52({0x103a7f060, 0x53fd29ae4000}, {0x103a73580, 0x53fd29ab8140}, 0x53fd298ee180, 0x53fd29ce05a0)
/Users/nmx/code/containerd/pkg/oci/spec_opts.go:424 +0x474
github.com/containerd/containerd/v
goroutine 1 [running]:
github.com/containerd/containerd/v2/internal/fsview.stack(...)
/Users/nmx/code/containerd/internal/fsview/mount.go:88
github.com/containerd/containerd/v2/internal/fsview.resolveMount({{0x53fd29702f54, 0x5}, {0x53fd297a1450, 0x4e}, {0x0, 0x0}, {0x53fd2978a960, 0x2, 0x2}}, {0x53fd29a16540, ...})
/Users/nmx/code/containerd/internal/fsview/mount.go:112 +0x210
github.com/containerd/containerd/v2/internal/fsview.resolveOverlayValue.func4(0x1)
/Users/nmx/code/containerd/internal/fsview/mount.go:237 +0xb0
reflect.Value.call({0x1038e59c0?, 0x53fd298b2db0?, 0x53fd29e1d6d8?}, {0x10307d799, 0x4}, {0x53fd29e1d920, 0x1, 0x102a33dcc?})
/usr/local/go/src/reflect/value.go:586 +0xac8
reflect.Value.Call({0x1038e59c0?, 0x53fd298b2db0?, 0x53fd29e1d7a8?}, {0x53fd29e1d920?, 0x53fd29e1d728?, 0x102a2d110?})
/usr/local/go/src/reflect/value.go:369 +0x94
text/template.safeCall({0x1038e59c0?, 0x53fd298b2db0?, 0x0?}, {0x53fd29e1d920?, 0x103a8e618?, 0x1038a0c00?})
/usr/local/go/src/text/template/funcs.go:366 +0x60
text/template.(*state).evalCall(0x53fd29739bd0, {0x0?, 0x0?, 0x1?}, {0x1038e59c0?, 0x53fd298b2db0?, 0x3?}, 0x0, {0x103a83250, 0x53fd298b2f30}, ...)
/usr/local/go/src/text/template/exec.go:870 +0x7a0
text/template.(*state).evalFunction(0x53fd29739bd0, {0x0?, 0x0?, 0x103955100?}, 0x53fd29e1dad8?, {0x103a83250, 0x53fd298b2f30}, {0x53fd2978ab40, 0x2, 0x2}, ...)
/usr/local/go/src/text/template/exec.go:676 +0x158
text/template.(*state).evalCommand(0x53fd29739bd0, {0x0?, 0x0?, 0x102a13794?}, 0x53fd29e1db78?, {0x103905ba0?, 0x103b79cc0?, 0x53fd29cde9c0?})
/usr/local/go/src/text/template/exec.go:563 +0x160
text/template.(*state).evalPipeline(0x53fd29739bd0, {0x0?, 0x0?, 0x102a3a650?}, 0x53fd2978f980)
/usr/local/go/src/text/template/exec.go:532 +0xf8
text/template.(*state).walk(0x53fd29739bd0, {0x0?, 0x0?, 0x30?}, {0x103a83370?, 0x53fd298b2f90})
/usr/local/go/src/text/template/exec.go:267 +0xb0
text/template.(*state).walk(0x53fd29739bd0, {0x0?, 0x0?, 0x53fd298b2db0?}, {0x103a832e0?, 0x53fd298b2f00?})
/usr/loc
ctr: failed to handle lowerdir option: template: :1:3: executing "" at <mount 1>: error calling mount: failed to resolve mount 1: mount type erofs cannot be directly viewed: not implemented