Enable to run containers on IPFS #496

ktock · 2021-10-25T06:21:34Z

See https://github.com/containerd/stargz-snapshotter/blob/main/docs/ipfs.md for the latest spec.

This commit enables to run of containers on IPFS.

OCI image is extended in an OCI-compatible way. Stargz Snapshotter mounts the image from IPFS to the container's rootfs with lazy pulling support.

The image must have the following OCI-compatible extension. This constructs a CID-based DAG of blobs in an OCI image.

Each descriptor in an image must have the following annotation
- key: containerd.io/snapshot/remote/ipfs/cid
- value: CID of the blob that the descriptor points to

This commit adds ipfs library which includes the basic functionality to make containerd aware of containers on IPFS. These components are eStargz-agnostic and can be used for running non-eStargz images without lazy pulling (e.g. on overlayfs snapshotter).

ipfs.IndexConvertFunc provides containerd's converter.ConvertFunc implementation which converts an image to the IPFS-enabled image format as described above. This also adds the image contents to IPFS.
ipfs.NewResolver provides containerd's remote.Resovler implementation for IPFS. If a descriptor contains the CID annotation, it fetches the pointed content from IPFS.

The following component enables lazy pulling of containers from IPFS.

(fs/remote/ipfs).Reader provides the way to read a range of a file on IPFS. This enables stargz snapshotter to mount the container's rootfs from IPFS with lazy pulling.

Examples

Storing image to IPFS and run with stargz snapshotter:

# ipfs daemon
# ctr-remote i pull ghcr.io/stargz-containers/python:3.9-org
# ctr-remote i ipfs-add ghcr.io/stargz-containers/python:3.9-org test
INFO[0098] Pushed                                        CID=QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV
# ipfs cat QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV | jq
{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "digest": "sha256:d8e862a13071692edbf4cffc3954591ca98954571615a0698a7ed59a09dc04df",
  "size": 342,
  "annotations": {
    "containerd.io/snapshot/remote/ipfs/cid": "QmSUi34RNpoxY4zqZFGZnCQcH6odXBRHT6VTGAduANwTj6"
  }
}
# # clear containerd cache here
# time ( ctr-remote i rpull --ipfs QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV && \
  ctr-remote run --snapshotter=stargz --rm -t QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV foo python -c 'print("Hello, World!")' )
fetching sha256:6fd287cf... application/vnd.oci.image.index.v1+json
fetching sha256:5c13568a... application/vnd.oci.image.manifest.v1+json
fetching sha256:236b4bd7... application/vnd.oci.image.config.v1+json
Hello, World!

real	0m1.609s
user	0m0.047s
sys	0m0.031s

The container can also run with overlayfs snapshotter without lazy pulling but it's slower than stargz snapshotter:

# time ( ctr-remote i rpull --snapshotter=overlayfs --ipfs QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV && \
  ctr-remote run --snapshotter=overlayfs --rm -t QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV foo python -c 'print("Hello, World!")' )
fetching sha256:d8e862a1... application/vnd.oci.image.index.v1+json
fetching sha256:b8df0fee... application/vnd.oci.image.manifest.v1+json
fetching sha256:236b4bd7... application/vnd.oci.image.config.v1+json
fetching sha256:94584b60... application/vnd.oci.image.layer.v1.tar+gzip
fetching sha256:cecf6f1a... application/vnd.oci.image.layer.v1.tar+gzip
fetching sha256:cfd1c7a0... application/vnd.oci.image.layer.v1.tar+gzip
fetching sha256:9c3d8238... application/vnd.oci.image.layer.v1.tar+gzip
fetching sha256:cce4a0fe... application/vnd.oci.image.layer.v1.tar+gzip
fetching sha256:3467b44a... application/vnd.oci.image.layer.v1.tar+gzip
fetching sha256:8ff4c537... application/vnd.oci.image.layer.v1.tar+gzip
fetching sha256:b60d5d60... application/vnd.oci.image.layer.v1.tar+gzip
fetching sha256:68277180... application/vnd.oci.image.layer.v1.tar+gzip
Hello, World!

real	0m11.655s
user	0m0.571s
sys	0m0.309s

AkihiroSuda · 2021-10-25T06:45:41Z

How can we detect whether the snapshotter is capable of handling IPFS?
Can we have some plugin annotation (Exports) that can be inspect with ctr plugins ls -v?

ktock · 2021-10-25T06:58:16Z

IPFS is handled by our special remote.Resolver but not by snapshotters. Containerd fetches layers from IPFS using that resolver and unpacks to snapshotters so any snapshotters can run images from IPFS in theory.
(lazy-pulling is only supported by stargz snapshotter, though)

cmd/ctr-remote/commands/ipfs-add.go

docs/ipfs.md

AkihiroSuda · 2021-10-26T02:42:41Z

Do we really need our own new annotation key?
Can we just put ipfs://<case-insensitive-cid-v1> in the urls field of OCI Descriptor?

https://docs.ipfs.io/how-to/address-ipfs-on-web/#native-urls

https://github.com/opencontainers/image-spec/blob/main/descriptor.md#properties

docs/ipfs.md

ktock · 2021-10-26T02:58:07Z

ipfs:// in the urls field of OCI Descriptor?

SGTM. I'll try it later.

ktock · 2021-10-26T12:16:22Z

ipfs/resolver.go

+// Resolve resolves the provided ref for IPFS. ref must be a CID.
+// TODO: Allow specifying `/ip[f|n]s`-prefixed path. This requires to modify `reference` pkg of containerd because
+//       it's incompatbile to the current reference specification used by containerd.
+func (r *resolver) Resolve(ctx context.Context, ref string) (name string, desc ocispec.Descriptor, err error) {


Our remote.Resolver only supports an IPFS CID as an image reference but doesn't supprot /ipfs-prefixed path as of now.
This is because containerd cannot handle IPFS path which is incompatible with the current image reference specification defined in the reference package of containerd.
We need to modify containerd implementation after deciding the image reference format for IPFS (containerd/nerdctl#465 (comment)).

AkihiroSuda · 2021-10-26T12:22:11Z

ipfs/converter.go

+}
+
+// ConvertDockerMediaTypeToOCI converts a media type string
+func ConvertDockerMediaTypeToOCI(mt string) string {


dupe: https://github.com/containerd/containerd/blob/0d0fb6858971e5dd92cb044a744979ccb1e9635a/images/converter/default.go#L414

AkihiroSuda · 2021-10-26T12:22:24Z

ipfs/converter.go

+}
+
+// ClearGCLabels clears GC labels for the given digest.
+func ClearGCLabels(labels map[string]string, dgst digest.Digest) {


dupe: https://github.com/containerd/containerd/blob/0d0fb6858971e5dd92cb044a744979ccb1e9635a/images/converter/default.go#L436

AkihiroSuda · 2021-10-26T12:24:13Z

script/integration/containerd/entrypoint.sh

+ctr-remote i rpull --snapshotter=overlayfs --ipfs "${CID}"
+ctr-remote run --rm --snapshotter=overlayfs "${CID}" test tar -c /usr | tar -xC "${USR_NORMALSN_IPFS}"
+
+echo "Diffing bitween two root filesystems(normal vs stargz snapshotter, IPFS rootfs)"


typo: bitween

Signed-off-by: Kohei Tokunaga <[email protected]>

AkihiroSuda · 2021-10-26T12:29:24Z

ipfs/converter.go

+	if err != nil {
+		return nil, err
+	}
+	dstDesc, err := IndexConvertFunc(api, layerConvert, platformMC)(ctx, client.ContentStore(), srcImg.Target)


Can we use the existing converter pkg with a simple custom ConvertFunc that just adds IPFS urls?

https://github.com/containerd/containerd/blob/0d0fb6858971e5dd92cb044a744979ccb1e9635a/images/converter/converter.go#L40
https://github.com/containerd/containerd/blob/0d0fb6858971e5dd92cb044a744979ccb1e9635a/images/converter/converter.go#L66

We need to modify descriptors of all blobs not only layers so we cannot use WithLayerConvertFunc here.
converter.Convert returns *images.Image but containerd drops urls field of OCI descriptor (*images.Image.Target) so we cannot use it for IPFS.

We need to modify converter.Converter to return the OCI descriptor of the converted image index as-is and to allow injecting a callback hook to modify descriptors.

ktock · 2021-10-26T12:45:40Z

This is highly experimental so mounting IPFS layers should be an opt-in feature of stargz snapshotter. I'll add that configuration option.

AkihiroSuda · 2021-10-26T13:16:23Z

fs/remote/resolver.go

 	var f fetcher
 	var size int64
-	if ipfs.Supported(desc) {
+	if r.blobConfig.IPFS && ipfs.Supported(desc) {


The blobConfig.IPFS value should be printed to debug log

Signed-off-by: Kohei Tokunaga <[email protected]>

AkihiroSuda · 2021-10-27T05:38:52Z

Merging, please propose your pkg/converter changes to the upstream.
IPFS resolver can be a separate PR for ease of reviewing.

fahedouch · 2021-10-27T13:05:00Z

@ktock The IPFS URL protocol seems to support only the CID v1 , may be we should mention this somewhere ; in the resolver comment for example

stargz-snapshotter/ipfs/resolver.go

Line 62 in 221ec42

    
           func (r *resolver) Resolve(ctx context.Context, ref string) (name string, desc ocispec.Descriptor, err error) {

ktock · 2021-10-27T13:23:07Z

Thanks. I'll add the description about CID version in ipfs.md.

ref passed to Resolver isn't limited to CID v1 though. (see containerd/nerdctl#465)

ktock marked this pull request as draft October 25, 2021 06:34

AkihiroSuda mentioned this pull request Oct 25, 2021

IPFS (nerdctl run /ipfs/QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV) containerd/nerdctl#465

Closed

ktock force-pushed the ipfs branch 2 times, most recently from 53fff07 to 880a4cd Compare October 25, 2021 08:51

ktock marked this pull request as ready for review October 25, 2021 09:16

ktock requested a review from AkihiroSuda October 25, 2021 09:16

AkihiroSuda reviewed Oct 25, 2021

View reviewed changes

cmd/ctr-remote/commands/ipfs-add.go Outdated Show resolved Hide resolved

AkihiroSuda reviewed Oct 25, 2021

View reviewed changes

cmd/ctr-remote/commands/ipfs-add.go Outdated Show resolved Hide resolved

AkihiroSuda reviewed Oct 25, 2021

View reviewed changes

docs/ipfs.md Show resolved Hide resolved

AkihiroSuda reviewed Oct 25, 2021

View reviewed changes

docs/ipfs.md Outdated Show resolved Hide resolved

ktock force-pushed the ipfs branch from 880a4cd to b7b7ab7 Compare October 26, 2021 02:08

ktock commented Oct 26, 2021

View reviewed changes

docs/ipfs.md Outdated Show resolved Hide resolved

ktock force-pushed the ipfs branch from b7b7ab7 to 0b2d030 Compare October 26, 2021 02:55

ktock marked this pull request as draft October 26, 2021 02:58

ktock force-pushed the ipfs branch from 0b2d030 to 1122d00 Compare October 26, 2021 12:15

ktock commented Oct 26, 2021

View reviewed changes

ktock force-pushed the ipfs branch 2 times, most recently from a3fa793 to 0158afa Compare October 26, 2021 12:19

AkihiroSuda reviewed Oct 26, 2021

View reviewed changes

Enable to run containers on IPFS

99edafb

Signed-off-by: Kohei Tokunaga <[email protected]>

ktock force-pushed the ipfs branch from 0158afa to 99edafb Compare October 26, 2021 12:27

AkihiroSuda reviewed Oct 26, 2021

View reviewed changes

AkihiroSuda approved these changes Oct 26, 2021

View reviewed changes

ktock marked this pull request as ready for review October 26, 2021 12:41

ktock marked this pull request as draft October 26, 2021 12:44

ktock marked this pull request as ready for review October 26, 2021 12:55

ktock force-pushed the ipfs branch from 7c79cce to 14bf058 Compare October 26, 2021 13:04

AkihiroSuda reviewed Oct 26, 2021

View reviewed changes

Make IPFS feature opt-in

162c9c7

Signed-off-by: Kohei Tokunaga <[email protected]>

ktock force-pushed the ipfs branch from 14bf058 to 162c9c7 Compare October 26, 2021 13:31

AkihiroSuda approved these changes Oct 26, 2021

View reviewed changes

AkihiroSuda merged commit 6a57853 into containerd:main Oct 27, 2021

ktock deleted the ipfs branch October 27, 2021 08:42

This was referenced Oct 27, 2021

docs: explicitly describe CID version in OCI descriptor for IPFS #502

Merged

Support distributing OCI images on IPFS containerd/containerd#6175

Closed

csweichel mentioned this pull request Feb 14, 2022

Speed up image pulls using IPFS gitpod-io/gitpod#8186

Closed

Enable to run containers on IPFS #496

Enable to run containers on IPFS #496

Uh oh!

Conversation

ktock commented Oct 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

See https://github.com/containerd/stargz-snapshotter/blob/main/docs/ipfs.md for the latest spec.

Examples

Uh oh!

AkihiroSuda commented Oct 25, 2021

Uh oh!

ktock commented Oct 25, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AkihiroSuda commented Oct 26, 2021

Uh oh!

Uh oh!

ktock commented Oct 26, 2021

Uh oh!

ktock Oct 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ktock commented Oct 26, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda commented Oct 27, 2021

Uh oh!

fahedouch commented Oct 27, 2021

Uh oh!

ktock commented Oct 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ktock commented Oct 25, 2021 •

edited

Loading

ktock Oct 26, 2021 •

edited

Loading