Merge pull request #189 from ktock/docs-esgz

Fix docs to focus more on eStargz

Author: AkihiroSuda

README.md

@@ -5,19 +5,41 @@
 
 Read also introductory blog: [Startup Containers in Lightning Speed with Lazy Image Distribution on Containerd](https://medium.com/nttlabs/startup-containers-in-lightning-speed-with-lazy-image-distribution-on-containerd-243d94522361)
 
-Pulling image is one of the time-consuming steps in the container lifecycle. Research shows that time to take for pull operation accounts for 76% of container startup time[[FAST '16]](https://www.usenix.org/node/194431). *Stargz Snapshotter* is an implementation of snapshotter which aims to solve this problem by *lazy pulling* leveraging [stargz image format by CRFS](https://github.com/google/crfs). The following histogram is the benchmarking result for startup time of several containers measured on Github Actions, using Docker Hub as a registry.
+Pulling image is one of the time-consuming steps in the container lifecycle.
+Research shows that time to take for pull operation accounts for 76% of container startup time[[FAST '16]](https://www.usenix.org/node/194431).
+*Stargz Snapshotter* is an implementation of snapshotter which aims to solve this problem by *lazy pulling*.
+*Lazy pulling* here means a container can run without waiting for the pull completion of the image and necessary chunks of the image are fetched *on-demand*.
 
-<img src="docs/images/benchmarking-result-288c338.png" width="600" alt="The benchmarking result on 288c338">
+[*eStargz*](/docs/stargz-estargz.md) is a lazily-pullable image format proposed by this project.
+This is compatible to [OCI](https://github.com/opencontainers/image-spec/)/[Docker](https://github.com/moby/moby/blob/master/image/spec/v1.2.md) images so this can be pushed to standard container registries (e.g. ghcr.io) as well as this is *still runnable* even on eStargz-agnostic runtimes including Docker.
+eStargz format is based on [stargz image format by CRFS](https://github.com/google/crfs) but comes with additional features like runtime optimization and content verification.
 
-`legacy` shows the startup performance when we use containerd's default snapshotter (`overlayfs`) with images copied from `docker.io/library` without optimization. For this configuration, containerd pulls entire image contents and `pull` operation takes accordingly. When we use stargz snapshotter with `stargz` images we are seeing performance improvement on the `pull` operation because containerd can start the container before the entire image contents locally available and fetches each file on-demand. But at the same time, we see the performance drawback for `run` operation because each access to files takes extra time for fetching them from the registry. When we use the further optimized version of images(`estargz`) we can mitigate the performance drawback observed in `stargz` images. This is because [stargz snapshotter prefetches and caches some files which will be most likely accessed during container workload](./docs/stargz-estargz.md). Stargz snapshotter waits for the first container creation until the prefetch completes so `create` sometimes takes longer than other types of image. But this wait only occurs just after the pull completion until the prefetch completion and it's shorter than waiting for downloading all files of all layers.
+The following histogram is the benchmarking result for startup time of several containers measured on Github Actions, using GitHub Container Registry.
 
-The above histogram is [the benchmarking result on the commit `288c338`](https://github.com/containerd/stargz-snapshotter/actions/runs/50632674). We are constantly measuring the performance of this snapshotter so you can get the latest one through the badge shown top of this doc. Please note that we sometimes see dispersion among the results because of the NW condition on the internet and the location of the instance in the Github Actions, etc. Our benchmarking method is based on [HelloBench](https://github.com/Tintri/hello-bench).
+<img src="docs/images/benchmarking-result-ecdb227.png" width="600" alt="The benchmarking result on ecdb227">
+
+`legacy` shows the startup performance when we use containerd's default snapshotter (`overlayfs`) with images copied from `docker.io/library` without optimization.
+For this configuration, containerd pulls entire image contents and `pull` operation takes accordingly.
+When we use stargz snapshotter with eStargz-converted images but without any optimization (`estargz-noopt`) we are seeing performance improvement on the `pull` operation because containerd can start the container without waiting for the `pull` completion and fetch necessary chunks of the image on-demand.
+But at the same time, we see the performance drawback for `run` operation because each access to files takes extra time for fetching them from the registry.
+When we use [eStargz with optimization](/docs/ctr-remote.md) (`estargz`), we can mitigate the performance drawback observed in `estargz-noopt` images.
+This is because [stargz snapshotter prefetches and caches *likely accessed files* during running the container](/docs/stargz-estargz.md).
+On the first container creation, stargz snapshotter waits for the prefetch completion so `create` sometimes takes longer than other types of image.
+But it's still shorter than waiting for downloading all files of all layers.
+
+The above histogram is [the benchmarking result on the commit `ecdb227`](https://github.com/containerd/stargz-snapshotter/actions/runs/398606060).
+We are constantly measuring the performance of this snapshotter so you can get the latest one through the badge shown top of this doc.
+Please note that we sometimes see dispersion among the results because of the NW condition on the internet and the location of the instance in the Github Actions, etc.
+Our benchmarking method is based on [HelloBench](https://github.com/Tintri/hello-bench).
 
 Stargz Snapshotter is a **non-core** sub-project of containerd.
 
 ## Quick Start with Kubernetes
 
-For using stargz snapshotter on kubernetes nodes, you need the following configuration to containerd as well as run stargz snapshotter daemon on the node. We assume that you are using containerd newer than at least [commit `d8506bf`](https://github.com/containerd/containerd/commit/d8506bfd7b407dcb346149bcec3ed3c19244e3f1) as a CRI runtime.
+- For more details about stargz snapshotter plugin and its configuration, refer to [Containerd Stargz Snapshotter Plugin Overview](/docs/overview.md).
+
+For using stargz snapshotter on kubernetes nodes, you need the following configuration to containerd as well as run stargz snapshotter daemon on the node.
+We assume that you are using containerd (> v1.4.2) as a CRI runtime.
 
 ```toml
 version = 2
@@ -36,16 +58,18 @@ version = 2
   disable_snapshot_annotations = false
 ```
 
-**Note that `disable_snapshot_annotations = false` is required since containerd > 1.4.2**
+**Note that `disable_snapshot_annotations = false` is required since containerd > v1.4.2**
 
-This repo contains [a Dockerfile as a KinD node image](./Dockerfile) which includes the above configuration. You can use it with [KinD](https://github.com/kubernetes-sigs/kind) like the following,
+This repo contains [a Dockerfile as a KinD node image](/Dockerfile) which includes the above configuration.
+You can use it with [KinD](https://github.com/kubernetes-sigs/kind) like the following,
 
 ```console
 $ docker build -t stargz-kind-node https://github.com/containerd/stargz-snapshotter.git
 $ kind create cluster --name stargz-demo --image stargz-kind-node
 ```
 
-Then you can create stargz pods on the cluster. In this example, we create a stargz-converted Node.js pod (`ghcr.io/stargz-containers/node:13.13-esgz`) as a demo.
+Then you can create eStargz pods on the cluster.
+In this example, we create a stargz-converted Node.js pod (`ghcr.io/stargz-containers/node:13.13-esgz`) as a demo.
 
 ```yaml
 apiVersion: v1
@@ -77,18 +101,24 @@ $ curl 127.0.0.1:8080
 Hello World!
 ```
 
-Stargz snapshotter also supports further configuration including private registry authentication, mirror registries, etc.
-For more details, refer to the [overview doc](./docs/overview.md).
+Stargz snapshotter also supports [further configuration](/docs/overview.md) including private registry authentication, mirror registries, etc.
 
-## Creating stargz images and further optimization
+## Creating eStargz images with optimization
 
-For more examples and details about the image converter `ctr-remote`, refer to [this doc](./docs/ctr-remote.md).
+- For more examples and details about the image converter `ctr-remote`, refer to [Optimize Images with `ctr-remote image optimize`](/docs/ctr-remote.md).
+- For more details about eStargz format, refer to [eStargz: Standard-Compatible Extensions to Tar.gz Layers for Lazy Pulling Container Images](/docs/stargz-estargz.md)
 
-For lazy pulling images, you need to prepare stargz images first. You can use [CRFS-official `stargzify`](https://github.com/google/crfs/tree/master/stargz/stargzify) command or our `ctr-remote` command which has further optimization functionality. You can also try our pre-converted images listed in [this doc](./docs/pre-converted-images.md). For more details about stargz and the optimization, refer to [this doc](./docs/stargz-estargz.md)
+For lazy pulling images, you need to prepare eStargz images first.
+You can use [`ctr-remote`](/docs/ctr-remote.md) command for do this.
+You can also try our pre-converted images listed in [Trying pre-converted images](/docs/pre-converted-images.md).
 
-In this section, we introduce `ctr-remote` command for converting images into stargz with further optimization for the performance of reading files. On-demand lazy pulling improves the performance of pull but it has runtime performance penalty because reading files induce remotely downloading contents. For solving this, `ctr-remote` has *workload-based* optimization for images. This section shows how to convert and pull an image lazily using `ctr-remote` command and briefly describes workload-based optimization.
+In this section, we introduce `ctr-remote` command for converting images into eStargz with optimization for reading files.
+As shown in the above benchmarking result, on-demand lazy pulling improves the performance of pull but causes runtime performance penalty because reading files induce remotely downloading contents.
+For solving this, `ctr-remote` has *workload-based* optimization for images.
 
-First, prepare the demo environment with the following command (put this repo on `${GOPATH}/src/github.com/containerd/stargz-snapshotter`).
+For trying the examples described in this section, you can also use the docker-compose-based demo environment.
+You can setup this environment as the following commands (put this repo on `${GOPATH}/src/github.com/containerd/stargz-snapshotter`).
+*Note that this runs privileged containers on your host.*
 
 ```console
 $ cd ${GOPATH}/src/github.com/containerd/stargz-snapshotter/script/demo
@@ -98,15 +128,22 @@ $ docker exec -it containerd_demo /bin/bash
 (inside container) # ./script/demo/run.sh
 ```
 
-Generally, container images are built with purpose and the *workloads* are defined in the Dockerfile with some parameters including entrypoint command, environment variables and user. By default, `ctr-remote` optimizes the performance of reading files that are most likely accessed in the workload defined in the Dockerfile. You can also specify the custom workload using options.
+Generally, container images are built with purpose and the *workloads* are defined in the Dockerfile with some parameters (e.g. entrypoint, envvars and user).
+By default, `ctr-remote` optimizes the performance of reading files that are most likely accessed in the workload defined in the Dockerfile.
+[You can also specify the custom workload using options if needed](/docs/ctr-remote.md).
 
-The following example converts the legacy `library/ubuntu:18.04` image into stargz. The command also optimizes the image for the workload of executing `ls` on `/bin/bash`. The thing actually done is it runs the specified workload in a sandboxed environment and profiles all file accesses. Then these files are marked in the image as likely accessed also in production. Then it pushes the converted image to the local registry (`registry2:5000`). The converted image is still __docker-compatible__ so you can run it with other runtimes (e.g. Docker).
+The following example converts the legacy `library/ubuntu:18.04` image into eStargz.
+The command also optimizes the image for the workload of executing `ls` on `/bin/bash`.
+The thing actually done is it runs the specified workload in a temporary container and profiles all file accesses with marking them as *likely accessed* also during runtime.
+Then it pushes the converted image to the local container registry (`registry2:5000`).
+The converted image is still **docker-compatible** so you can run it with eStargz-agnostic runtimes (e.g. Docker).
 
 ```console
 # ctr-remote image optimize --plain-http --entrypoint='[ "/bin/bash", "-c" ]' --args='[ "ls" ]' ubuntu:18.04 http://registry2:5000/ubuntu:18.04
 ```
 
-Finally, the following commands pull the stargz image lazily. Stargz snapshotter prefetches files that are most likely accessed in the optimized workload, which hopefully increases the cache hit rate for that workload and mitigates runtime overheads as shown in the benchmarking result shown top of this doc.
+Finally, the following commands pull the eStargz image lazily.
+Stargz snapshotter prefetches files that are most likely accessed in the optimized workload, which hopefully increases the cache hit rate for that workload and mitigates runtime overheads as shown in the benchmarking result shown top of this doc.
 
 ```console
 # ctr-remote images rpull --plain-http registry2:5000/ubuntu:18.04

docs/ctr-remote.md

@@ -13,7 +13,7 @@ This optimization is done by baking the information about files that are likely
 On runtime, Stargz Snapshotter prefetches these prioritized files before mounting the layer for making sure these files are locally accessible.
 This can avoid downloading chunks on every file read and mitigate the runtime performance drawbacks.
 
-For more details about eStargz and its optimization, refer also to the [doc about image formats](/docs/stargz-estargz.md).
+For more details about eStargz and its optimization, refer also to [eStargz: Standard-Compatible Extensions to Tar.gz Layers for Lazy Pulling Container Images](/docs/stargz-estargz.md).
 
 ## Requirements
 

docs/images/benchmarking-result-288c338.png

@@ -1,4 +1,4 @@
-# Containerd Stargz Snapshotter Overview
+# Containerd Stargz Snapshotter Plugin Overview
 
 __Before get through this overview document, we recommend you to read [README](README.md).__
 
@@ -15,8 +15,10 @@ The actual image contents can be fetched *lazily* so runtimes can startup contai
 We call these remotely mounted layers as *remote snapshots*.
 
 *Stargz Snapshotter* is a remote snapshotter plugin implementation which supports standard compatible remote snapshots functionality.
-This leverages [*stargz* image format by Google](https://github.com/google/crfs) which enables lazy distribution but is backwards-compatible with container standards.
-When you run a container image and it is formatted by stargz, stargz snapshotter prepares container's rootfs layers as remote snapshots by mounting layers from [OCI](https://github.com/opencontainers/distribution-spec)/[Docker](https://docs.docker.com/registry/spec/api/) standard registries to the node, instead of pulling the entire image contents.
+This snapshotter leverages [eStargz](/docs/stargz-estargz.md) image, which is lazily-pullable and still standard-compatible.
+Because of this conpatibility, eStargz image can be pushed to and lazily pulled from [OCI](https://github.com/opencontainers/distribution-spec)/[Docker](https://docs.docker.com/registry/spec/api/) registries (e.g. ghcr.io).
+Furthermore, images can run even on eStargz-agnostic runtimes (e.g. Docker).
+When you run a container image and it is formatted by eStargz, stargz snapshotter prepares container's rootfs layers as remote snapshots by mounting layers from the registry to the node, instead of pulling the entire image contents.
 
 This document gives you a high-level overview of stargz snapshotter.
 
@@ -26,11 +28,11 @@ This document gives you a high-level overview of stargz snapshotter.
 
 Stargz snapshotter is implemented as a [proxy plugin](https://github.com/containerd/containerd/blob/04985039cede6aafbb7dfb3206c9c4d04e2f924d/PLUGINS.md#proxy-plugins) daemon (`containerd-stargz-grpc`) for containerd.
 When containerd starts a container, it queries the rootfs snapshots to stargz snapshotter daemon through an unix socket.
-This snapshotter remotely mounts queried stargz layers from registries to the node and provides these mount points as remote snapshots to containerd.
+This snapshotter remotely mounts queried eStargz layers from registries to the node and provides these mount points as remote snapshots to containerd.
 
 Containerd recognizes this plugin through an unix socket specified in the configuration file (e.g. `/etc/containerd/config.toml`).
 Stargz snapshotter can also be used through Kubernetes CRI by specifying the snapshotter name in the CRI plugin configuration.
-We assume that you are using containerd newer than at least [commit `d8506bf`](https://github.com/containerd/containerd/commit/d8506bfd7b407dcb346149bcec3ed3c19244e3f1)
+We assume that you are using containerd (> v1.4.2).
 
 ```toml
 version = 2
@@ -53,7 +55,7 @@ This repo contains [a Dockerfile as a KinD node image](/Dockerfile) which includ
 
 ## State directory
 
-Stargz snapshotter mounts stargz layers from registries to the node using FUSE.
+Stargz snapshotter mounts eStargz layers from registries to the node using FUSE.
 The all files metadata in the image are preserved on the filesystem and files contents are fetched from registries on demand.
 
 At the root of the filesystem, there is a *state directory* (`/.stargz-snapshotter`) for status monitoring for the filesystem.