Enable to export layers from Additional Layer Store

[ktock]

Currently, layers aquired from additional layer store cannot be exported (e.g. podman save, podman push) as mentioned in #795 (comment).

# podman save ghcr.io/stargz-containers/postgres:13.1-esgz > /tmp/test.tar
Error: unable to save "ghcr.io/stargz-containers/postgres:13.1-esgz": error copying image to the remote destination: Error writing blob: error happened during read: Digest did not match, expected sha256:0167446b81cb115e90d668017ed409b449aee3a8fade44c7a96d36ebb0937ca5, got sha256:0a45da8cdff8388828644ad2052e91595733e558067b60b9f4bd290035964193

This is because the current additional layer store exposes only extracted view of layers. Tar is not reproducible so the runtime cannot reproduce the tar archive that has the same diff ID as the original.

This commit solves this issue by introducing a new API "blob" to the following location of the additional layer store. This file exposes the raw contents of that layer. When *(c/storage).layerStore.Diff is called, it acquires the diff contents from this blob file which has the same digest as the original layer.

<ALS root>/[<image-reference>/]<layer-digest>/
- blob

Testing

The following example shows how this patch works.

We can manually create an additional layers store using something like the following script:

#!/bin/bash

set -euo pipefail

ORG="${1}"
STORE="${2}"

if [ "${1}" == "--ref" ] ; then
    ORG="${2}"
    STORE="${3}/$(echo -n ${2} | base64)"
fi

OCI="$(mktemp -d)"
skopeo copy docker://${ORG} oci://${OCI}
cat ${OCI}/blobs/sha256/$(cat ${OCI}/index.json | jq -r '.manifests[0].digest' | sed 's/sha256://') \
    | jq -r '.layers[].digest' | while read DGST ; do
    ENCODED_DGST=$(echo -n ${DGST} | sed 's/sha256://')
    mkdir -p ${STORE}/${DGST}/diff
    tar -xf ${OCI}/blobs/sha256/${ENCODED_DGST} -C ${STORE}/${DGST}/diff
    cp ${OCI}/blobs/sha256/${ENCODED_DGST} ${STORE}/${DGST}/blob
    touch ${STORE}/${DGST}/use
    cat <<EOF > ${STORE}/${DGST}/info
{
  "compressed-diff-digest": "${DGST}",
  "compressed-size": $(stat --printf="%s" ${OCI}/blobs/sha256/${ENCODED_DGST}),
  "diff-digest": "sha256:$(cat ${OCI}/blobs/sha256/${ENCODED_DGST} | gunzip | sha256sum | sed -E 's/([^ ]*).*/\1/g')",
  "diff-size": $(cat ${OCI}/blobs/sha256/${ENCODED_DGST} | gunzip | wc -c),
  "compression": 2
}
EOF
done

For example, the following prepares additional layer store that contains ghcr.io/stargz-containers/python:3.9-esgz at /tmp/storagedemo.

create_store.sh ghcr.io/stargz-containers/python:3.9-esgz /tmp/storagedemo

At least the following configuration is needed to /etc/containers/storage.conf to enable addtional layer store.

cat <<EOF > /etc/containers/storage.conf
[storage]
driver = "overlay"
graphroot = "/var/lib/containers/storage"
runroot = "/run/containers/storage"

[storage.options]
additionallayerstores = ["/tmp/storagedemo"]
EOF

As shown in the following example, this image can aquired from the addtional layer store without pulling.
And this also can be exported to somewhere else because of blob API.

# Layers are used from Additional Layer Store with skipping pulling layers

$ podman pull ghcr.io/stargz-containers/python:3.9-esgz
Copying blob 11f19ef2588b skipped: already exists  
Copying blob 1320c6c8123e skipped: already exists  
Copying blob 58c06a9c95bc skipped: already exists  
...

# Image can run

$ podman run --rm -it ghcr.io/stargz-containers/python:3.9-esgz /bin/echo 'Hello, World!'
Hello, World!

# Image can be exported in the same way as normal images

$ podman save ghcr.io/stargz-containers/python:3.9-esgz > /tmp/test.tar
$ podman tag ghcr.io/stargz-containers/python:3.9-esgz ghcr.io/ktock/python:3.9-my
$ podman push ghcr.io/ktock/python:3.9-my

[ktock]

@giuseppe Thank you for the review. Fixed them all.

[giuseppe]

LGTM

[TomSweeneyRedHat]

Changes LGTM in general, but this appears to need a rebase.
It would be nice to get a @nalind or @vrothberg head nod too.

[ktock]

@TomSweeneyRedHat Thanks for the review. Added the comment and rebased.

[ktock]

Can we move this forward?

[rhatdan]

LGTM