Containerd snapshotter creates malformed images with empty blob references for base layers

[jjbustamante]

Summary

When using Docker with the containerd snapshotter storage driver, imgutil creates malformed images where the manifest layer digests don't match the config's diff_ids. The first several base image layers are replaced with empty blob references in the manifest, causing image integrity validation tools (like skopeo) to fail.

Environment

• imgutil version: v0.0.0-20250814164739-4b1c8875ba7e (introduced in buildpacks/pack PR #2427)
• Docker version: 29.0.2
• Storage driver: overlayfs with driver-type: io.containerd.snapshotter.v1
• OS: Linux

Reproduction Steps

1. Enable containerd snapshotter in Docker:

   # Edit /etc/docker/daemon.json
   {
       "features": {
           "containerd-snapshotter": true
       }
   }
   
   # Restart Docker
   sudo systemctl restart docker
   
   # Verify
   docker info | grep -A 2 "Storage Driver"
   # Should show: overlayfs with driver-type: io.containerd.snapshotter.v1

2. Create an image using imgutil (via pack):

   git clone https://github.com/buildpacks/samples
   cd samples
   pack builder create test-builder:latest --config builders/noble/builder.toml

3. Attempt to validate/copy the image:

   skopeo copy docker-daemon:test-builder:latest oci:/tmp/test

Expected Behavior

The image should be valid with manifest layer digests matching the config's diff_ids.

Actual Behavior

FATA[0001] determining manifest MIME type for docker-daemon:test-builder:latest: 
Layer tarfile "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" 
used for two different DiffID values

Root Cause Analysis

Examining the saved image reveals:

Manifest layers (first 6 layers):

{
  "digest": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "size": 0
}

All 6 are the same empty blob hash.

Config diff_ids (first 6 layers):

sha256:a8346d259389bc6221b4f3c61bad4e48087c5b82308e8f53ce703cfc8333c7b3
sha256:4ea3b5141be57efe60a721cf8e9e5324602348abb0e51413ffe04421de0927cf
sha256:126284c049b193fca15f137c8d7267bbcf26f618ae200fcc0227fbc16dee8220
sha256:9979cfa0335fd5553103aeb045dbb7cf4dc4e51eace5cda6112c1a73045191de
sha256:e1f7fa38e4677bd5fcc80f14136e7a432d280fceeb3b1da47ec02e186d420ee1
sha256:f2b4f09b9169645689f604904c1f49089fc1b4b26eca1da3d8575c488a09b944

These are the correct layer hashes from the base image.

The Problem: The manifest references empty blobs for base image layers, while the config correctly references the actual layer diff_ids. This creates an invalid image that fails OCI spec validation.

Suspected Cause

The issue appears related to the "fast path" optimization introduced in recent imgutil updates for containerd storage driver support. The optimization likely "omits base layers" incorrectly by replacing them with empty blob references in the manifest instead of properly referencing the existing base image layers.

Relevant code change from buildpacks/pack#2427:

"Always try the fast path first (omitting base layers) regardless of storage type."

Additional Context

• Works correctly with overlay2 storage driver (traditional Docker storage)
• Fails with containerd snapshotter (overlayfs with io.containerd.snapshotter.v1)
• First reported in [REGRESSION] produced builder image is incorrect pack#2490
• Regression from imgutil version in pack v0.38.2

Impact

This affects:

• Any pack builder creation with containerd snapshotter
• Any tool that validates OCI image integrity (skopeo, crane, etc.)
• Image portability and distribution

Workaround

Disable containerd snapshotter and use overlay2:

{
    "features": {
        "containerd-snapshotter": false
    }
}

---

Related: buildpacks/pack#2490

[jjbustamante]

Root Cause Analysis - Issue Resolved

Thank you for the detailed report! I've completed a thorough root cause analysis. The good news is this issue has already been resolved via PR #297 which reverted the problematic changes.

Root Cause

The issue was introduced by optimization commits attempting a "fast path" for image saves by omitting base layers:

• Commit 1924a0b: "containerd: optimizations"
• Commit 6969191: "Always try the fast path first (omitting base layers) regardless of storage type"

How the Fast Path Works

When saving an image to the Docker daemon, imgutil creates a tarball with blank placeholders (zero-byte tar entries) for base image layers that already exist in the daemon:

<image>.tar
├── <config-hash>.json
├── manifest.json
├── /<new-layer>.tar      # Full layer data
├── blank_0                # Empty placeholder for base layer
├── blank_1                # Empty placeholder for base layer
└── ...

Why It Failed with Containerd Snapshotter

The fast path makes an incorrect assumption about how storage drivers handle blank entries:

overlay2 behavior:

1. Docker receives tarball with blank_0, blank_1 placeholders
2. Docker recognizes these as references to existing layers
3. Docker creates correct manifest references to existing blobs
4. ✅ Result: Valid image with proper manifest/config correspondence

containerd snapshotter behavior:

1. Docker/containerd receives tarball with blank placeholders
2. Containerd strictly follows OCI spec, imports blank files as literal empty blobs
3. Containerd creates manifest with empty blob digests (sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855)
4. Config retains correct diff_ids from original base image
5. ❌ Result: Invalid image where manifest layer digests ≠ config diff_ids

This is exactly the error you encountered:

Layer tarfile "blobs/sha256/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" 
used for two different DiffID values

The Fix (Already Applied)

PR #297 reverted all the problematic commits. The current code in main (local/store.go:92-98) correctly:

canOmitBaseLayers := !usesContainerdStorage(s.dockerClient)
if canOmitBaseLayers {
    // Only try fast path with overlay2
    inspect, err = s.doSave(img, withName)
}
if !canOmitBaseLayers || err != nil {
    // Always do full save for containerd
    if err = img.ensureLayers(); err != nil {
        return "", err
    }
    inspect, err = s.doSave(img, withName)
}

This:

• ✅ Detects containerd storage via usesContainerdStorage() check
• ✅ Skips fast path for containerd, always does full save with all layer data
• ✅ Retains fast path optimization for overlay2 (performance benefit)
• ✅ Creates valid OCI-compliant images for both storage drivers

Timeline

• Aug 8, 2025: Commit 1924a0b introduced containerd optimizations
• Aug 13, 2025: Commit 6969191 enabled fast path for all storage types
• Issue reported: Validation failures with containerd snapshotter
• PR revert containerd changes #297 merged: All containerd optimization attempts reverted
• Current state: ✅ Fixed - full saves for containerd, fast path for overlay2

Verification

The fix has been in main since commit 9db16db. Any release built from current main will work correctly with containerd snapshotter.

Closing this issue as resolved. If you encounter this problem with the latest code from main, please reopen with details.