cleanup orphan remote snapshots after restart #2303

Problem

When containerd-stargz-grpc crashes or is killed while an image is being removed from containerd, the snapshotter can miss the removal event. After restart, the local stargz snapshot metadata may still contain remote snapshots that are no longer referenced by containerd.

This leaves orphan snapshot data behind. In addition, because the process was killed, filesystem cache directories such as fscache and httpcache may also remain on disk without any in-memory layer/blob state that can clean them up later.

Symptoms

A typical reproduction is:

1. Pull an image with stargz snapshotter.
2. Kill containerd-stargz-grpc with SIGKILL.
3. Remove the image from containerd while the snapshotter is down.
4. Restart containerd-stargz-grpc.

After restart, the local snapshotter can keep stale remote snapshot metadata and disk data. If FUSE mounts from the killed process are left behind, cleanup may also hit errors such as:

transport endpoint is not connected

Even when snapshot directories are removed successfully, old fscache and httpcache directories can remain because the new process no longer has the old in-memory layer/blob references.

Solution

On startup, the snapshotter should reconcile local remote snapshot metadata with containerd state:

• Force-unmount stale snapshot mounts before recovery cleanup.
• Clean up filesystem cache directories left by the previous process before restoring remote snapshots.
• Scan local metadata for remote snapshots.
• Connect to containerd and collect currently referenced snapshot keys from containers, content GC references, and leases across namespaces.
• Keep referenced snapshots and their parent chains.
• Treat the remaining local remote snapshots as orphaned.
• Remove orphan snapshot metadata in child-first order.
• Run snapshotter cleanup to remove the corresponding disk directories.