init

NOTE-containerd-referrers.md

@@ -0,0 +1,60 @@
+# containerd-snapshotter: pull-through mirror fails with "failed to decode referrers index"
+
+## Summary
+
+When Docker Engine uses the containerd image store (`"features": {"containerd-snapshotter": true}`)
+and a pull-through registry mirror (Docker Distribution `registry:2`), image pulls fail with:
+
+```
+failed to decode referrers index: invalid character '<' looking for beginning of value
+```
+
+or:
+
+```
+failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:...: not found
+```
+
+## Root cause
+
+After pulling an image through the mirror, containerd makes a **referrers API request**
+(OCI Distribution Spec 1.1) directly to the **upstream registry** (e.g. `registry-1.docker.io`),
+bypassing the mirror entirely. The upstream sometimes returns an HTML error page instead of
+a valid JSON response for the referrers endpoint, causing containerd to fail the entire pull.
+
+Key observations:
+- The image layers and manifests pull successfully through the mirror
+- The referrers request goes **directly to upstream**, not through the mirror
+- The `hosts.toml` mirror config with `capabilities = ["pull", "resolve"]` does not
+  cover referrers requests
+- The error is **not** intermittent — it happens consistently for certain images
+- Without `containerd-snapshotter` (classic Docker storage driver), the issue does not occur
+- Affects both Docker 28.x and 29.x when containerd-snapshotter is enabled
+
+## Affected versions
+
+- Docker Engine 28.x and 29.x with `containerd-snapshotter: true`
+- containerd 2.x (bundled with Docker)
+- Registry mirror: Docker Distribution `registry:2` (v2.8.3)
+- Tested on Debian 12 (bookworm), amd64
+
+## Workaround
+
+Disable the containerd image store and use the classic Docker storage driver.
+The classic driver does not make referrers API requests.
+
+Remove from `/etc/docker/daemon.json`:
+```json
+{
+  "features": {
+    "containerd-snapshotter": true
+  }
+}
+```
+
+Use `registry-mirrors` in `daemon.json` instead of `hosts.toml` for Docker Hub mirroring.
+Note: `registry-mirrors` only supports Docker Hub, not per-registry mirrors (e.g. ghcr.io).
+
+## Reproduction
+
+See `scripts/reproduce-containerd-referrers.sh` for a self-contained reproduction script.