# containerd-snapshotter: pull-through mirror fails with "failed to decode referrers index"

## Summary

When Docker Engine uses the containerd image store (`"features": {"containerd-snapshotter": true}`)
and a pull-through registry mirror (Docker Distribution `registry:2`), image pulls fail with:

```
failed to decode referrers index: invalid character '<' looking for beginning of value
```

or:

```
failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:...: not found
```

## Root cause

After pulling an image through the mirror, containerd makes a **referrers API request**
(OCI Distribution Spec 1.1) directly to the **upstream registry** (e.g. `registry-1.docker.io`),
bypassing the mirror entirely. The upstream sometimes returns an HTML error page instead of
a valid JSON response for the referrers endpoint, causing containerd to fail the entire pull.

Key observations:
- The image layers and manifests pull successfully through the mirror
- The referrers request goes **directly to upstream**, not through the mirror
- The `hosts.toml` mirror config with `capabilities = ["pull", "resolve"]` does not
  cover referrers requests
- The error is **not** intermittent — it happens consistently for certain images
- Without `containerd-snapshotter` (classic Docker storage driver), the issue does not occur
- Affects both Docker 28.x and 29.x when containerd-snapshotter is enabled

## Affected versions

- Docker Engine 28.x and 29.x with `containerd-snapshotter: true`
- containerd 2.x (bundled with Docker)
- Registry mirror: Docker Distribution `registry:2` (v2.8.3)
- Tested on Debian 12 (bookworm), amd64

## Workaround

Disable the containerd image store and use the classic Docker storage driver.
The classic driver does not make referrers API requests.

Remove from `/etc/docker/daemon.json`:
```json
{
  "features": {
    "containerd-snapshotter": true
  }
}
```

Use `registry-mirrors` in `daemon.json` instead of `hosts.toml` for Docker Hub mirroring.
Note: `registry-mirrors` only supports Docker Hub, not per-registry mirrors (e.g. ghcr.io).

## Reproduction

See `scripts/reproduce-containerd-referrers.sh` for a self-contained reproduction script.