60 lines
2.2 KiB
Markdown
60 lines
2.2 KiB
Markdown
# containerd-snapshotter: pull-through mirror fails with "failed to decode referrers index"
|
|
|
|
## Summary
|
|
|
|
When Docker Engine uses the containerd image store (`"features": {"containerd-snapshotter": true}`)
|
|
and a pull-through registry mirror (Docker Distribution `registry:2`), image pulls fail with:
|
|
|
|
```
|
|
failed to decode referrers index: invalid character '<' looking for beginning of value
|
|
```
|
|
|
|
or:
|
|
|
|
```
|
|
failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:...: not found
|
|
```
|
|
|
|
## Root cause
|
|
|
|
After pulling an image through the mirror, containerd makes a **referrers API request**
|
|
(OCI Distribution Spec 1.1) directly to the **upstream registry** (e.g. `registry-1.docker.io`),
|
|
bypassing the mirror entirely. The upstream sometimes returns an HTML error page instead of
|
|
a valid JSON response for the referrers endpoint, causing containerd to fail the entire pull.
|
|
|
|
Key observations:
|
|
- The image layers and manifests pull successfully through the mirror
|
|
- The referrers request goes **directly to upstream**, not through the mirror
|
|
- The `hosts.toml` mirror config with `capabilities = ["pull", "resolve"]` does not
|
|
cover referrers requests
|
|
- The error is **not** intermittent — it happens consistently for certain images
|
|
- Without `containerd-snapshotter` (classic Docker storage driver), the issue does not occur
|
|
- Affects both Docker 28.x and 29.x when containerd-snapshotter is enabled
|
|
|
|
## Affected versions
|
|
|
|
- Docker Engine 28.x and 29.x with `containerd-snapshotter: true`
|
|
- containerd 2.x (bundled with Docker)
|
|
- Registry mirror: Docker Distribution `registry:2` (v2.8.3)
|
|
- Tested on Debian 12 (bookworm), amd64
|
|
|
|
## Workaround
|
|
|
|
Disable the containerd image store and use the classic Docker storage driver.
|
|
The classic driver does not make referrers API requests.
|
|
|
|
Remove from `/etc/docker/daemon.json`:
|
|
```json
|
|
{
|
|
"features": {
|
|
"containerd-snapshotter": true
|
|
}
|
|
}
|
|
```
|
|
|
|
Use `registry-mirrors` in `daemon.json` instead of `hosts.toml` for Docker Hub mirroring.
|
|
Note: `registry-mirrors` only supports Docker Hub, not per-registry mirrors (e.g. ghcr.io).
|
|
|
|
## Reproduction
|
|
|
|
See `scripts/reproduce-containerd-referrers.sh` for a self-contained reproduction script.
|