containerd-snapshotter: pull-through mirror fails with "failed to decode referrers index"

Summary

When Docker Engine uses the containerd image store ("features": {"containerd-snapshotter": true}) and a pull-through registry mirror (Docker Distribution registry:2), image pulls fail with:

failed to decode referrers index: invalid character '<' looking for beginning of value

or:

failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:...: not found

Root cause

After pulling an image through the mirror, containerd makes a referrers API request (OCI Distribution Spec 1.1) directly to the upstream registry (e.g. registry-1.docker.io), bypassing the mirror entirely. The upstream sometimes returns an HTML error page instead of a valid JSON response for the referrers endpoint, causing containerd to fail the entire pull.

Key observations:

The image layers and manifests pull successfully through the mirror
The referrers request goes directly to upstream, not through the mirror
The hosts.toml mirror config with capabilities = ["pull", "resolve"] does not cover referrers requests
The error is not intermittent — it happens consistently for certain images
Without containerd-snapshotter (classic Docker storage driver), the issue does not occur
Affects both Docker 28.x and 29.x when containerd-snapshotter is enabled

Affected versions

Docker Engine 28.x and 29.x with containerd-snapshotter: true
containerd 2.x (bundled with Docker)
Registry mirror: Docker Distribution registry:2 (v2.8.3)
Tested on Debian 12 (bookworm), amd64

Workaround

Disable the containerd image store and use the classic Docker storage driver. The classic driver does not make referrers API requests.

Remove from /etc/docker/daemon.json:

{
  "features": {
    "containerd-snapshotter": true
  }
}

Use registry-mirrors in daemon.json instead of hosts.toml for Docker Hub mirroring. Note: registry-mirrors only supports Docker Hub, not per-registry mirrors (e.g. ghcr.io).

Reproduction

See scripts/reproduce-containerd-referrers.sh for a self-contained reproduction script.

2.2 KiB Raw Blame History