ImagePullBackOff: images that cannot be pulled

ImagePullBackOff (and ErrImagePull before it) means the kubelet could not pull the container image. The pod never starts. The reason is almost always in the pod's events.

Find the reason with kubectl

  1. kubectl describe pod <pod> -n <ns> and read the Events at the bottom. The Failed to pull image message names the cause.
  2. Check the exact image reference: kubectl get pod <pod> -n <ns> -o jsonpath="{.spec.containers[*].image}".
  3. Try the pull from a machine with the same network access, for example crictl pull <image> on the node.

Messages and what they mean

Message containsCauseFix
manifest unknown, not foundThe tag or digest does not existFix the tag, or push the image. Check for typos and for tags deleted by a cleanup policy.
unauthorized, authentication required, 403The registry needs credentialsCreate a docker-registry Secret and reference it in imagePullSecrets (or on the ServiceAccount).
i/o timeout, no such host, connection refusedThe node cannot reach the registryDNS, firewall, proxy settings of the container runtime, or a registry mirror for air-gapped clusters.
x509: certificate signed by unknown authorityThe registry uses a private CAAdd the CA to the container runtime's trusted certificates on every node.
toomanyrequestsRegistry rate limit (for example Docker Hub)Authenticate, use a pull-through cache or mirror the image.
no matching manifest forNo image for the node's architectureBuild a multi-arch image or schedule the pod on matching nodes.

Prevent it

  • Pin images by digest or by immutable tags in production.
  • Keep pull secrets on the ServiceAccount so every pod in the namespace gets them.
  • With GitOps, a bad tag in Git shows up as a Degraded Argo CD application: fix the tag in Git, not in the cluster.

The Kubby way

Kubby lists ImagePullBackOff pods at the top of Overview and Pods with the image and the registry's message on the row, so you see the cause without opening describe. In the Argo CD centre, "Why Degraded?" points at the pod that cannot pull its image.

Ask Kubby explains that an Argo CD application is Degraded because its new image tag does not exist in the registry