Google Cloud Account Purchase: Pods Frequently Report ImagePullBackOff / ErrImagePull a Complete Set of Troubleshooting and Pit Avoidance Guidelines
In the daily operation and CI/CD release of Google Kubernetes Engine (GKE), the most troublesome thing is the dazzling line in the Pod status bar.
ErrImagePull
or
ImagePullBackOff
.
Simply put,
ErrImagePull
is the specific error thrown by the Kubelet when the first attempt to pull the image fails; and
ImagePullBackOff
It is the self-protection mechanism of K8s-when it finds that the mirror cannot be pulled down, it will enter the "backoff retry" (exponentially extending the retry interval) state.
If you encounter this problem, don't panic. The following is a set of press
Exclude priority from high to low
The organized actual combat investigation plan will help you hit the focus directly.
The first step: the first on-site diagnosis, accurate capture error log
Blindly guess permissions or network is a waste of time. First, get the first-hand error information recorded by the Kubernetes through the command line.
1. View Pod Events
Run the following command to pull to the last view
Events
Blocks:
Bash
kubectl describe pod <pod-name> -n <namespace>
Pay attention
Failed
The specific output of the event is usually directly exposed:
manifest unknown or not found: the mirror path/Tag is written incorrectly, or there is no one in the warehouse at all.
PermissionDenied or Unauthorized: The node cannot access the warehouse because of insufficient IAM permissions.
connection refused or I/o timeout: network failure (common in private clusters or firewall interception).
Step 2: Five Core Incentives and Solutions
According to the experience of the production environment, more than 90% of GKE image pull failures are caused by the following five reasons:
1. Mirror path or Tag spelling error (the most common low-level error)
After Google Cloud's mirroring service fully shifted to Artifact Registry (GAR), the path format was quite rigorous:
Plaintext
[REGION]-docker.pkg.dev/[PROJECT-ID]/[REPOSITORY]/[IMAGE]:[TAG]
# For example: asia-east1-docker.pkg.dev/my-gcp-project/my-repo/my-app:v1.0.0
Troubleshooting and repair:
check spelling: check
Check whether the region prefix (such as asia-east1), project ID, warehouse name, and image name are misspelled.
Check whether the Tag exists: run gcloud on the terminal to verify whether the image version really exists in the cloud: Bashgcloud artifacts docker images list asia-east1-docker.pkg.dev/my-project/my-repo
Note the tag variability: If the tag immutability (tag Immutability) is enabled for your repository, pushing the tag with the same name will fail, resulting in the cluster not being able to pull the latest image.
2. Insufficient GCP IAM permissions (Service Account not authorized)
The GKE node (Node) relies on the Artifact Registry when requesting a pull image.
The GCP service account bound to the node pool (Node Service Account)
instead of your personal account or the Workload Identity of the pod.
If you use the default service account when creating the GKE node pool, or use a custom service account, but do not give the Artifact Registry read permissions,
PermissionDenied
.
Troubleshooting and repair:
Obtain the service account used by the cluster node pool: Bashgcloud container node-pools describe <node-pool-name> \ -- cluster <cluster-name> \ -- zone <zone> \ -- format = "value(config.serviceAccount)"
Grant Artifact Registry read permissions to the service account (Artifact Registry Reader / roles/artifactregistry.reader):Bashgcloud projects add-iam-policy-binding <PROJECT-ID> \ -- member = "serviceAccount:<NODE-SERVICE-ACCOUNT-EMAIL>" \ -- role = "roles/artifactregistry.reader"
* (Note: If your GKE cluster is in Project A and the mirror warehouse is in Project B, be sure to be in Project B (warehouse
project) *
Give the Artifact Registry Reader permission to the node service account of project A!)
3. Private cluster network is not working (Private GKE Networking Issue)
If you are using
Private GKE cluster (Private Cluster)
the node does not have a public IP address. If the cluster is not configured with an extranet exit and private Google access is not turned on, the node cannot connect to
*.pkg.dev
Download the image.
Troubleshooting and Repair:
If you are pulling Google's internal image (Artifact Registry): you need to open Private Google Access (private Google access) on the subnet, so that traffic can directly access GAR through Google's internal network without going through the public network. Bashgcloud compute networks subnets update <SUBNET-NAME> \ --region <REGION> \ --enable-private-google-access
If you are pulling an external image (such as Docker Hub, GitHub Packages): Private nodes must use Cloud NAT to access the external network. Make sure that you have the correct Cloud NAT gateway configured in the VPC and Region where the cluster is located.
4. Pull the missing imagePullSecrets of the private third-party warehouse.
If your images are stored in a third-party private repository (such as self-built Harbor, private Docker Hub, GitLab Registry, etc.),GKE nodes cannot use GCP built-in authentication.
Troubleshooting and Repair:
Create the corresponding Docker key in K8s: Bashkubectl create secret docker-registry my-registry-key \ -- docker-server =<YOUR-PRIVATE-REGISTRY> \ -- docker-username =<USERNAME> \ -- docker-password =<PASSWORD-OR-TOKEN> \ -- docker-email =<EMAIL>
Explicitly reference the secret:YAMLspec: imagePullSecr in the Deployment / Pod YAML
ets: - name: my-registry-key containers: - name: my-app image: your-registry.com/team/app:v1
5. Resource overrun or project quota problem (service is disabled due to account arrears)
Sometimes the problem is not the K8s configuration itself, but the infrastructure base. If a GCP project suffers from resource quota limits, quota locking, or, most commonly--
Google Cloud account top-up
If the service is not completed in time, GCP will temporarily suspend or limit some API services (including the download bandwidth or storage access capacity of the Artifact Registry).
Troubleshooting and Repair:
Go to the GCP Console View Billing (Settlement) page to ensure that the settlement account status is normal.
Ensure that the company's Google cloud account recharge channel is unblocked to avoid automatic service degradation caused by credit card expiration or budget limit, which may lead to online Pod image pull timeout or connection rejection.
Step 3: Quick Verification and Prevention Recommendations
After completing the repair, you can force the Kubernetes to re-pull the image verification with the following command:
Bash
# Scheme A: Delete the Pod that reported the error and let the Deployment rebuild automatically
kubectl delete pod <pod-name> -n <namespace>
# Scheme B: Restart the Deployment directly
kubectl rollout restart deployment <deployment-name> -n <namespace>
Best practice recommendations:
Avoid using: latest tags: Use a fixed version number or Git Commit SHA as a tag to prevent Kubelet from pulling unexpected versions due to caching mechanisms.
Enable image cache or Image Streaming in the production environment: For large images, you can enable the Image Streaming function on GKE to greatly shorten the pod startup time and image download waiting period.
Sound CI/CD mechanism: Before the pipeline update Manifest, add a step of image existence check (Image Existence Check) to avoid the ImagePullBackOff caused by the deployment of the image without successful push from the source.
Following the above logic step by step troubleshooting, most GKE image pull faults can be located and resolved within a few minutes.