β Kubernetes: list all ImagePullBackOff container images
β’ 227 words β’ 2 min β’ updated
Problem statement Upgrade the
kube-prometheus-stack
helm chart in Azure China to the latest version.
Side effect: Many container images are now missing from our internal
ACR (Azure
Container Registry). They need to be synced with skopeo.
Which images need to be synced? Use the command below, it is quite readable1:
shell
kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(any(.status.containerStatuses[]; .state.waiting.reason == "ImagePullBackOff")) | .metadata.namespace + "/" + .metadata.name + "\t" + ( [.status.containerStatuses[] | select(.state.waiting.reason == "ImagePullBackOff").image] | join(", ") )'Sample run:
shell
ubuntu@universe:~ $ kubectl get pods --all-namespaces -o json | jq -r '.items[]
| select(any(.status.containerStatuses[]; .state.waiting.reason == "ImagePullBackOff"))
| .metadata.namespace + "/" + .metadata.name + "\t" +
( [.status.containerStatuses[] | select(.state.waiting.reason == "ImagePullBackOff").image] | join(", ") )'
monitoring/clustermon-kube-prometheus-operator-67c8b6c87c-w72km {my-registry-name}.azurecr.cn/quay.io/prometheus-operator/prometheus-operator:v0.83.0
monitoring/clustermon-prometheus-node-exporter-v7bpt {my-registry-name}.azurecr.cn/quay.io/prometheus/node-exporter:v1.9.1How to skopeo? First, log in:
shell
echo -n "{password}" | skopeo login {my-registry-name}.azurecr.cn --password-stdin --username {my-username}How do you get these credentials? Log in to the Azure China Portal, find the ACR resource, navigate to Settings > Access keys.
Then sync each image2:
shell
% skopeo sync \
--override-os linux --override-arch amd64 \
--src docker --dest docker \
quay.io/prometheus/node-exporter:v1.9.1 \
{my-registry-name}.azurecr.cn/quay.io/prometheus