Kubernetes pod not running? Find the cause, step by step

What does the STATUS column show?

kubectl get pods -n <namespace>

• Pending→
• ImagePullBackOff or ErrImagePull→
• CrashLoopBackOff→
• CreateContainerConfigError→
• Evicted→
• kubectl itself failsBefore any pod is listed→

Likely cause

No node can take the pod

The scheduler found no node with enough requested CPU or memory, or the pod asks for something the nodes don't have (a selector, a taint, a volume). The events say which, node by node.

Check it

kubectl describe pod <pod> | sed -n "/^Events:/,$p"

Not this?Start over

Likely cause

The node can't download the image

A wrong name or tag, missing registry credentials, Docker Hub's rate limit, the network, or an image built for another CPU. The pod's events have the exact pull error.

Check it

kubectl describe pod <pod> | sed -n "/^Events:/,$p"

Not this?Start over

Likely cause

The container starts, then exits, again and again

Kubernetes keeps restarting it and waits longer each time. The logs of the previous run and the last exit code (1, 137 for out of memory, 0 for a program that just ends) tell you why.

Check it

kubectl logs <pod> --previous
kubectl describe pod <pod> | grep -A3 "Last State"

Not this?Start over

What do the pod's events say?

kubectl describe pod <pod> | sed -n "/^Events:/,$p"

• secret "…" not found, configmap "…" not found, or couldn't find key→
• container has runAsNonRoot and image will run as root→

Start over

Likely cause

The node removed the pod to save itself

The node ran low on disk, memory or process IDs and evicted pods. The pod's status says which pressure.

Check it

kubectl get pod <pod> -o jsonpath="{.status.message}"

Not this?Start over

What does kubectl say?

• The connection to the server localhost:8080 was refused→
• x509: certificate signed by unknown authority→
• Forbidden: User "…" cannot list resource→

Start over

Likely cause

A Secret or ConfigMap the pod needs is missing

The pod takes environment variables from a Secret or ConfigMap that doesn't exist in its namespace, or from a key it doesn't have.

Check it

kubectl get secret,configmap -n <namespace>

Not this?Start over

Likely cause

The pod must run as non-root, and the image runs as root

A security setting (runAsNonRoot, often from a policy) refuses an image whose user is root, or a name instead of a number. Fix the image's USER, or set runAsUser.

Not this?Start over

Likely cause

kubectl has no kubeconfig

localhost:8080 is what kubectl tries when it found no configuration at all: no ~/.kube/config, the wrong KUBECONFIG, or a command run with sudo.

Check it

kubectl config view --minify

Not this?Start over

Likely cause

kubectl doesn't trust the cluster's certificate

Your kubeconfig has another CA than the cluster now uses (a recreated cluster), or a proxy inspects TLS on the way.

Not this?Start over

Likely cause

You're logged in, and RBAC says no

The cluster knows who you are, and no Role or ClusterRole lets you do that. Ask the cluster what you may do, then grant exactly that.

Check it

kubectl auth can-i list pods -n <namespace>

Not this?Start over