Kubernetes pod stuck in Pending: "0/3 nodes are available: Insufficient cpu"
Fix pods stuck in Pending: read the FailedScheduling event, then fix CPU and memory requests, taints, node selectors, volumes or the cluster size.
by OhSudo ··
Not sure this is your cause?Find it step by step: Pod not running →
Part of a learning path:Kubernetes: debugging pods
The pod is created, but never starts:
$ kubectl get podsNAME READY STATUS RESTARTS AGEapi-5d7c9f8b6d-r2lkp 0/1 Pending 0 6mPending means the scheduler hasn’t found a node for the pod yet. Nothing has been pulled or started, so there are no logs. The scheduler explains its decision in an event — one line that lists, node by node, why each one was rejected.
- Podrequests CPU and memoryto place
- Schedulerchecks requests, taints, selectorsevery node rejected(fails)
- No node fitsInsufficient cpu, untolerated taint…(breaks here)
TL;DR
kubectl describe pod api-5d7c9f8b6d-r2lkp # the FailedScheduling event at the bottomkubectl describe nodes | grep -A8 'Allocated resources'Warning FailedScheduling 2m default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.Read it as: of 3 nodes, 1 was excluded by a taint, and 2 didn’t have enough CPU left to reserve. Versions before 1.37 also name the taint: had untolerated taint {node-role.kubernetes.io/control-plane: }.
1. “Insufficient cpu” / “Insufficient memory” is about requests
The scheduler doesn’t look at how busy a node is. It looks at what the pods on it have requested (resources.requests). A node at 5% real CPU use can still be “full” if its pods reserved all of it. Compare both:
kubectl describe node worker-1 | grep -A8 'Allocated resources'kubectl top nodes # real usage (needs metrics-server)Allocated resources: Resource Requests Limits -------- -------- ------ cpu 3850m (96%) 6 (150%) memory 5Gi (68%) 9Gi (122%)96% of the CPU is reserved. A pod that asks for 500m (half a core) doesn’t fit, even if the node is idle.
Ways out, in order of preference:
- Requests that are too high. Set them close to what the app really uses (
kubectl top pods, or your monitoring), not “just in case”:
resources: requests: cpu: 250m memory: 256Mi limits: memory: 512Mi- Leftovers. Old deployments, test namespaces and finished experiments still reserve their requests.
kubectl get pods -Aand clean up. - A namespace default you didn’t write. A
LimitRangecan add default requests to every pod in a namespace:kubectl get limitrange -n <namespace> -o yaml. - A real lack of capacity. Add nodes or bigger ones. With a cluster autoscaler, a Pending pod is exactly what triggers a new node — check its logs if no node appears.
2. The other reasons in the same message
The event lists every reason, with the number of nodes for each. Fix the ones that cover all nodes:
| In the event | Meaning | Fix |
|---|---|---|
had untolerated taint(s) (or taint {…} before 1.37) |
The node repels pods without a matching toleration (control-plane nodes, dedicated node pools) | Normal for control-plane nodes; otherwise add a toleration, or use other nodes |
didn't match Pod's node affinity/selector |
The pod’s nodeSelector or affinity asks for a label no node has |
kubectl get nodes --show-labels; fix the label or the selector |
pod has unbound immediate PersistentVolumeClaims |
The volume the pod needs doesn’t exist yet | kubectl get pvc; check the StorageClass and the PVC’s events |
node(s) had volume node affinity conflict |
The volume lives in another zone than the free nodes | Schedule in that zone, or use a zone-aware StorageClass (WaitForFirstConsumer) |
Too many pods |
The node reached its maximum number of pods | More or bigger nodes, or a higher max-pods setting |
didn't have free ports for the requested pod ports |
A hostPort is already used on those nodes |
Avoid hostPort; use a Service |
3. Check what the pod really asks for
The deployment you wrote and the pod that runs can differ (defaults from a LimitRange, a Helm value you forgot). Read it from the pod:
kubectl get pod api-5d7c9f8b6d-r2lkp -o jsonpath='{.spec.containers[*].resources}'kubectl get pod api-5d7c9f8b6d-r2lkp -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.tolerations}'A single pod with a typo — cpu: 4 instead of cpu: 400m — asks for four whole cores and may never fit anywhere.
Note
Priority and preemption. If the pod has a higher priorityClassName than others, the scheduler may evict lower-priority pods to make room — that’s what the “preemption” part of the message is about. “No preemption victims found” just means it couldn’t, or wasn’t allowed to.
Shell, we have a problem!
Similar, but different: ImagePullBackOff / ErrImagePull — the pod was scheduled, but the image can’t be pulled; see Kubernetes ImagePullBackOff. CrashLoopBackOff — it started and keeps exiting; see Kubernetes CrashLoopBackOff. exceeded quota in the ReplicaSet’s events — a ResourceQuota stops the pod from even being created: kubectl describe resourcequota -n <namespace>. If pods were removed from a node, see Evicted.
$ ./practice.sh
Practice this in the terminal
Open the Terminal Playground with this article’s scenario loaded. It is safe and runs in your browser.
Found this useful? Share it.
Found a mistake? Let us know at contact@ohsudo.com.