Kubernetes pods "Evicted": find the pressure on the node, and stop it from coming back

1. Read why

kubectl describe pod myapp-7d9f8b6c4-x2k9p

Status:   Failed
Reason:   Evicted
Message:  The node was low on resource: ephemeral-storage. Threshold quantity: 5251858022, available: 4029804Ki. Container myapp was using 4Gi, request is 0, has larger consumption of ephemeral-storage.

The message says what ran short, and which container used a lot of it. The threshold is in bytes: here, the default 10% of the node’s disk. Across the namespace:

kubectl get events --field-selector reason=Evicted --sort-by=.lastTimestamp

Events only last about an hour by default, while the pod’s message stays for as long as the pod does. So don’t delete them before you’ve read one.

2. DiskPressure

kubectl describe node worker-2 | grep -E 'DiskPressure|ephemeral-storage'

DiskPressure     True    KubeletHasDiskPressure    kubelet has disk pressure

The kubelet’s default hard limits are nodefs.available<10% and imagefs.available<15%, among others. See what’s using the disk, on the node itself, through a debug pod:

kubectl debug node/worker-2 -it --image=ubuntu
chroot /host
df -h
du -xh --max-depth=1 /var/lib | sort -h | tail
crictl images

Three usual causes:

Container logs. An application that writes a lot to stdout fills /var/log/pods. Limit them with containerLogMaxSize and containerLogMaxFiles in the kubelet config, and make the application less talkative.

Writable layers and emptyDir. An application writing into its own filesystem, such as temp files or caches, counts as ephemeral storage. Find the biggest:

kubectl get --raw "/api/v1/nodes/worker-2/proxy/stats/summary" | \
  python3 -c 'import json,sys; [print(p["ephemeral-storage"]["usedBytes"]//1048576, "MiB", p["podRef"]["name"]) for p in json.load(sys.stdin)["pods"]]' | sort -n | tail

Unused images. The kubelet cleans those itself, and sudo crictl rmi --prune does it now. A CI node that builds images needs this most.

No space left on device has more on hunting down space on a Linux machine. Then either make the disk bigger, or set limits, as in section 6.

3. MemoryPressure

kubectl describe node worker-2 | grep MemoryPressure
kubectl top nodes
kubectl top pods -A --sort-by=memory | head -n 15

When memory’s short, the kubelet evicts in this order, roughly: pods using more than they requested go first, and among those, by priority and how far above the request they are. So a pod with no requests at all is an easy target.

Memory shortage can also bring in the kernel’s OOM killer, which ends one process rather than the pod. That shows up as OOMKilled, with exit code 137: see that article.

4. PIDPressure

kubectl describe node worker-2 | grep PIDPressure

Something’s creating processes without limit, often a process that doesn’t clean up its children. Find it on the node with ps -eLf | wc -l and ps aux --sort=-nlwp | head. An init process in the container, or podSpec.shareProcessNamespace, may help with zombies.

5. Other reasons pods get removed

Not everything that’s removed is evicted by pressure:

• Preempted: a higher-priority pod needed the room. The message says so.
• The API or kubectl drain: a planned action, and it respects PodDisruptionBudgets.
• Node shutdown: spot or preemptible nodes being reclaimed.
• Over its own limit: a pod exceeding its ephemeral-storage limit is evicted even though the node is fine, with a message such as Pod ephemeral local storage usage exceeds the total limit of containers.

6. Set requests and limits

This is the lasting fix. Requests make the scheduler place pods where they fit, and lower their position in the eviction queue:

resources:
  requests:
    cpu: 100m
    memory: 256Mi
    ephemeral-storage: 1Gi
  limits:
    memory: 512Mi
    ephemeral-storage: 2Gi

Set them from real use, as seen in kubectl top, rather than guessing. The Kubernetes Manifest Generator includes CPU and memory requests by default. For temp files, limit that as well:

volumes:
  - name: tmp
    emptyDir:
      sizeLimit: 1Gi

Important workloads can have a higher priority with a PriorityClass. And add monitoring for node disk and memory, as it’s easy not to notice pods being replaced.

7. Clean up the dead pods

Evicted pods stay around so that you can read why. They use no resources besides an API object. Remove them when you’re done:

kubectl delete pods --field-selector=status.phase=Failed
kubectl delete pods -A --field-selector=status.phase=Failed

The first is for one namespace, and the second for all. Note that this removes all failed pods, including those from other failures, which you may want to look at first. Kubernetes only cleans them up by itself after a high count, 12,500 by default.