Disk full or server out of memory? Find the cause, step by step

What do you see?

• No space left on device→
• Read-only file systemOn a disk that should be writable→
• A process was "Killed", or something ran out of memory→
• Too many open files→
• A service stops writing because its disk is fullElasticsearch, Redis, Kubernetes pods→

What does df say?

df -h
df -i

• A filesystem is at 100% (or close)In the Use% column of df -h→
• There's free space, and writing still failsIUse% at 100% in df -i, or nothing full at all→
• I deleted big files, and df still says it's full→

Start over

Likely cause

The kernel switched the filesystem to read-only

Usually to protect it, after a disk or filesystem error. The kernel log says why; repair it unmounted, and check whether the disk is failing.

Check it

sudo journalctl -k | grep -iE "remount|error" | tail

Not this?Start over

Where did it run out?

• A process on the server was "Killed"Or a service died with no error of its own→
• A Docker container exited with code 137→
• Redis: OOM command not allowed→

Start over

Likely cause

A process reached its file descriptor limit

Each open file and network connection uses one. Check the limit of the process itself (not of your shell), then whether it's too low or the program leaks them.

Check it

cat /proc/$(pidof -s nginx)/limits | grep "open files"

Not this?Start over

Which service?

• Elasticsearch: index read-only, flood stage watermark→
• Redis: MISCONF … unable to persist to disk→
• Kubernetes: pods Evicted, DiskPressure→

Start over

Likely cause

The filesystem is full

Find which directories grew (logs, Docker images, caches, old kernels in /boot), then free space safely, without deleting files programs still have open.

Check it

sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -n 15

Not this?Start over

Likely cause

Out of inodes, or another limit

A filesystem can run out of inodes (one per file) while it still has space, usually from millions of tiny files: sessions, caches, mail queues. A quota or a read-only mount can also refuse writes.

Check it

df -i

Not this?Start over

Likely cause

A running program still holds the deleted file

The space comes back only when the last program closes the file. A log deleted under a running service is the classic case: restart the service, or empty the file instead of deleting it.

Check it

sudo lsof +L1 | sort -k7 -n | tail

Not this?Start over

Likely cause

The kernel's OOM killer stopped it

The machine (or the service's cgroup) ran out of memory, and the kernel killed the biggest process to save itself. The kernel log names it.

Check it

sudo journalctl -k | grep -iE "out of memory|oom-kill"

Not this?Start over

Likely cause

The container went over its memory limit, or was killed on stop

137 is SIGKILL. Docker records OOMKilled=true when the container hit its memory limit.

Check it

docker inspect -f "{{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}" api

Not this?Start over

Likely cause

Redis is at its memory limit

used_memory reached maxmemory, and the eviction policy doesn't allow removing keys.

Check it

redis-cli INFO memory | grep -E "used_memory_human|maxmemory_human|maxmemory_policy"

Not this?Start over

Likely cause

Elasticsearch blocked writes to protect the disk

Over the flood-stage watermark (95% by default), it makes indices read-only. Free space first; recent versions lift the block by themselves once the disk is below the limit.

Check it

curl -s localhost:9200/_cat/allocation?v

Not this?Start over

Likely cause

Redis can't save to disk, so it stops writes

The last background save failed: a full disk, a folder Redis can't write, or not enough memory to fork.

Not this?Start over

Likely cause

The node evicted pods to save itself

The node ran low on disk (or memory) and removed pods. Clean up the node's disk, and set requests and limits so it doesn't happen again.

Check it

kubectl describe node <node> | grep -A6 Conditions

Not this?Start over