Linux process "Killed" with no error: find the OOM killer and stop it happening again

1. Confirm it was the OOM killer

sudo dmesg | grep -i "killed process"

[ 3290.902355] Out of memory: Killed process 1874 (python3) total-vm:3412880kB, anon-rss:3021440kB, file-rss:4kB, shmem-rss:0kB, UID:998 pgtables:6040kB oom_score_adj:0

• Killed process 1874 (python3) — the victim.
• anon-rss:3021440kB — the memory it really used when it was killed: about 2.9 GB.

dmesg only keeps recent messages, and only since the last boot. The journal keeps more, with real dates:

sudo journalctl -k --no-pager | grep -i oom

The line oom-kill:constraint=… tells you which kind of OOM it was:

• global_oom — the whole machine ran out of memory.
• CONSTRAINT_MEMCG, and the log says Memory cgroup out of memory — the process hit its own limit (a container’s memory limit, or MemoryMax= of a systemd service), even if the machine had memory left. Docker and Kubernetes then report the container as OOMKilled — see Docker exited with code 137.

For a systemd service, systemctl status says it too:

systemctl status report-worker --no-pager

× report-worker.service - Nightly report worker
     Active: failed (Result: oom-kill) since …

2. Why the machine ran out

free -h

               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       3.3Gi       189Mi        12Mi       342Mi       483Mi
Swap:             0B          0B          0B

Look at available (not “free” — the cache can be given back) and at Swap. No swap at all means the kernel has nowhere to move rarely used memory, so it reaches the OOM killer sooner. See what uses memory right now:

ps aux --sort=-%mem | head

Then decide which case it is:

• One program grows until it’s killed — a memory leak, or a job that loads too much at once (a whole file, a whole table). Fix the program: process data in chunks, stream instead of loading everything.
• Everything together doesn’t fit — too many workers, a database cache sized for a bigger machine, several apps that each assume they’re alone.
• A short spike — a build, a backup, a big query. Swap usually absorbs it.

3. Fixes

Add swap (a modest amount, as a safety net — not as extra RAM):

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Limit the program, so it’s the one that dies — not your database. For a systemd service, a memory limit makes the kernel stop that service when it goes too far, instead of choosing any process on the machine:

sudo systemctl edit report-worker

[Service]
MemoryMax=1G
Restart=on-failure

Protect what must survive. OOMScoreAdjust=-500 in a service’s unit makes the OOM killer much less likely to pick it (-1000 means never — use it only for something truly essential, like sshd).

Size things to the machine: fewer workers, a smaller database cache, Java’s -XX:MaxRAMPercentage, Node.js’s --max-old-space-size. And if the work really needs more memory, the honest fix is a bigger machine.