Skip to content
ohsudo, Home

Module 5: Real incidents

Incident: the site gives a 502

The home page works, and the API gives a 502 Bad Gateway. Nginx is up, and something behind it isn't. This one plays out differently on each tab, so try both.

● Runs 100% in your browser. Nothing here can harm a real system.

Step 1 of 6

See it yourself

-i shows the response's headers along with its body. Always reproduce a problem first, or you can't tell when you've fixed it.

Your task

Request http://localhost/api/health, with the headers.

guest@ohsudo: ~ — Ubuntu 24.04

Starting the terminal…

What you'll practice

  1. See it yourself. -i shows the response's headers along with its body. Always reproduce a problem first, or you can't tell when you've fixed it.
  2. Ask Nginx. A 502 means Nginx couldn't get an answer from behind it, and its error log says why. (111: Connection refused) means nothing's listening on that port.
  3. What's behind it?. The log names the address, 127.0.0.1:8000. Check whether anything's listening there. No output means nothing is.
  4. Find the service. The application here is a service called webapp. Result: oom-kill means the kernel ended it because the machine ran out of memory.
  5. Bring it back. Starting it restores the site. Then look at memory, or this happens again. The article on the OOM killer covers that.
  6. Still a 502? (RHEL, Rocky, AlmaLinux). On this family, SELinux is on by default, and it doesn't let Nginx connect to other programs unless it's allowed to. (13: Permission denied) in the log, with a service that's running, is the giveaway.
  7. Allow it (RHEL, Rocky, AlmaLinux). httpd_can_network_connect is the setting for this. -P keeps it after a reboot. It's much better than turning SELinux off, which removes the protection everywhere.
  8. Confirm the fix. Run the same request you started with. Now the answer should come from your application.

See also