Essential Linux Commands: A Backend Engineer's Cheat Sheet

Production Debugging: When You SSH Into a Burning Server

An e-commerce API's latency spiked 50ms → 2,000ms. Dashboards showed nothing wrong. One ss -tnp | wc -l: 48,000 TCP connections. One lsof -i command revealed a missing Close() call in a recent deploy. Finding it required knowing which Linux commands to reach for.

Triage by Symptom

When you SSH into a burning server, the question is "where is the bottleneck" — not "what does each command do." Route by symptom, not by command name:

graph TD
    Burning[Server is on fire] --> Sym{What is<br/>the symptom?}
    Sym -->|High CPU| CPU[top -o %CPU<br/>+ pidstat -u 1<br/>+ perf top]
    Sym -->|High memory| Mem[free -h<br/>+ ps aux --sort=-%mem<br/>+ smem]
    Sym -->|Disk full or slow| Disk[df -h<br/>+ du -sh */<br/>+ iostat -x 1]
    Sym -->|Network slow or stuck| Net[ss -tnp<br/>+ ss -s<br/>+ tcpdump -i any port X]
    Sym -->|Process stuck| Proc[strace -p PID<br/>+ lsof -p PID<br/>+ cat /proc/PID/wchan]
    Sym -->|Service down| Svc[systemctl status<br/>+ journalctl -u svc -b<br/>+ journalctl -p err]
    CPU -->|Single hot core| Hot[mpstat -P ALL 1<br/>find which thread]
    Mem -->|OOMKilled| OOM["dmesg | grep -i kill<br/>find which proc died"]
    Net -->|TCP exhaustion| TCP[ss -s shows TIME-WAIT spike<br/>fix: close, pool, or kernel knobs]
    style Burning fill:#fdd
    style OOM fill:#fdd
    style TCP fill:#fdd
    style Hot fill:#ffd

The diagram is the entire production-debugging discipline in one picture: classify the symptom, pick three commands, never start with "let me run htop and see what's happening."

The Essential Dozen

File & Text Inspection

Process & Performance Monitoring

The pid → process tree → resource view in one diagram — every triage starts at one of these vertices:

graph TB
    PID[PID 1234] --> Top[top / htop<br/>CPU + memory ranking]
    PID --> Files[lsof -p 1234<br/>open fds, sockets, files]
    PID --> Stack[strace / py-spy / pprof<br/>syscall + stack profile]
    PID --> Tree[pstree -p 1234<br/>parent + children + threads]
    Tree --> Children[Child PIDs<br/>recursive triage]
    System[System-wide<br/>aggregates] --> VM[vmstat 1<br/>cpu, mem, swap, io]
    System --> Free[free -h<br/>RAM + swap totals]
    System --> Disk[df -h<br/>du -sh path<br/>iostat -x]
    System --> Kern[dmesg + journalctl -k<br/>OOM-killer, page faults,<br/>kernel panics]
    Kern -.->|grep -i 'killed process'| OOM[OOM-killer victim<br/>+ rss at kill time]
    style OOM fill:#fdd
    style Top fill:#dfd
    style VM fill:#dfd

Service Management (systemd & journalctl)

Crashed service: check systemctl status app, then journalctl -u app -n 100, then journalctl -u app -b -1 if boot failure.

Networking & Connectivity

SSH & Permissions

Firewall & Scheduling

Cron gotcha: Minimal $PATH — use absolute paths.

Disk & Kernel Tuning

Key kernel tunables: net.core.somaxconn=65535, fs.file-max=2097152, vm.swappiness=10

Note: Limits must match at 3 layers (limits.conf, systemd LimitNOFILE, sysctl fs.file-max); effective limit is minimum.

Advanced Debugging

When to use: strace = syscalls. lsof = open state. perf = CPU hotspots. tcpdump = packets.

When to use what

fd note: On Debian/Ubuntu the apt package is fd-find and the binary installs as fdfind (the fd name collides with an existing package). Run it as fdfind, or alias it: ln -s "$(command -v fdfind)" ~/.local/bin/fd.

Gotchas that bite in production

1. rm -rf $VAR with an empty variable expands to rm -rf /

   • rm -rf $BUILD_ROOT/ with BUILD_ROOT unset deletes / in seconds.
   • Fix: rm -rf "${VAR:?unset}" fails loudly when unset; set -u at the top of every script; test -n "$VAR" before any destructive path.

2. OOM kill is silent — exit code 137, nothing in app logs

   • The kernel logs it, not your app; systemd restarts it quietly.
   • Fix: dmesg | grep -i oom / journalctl -k | grep oom. Set MemoryLimit below real peak so failure is fast and visible; watch ps aux --sort=-%mem under load.

3. Cron jobs fail silently — cron's $PATH is /usr/bin:/bin

   • The script calls jq from /usr/local/bin; cron can't find it; backups quietly never run.
   • Fix: absolute paths inside scripts, or PATH=/usr/local/bin:/usr/bin:/bin at the top of the crontab.

4. ulimit doesn't stick — three layers conflict, minimum wins

   • Shell ulimit works; the restarted service is back at 1024. Effective limit = min(limits.conf, systemd LimitNOFILE, sysctl fs.file-max).
   • Fix: set all three — sysctl fs.file-max = 2097152 (plain integer, 2M rejected), limits.conf app soft/hard nofile 65535, systemctl set-property myapp.service LimitNOFILE=65535. Verify with cat /proc/<PID>/limits.

5. SSH key with permissions > 600 is silently ignored

   • A key copied under default umask lands at 644; SSH refuses it ("Permissions too open") and the error looks like a key mismatch.
   • Fix: chmod 600 ~/.ssh/id_* after every copy, or umask 077 before copying.

6. Port listening locally but unreachable externally

   • ss -tlnp shows 8080 bound; external nc -zv host 8080 times out — the outbound rule on the source is as likely as the inbound rule on the target.
   • Fix: check the firewall in BOTH directions (ufw allow in/out 8080/tcp, or firewall-cmd), then re-test with nc -zv -w 5 host 8080.

Performance triage in 90 seconds

One connect, one paste, one decision. Five commands, top-to-bottom, then route to the right deep dive.

# 1. Top — what is hot right now (sorted by CPU, refreshed every second)
top -b -n 1 -o %CPU | head -20
 
# 2. vmstat — run-queue depth, context switches, swap activity (5 samples, 1s apart)
vmstat 1 5
 
# 3. iostat — per-device queue depth, await, %util (the disk truth)
iostat -xz 1 3
 
# 4. ss — listen backlog, established/time-wait counts, retransmits
ss -s && ss -tnp state established | head
 
# 5. dmesg — last 30 kernel events: OOM-killer, segfaults, NIC resets, EXT4 errors
dmesg -T --level=err,warn | tail -30

top reveals the heavy process and load average — load > cores means CPU saturation. vmstat's r column is the run-queue depth (anything sustained over nproc is CPU-starved); the si/so columns being non-zero means the box is swapping and you should stop reading and reboot or kill the leaker. iostat's %util near 100 with await over 50ms is disk-bound; r_await > w_await points at read-heavy hotspots. ss -s shows the socket summary — if synrecv or timewait is unbounded you are TCP-saturated, not application-slow. dmesg is the only place the kernel admits it killed your process.

perf for production CPU profiles

When top blames a process but you need to know which function inside it is burning CPU, perf samples the running stack at 99 Hz and produces a flame graph that reads top-down: wide bars are hot, deep stacks are call paths. The workflow below profiles PID 1234 for 30 seconds, then renders SVG you can open in any browser.

# Capture stack samples (-g = call graphs, -F 99 = 99 Hz, --all-user = skip kernel time)
sudo perf record -F 99 -g --all-user -p 1234 -- sleep 30
 
# Fold stacks into the format flamegraph.pl expects
sudo perf script | /opt/FlameGraph/stackcollapse-perf.pl > out.folded
 
# Render the SVG (one HTML file, no JS server needed)
/opt/FlameGraph/flamegraph.pl out.folded > flame.svg

Wide plateaus near the top are the functions actually consuming CPU; tall narrow towers are deep call chains that are not hot. Add -e cycles:pp for precise sampling on Intel skid-prone loops, and --call-graph dwarf if your binaries lack frame pointers (Go binaries built before 1.20 are the usual offender).

eBPF tracing without recompiling

bpftrace attaches probes to live kernel and userspace functions without restarting the process — so you can answer "which file is the slow read" or "which syscalls is PID 1234 actually making" on a production box that you cannot recompile. The two one-liners below cover the most common questions.

# Histogram of read() latency by PID — find the slow reader
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
  tracepoint:syscalls:sys_exit_read /@start[tid]/
  { @ns[pid, comm] = hist(nsecs - @start[tid]); delete(@start[tid]); }'
 
# Block I/O latency by device — is the SSD actually slow, or is queueing the problem?
sudo bpftrace -e 'tracepoint:block:block_rq_issue { @start[args->dev] = nsecs; }
  tracepoint:block:block_rq_complete /@start[args->dev]/
  { @us[args->dev] = hist((nsecs - @start[args->dev]) / 1000); }'

Press Ctrl+C and bpftrace prints power-of-two histograms — the bucket where most events land is your real latency, not the average. Overhead is sub-1% on modern kernels (5.4+) and the probes detach cleanly on exit, which is why this replaced strace -c for production tracing. ^{[Linux kernel docs]}

Frequently Asked Questions

Keep Reading

• Essential Docker Commands Cheat Sheet — Container lifecycle and debugging.
• Essential Kubernetes Commands Cheat Sheet — When the burning service is running on Kubernetes, route through kubectl first.
• The 3 Pillars of Observability — Metrics, logs, traces when CLI tools aren't enough.
• SRE: SLOs, SLIs, and Error Budgets — When triage is over, the next question is "did this burn an error budget."
• Go Graceful Shutdown in Production — The application-side counterpart to journalctl -u svc -b after a crash loop.