The Linux Troubleshooting Toolkit

Table of Contents#

1. Why a Toolkit, Not Just a List of Commands
2. The 60-Second Triage Checklist
3. CPU Tools
4. Memory Tools
5. Disk Tools
6. Network Tools
7. Process-Level Deep Dive Tools
8. Log Tools
9. tcpdump — Watching the Actual Network Traffic
10. strace — Watching a Process Talk to the Kernel
11. A Full Worked Incident, Start to Finish
12. Building Your Own Cheat Sheet
13. Common Mistakes
14. Worked Practice Problems
15. Summary — The Complete Linux & Networking Series

---

Why a Toolkit, Not Just a List of Commands#

Parts 1 and 2 explained the concepts — processes, memory, TCP, DNS. This part is about the actual commands you reach for during a real incident, organized the same way the USE method from the Monitoring Methodologies series organizes an investigation: CPU, then Memory, then Disk, then Network, then process-level detail. Memorizing commands in isolation is much less useful than knowing which resource each one checks and in what order to reach for them.

---

The 60-Second Triage Checklist#

This is directly inspired by Brendan Gregg's well-known "Linux Performance Analysis in 60 Seconds" checklist, already referenced in the Monitoring Methodologies series — here it is in full, hands-on detail, as the backbone of this entire tutorial.

Diagram

Why running these in this specific order matters: each command takes seconds to run and immediately either rules out or points toward a category of problem — by the time you've run all eight (genuinely achievable in under a minute with practice), you have a full USE-method picture across CPU, memory, disk, and network, and usually already know which specific area deserves deeper investigation.

---

CPU Tools#

Copy
# Quick overview: load average over 1/5/15 minutes
uptime
#  14:32:01 up 10 days,  3:14,  2 users,  load average: 8.42, 6.15, 4.03

Reading load average correctly is a genuinely common interview question. Load average roughly represents the average number of processes wanting CPU time (running + waiting) over the last 1/5/15 minutes. The critical context it needs: how many CPU cores does the machine have? A load average of 8 is perfectly healthy on a 16-core machine (half utilized) but a serious red flag on a 4-core machine (2x oversubscribed).

Copy
# See CPU usage broken down PER CORE — reveals uneven load
# (e.g. one core pegged at 100% while others sit idle, often
# a sign of a single-threaded bottleneck)
mpstat -P ALL 1

# Live, refreshing view of overall system activity
vmstat 1
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
#  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
#  12 0      0 812340  45012 891200    0    0    45   120 3200 8500 45 12 40  3  0

The r (run queue) column in vmstat is exactly the CPU saturation metric referenced throughout the Monitoring Methodologies series — if r consistently exceeds the number of CPU cores, processes are genuinely waiting for a core, not just using one efficiently.

---

Memory Tools#

Copy
# Overview — always check "available," not "free" (Part 1's lesson)
free -h

# Per-process memory usage, sorted by usage
ps aux --sort=-%mem | head -10

# The single best live view: per-process memory AND CPU together
top
htop

Copy
# Check for recent OOM kills — the single most important
# memory-related log check, directly from Part 1
dmesg -T | grep -i "killed process"

---

Disk Tools#

Copy
# Disk space usage by filesystem/mount point
df -h

# Disk space usage by DIRECTORY — find what's actually consuming space
du -sh /var/log/* | sort -rh | head -10

# The single most important disk I/O command: shows utilization,
# queue depth, and wait time PER DISK
iostat -xz 1
# Device  r/s  w/s  rkB/s  wkB/s  await  %util
# nvme0n1 45.2 120.5 1802  15420   2.10   85.30

Reading iostat correctly, directly reusing the USE method from the Monitoring Methodologies series: %util is Utilization (how busy the disk is), await (average wait time in milliseconds) is the practical Saturation signal — a high %util with low await can still be healthy (the disk is busy but keeping up); a high await specifically means requests are genuinely queueing and waiting, the real sign of a disk-level bottleneck.

Copy
# A classic, very common "what's filling up my disk" one-liner
du -h --max-depth=1 / 2>/dev/null | sort -rh | head -20

---

Network Tools#

Copy
# All current connections and their TCP states (Part 2)
ss -tan

# Which PROCESS owns which network connection/port — genuinely
# essential for "what's listening on port 8080" type questions
ss -tlnp
# or:
sudo lsof -i :8080

# Live bandwidth usage per network interface
sar -n DEV 1

# Test basic reachability
ping example.com

# Test whether a SPECIFIC port is actually reachable
# (ping alone doesn't confirm a service is listening)
nc -zv example.com 443
telnet example.com 443

Why ping succeeding doesn't mean a service is actually working, a genuinely common, practical distinction: ping only tests basic ICMP reachability at the network layer — a host can respond to ping perfectly while the actual application listening on port 443 is completely down, or while a firewall specifically blocks that one port but allows ICMP through. Always test the actual port/service directly, not just basic host reachability, when diagnosing "is this thing actually up."

---

Process-Level Deep Dive Tools#

Copy
# What files/sockets does this process actually have open?
# (directly reuses the file descriptor discussion from Part 1)
lsof -p <PID>

# What's this process's memory map look like?
cat /proc/<PID>/status | grep -i vm

# What signals is this process currently blocking/handling?
cat /proc/<PID>/status | grep -i sig

# Live view filtered to just this process's resource usage
top -p <PID>

---

Log Tools#

Copy
# On modern systemd-based systems - the primary log query tool
journalctl -u myapp.service --since "10 minutes ago"

# Follow logs live, like tail -f
journalctl -u myapp.service -f

# Kernel ring buffer — the FIRST place to check for OOM kills,
# hardware errors, or kernel-level networking issues
dmesg -T | tail -50

# Classic log file tailing, still extremely common
tail -f /var/log/nginx/access.log

# Search across logs for a specific pattern with context
grep -B2 -A5 "ERROR" /var/log/myapp/app.log | tail -50

---

tcpdump — Watching the Actual Network Traffic#

tcpdump captures raw network packets, letting you see exactly what's actually being sent and received — the closest you can get to ground truth when something seems "off" about network behavior.

Copy
# Capture all traffic on a specific interface
sudo tcpdump -i eth0

# Capture traffic to/from a specific host and port
sudo tcpdump -i eth0 host 10.0.0.5 and port 443

# Capture and save to a file for later analysis (e.g. in Wireshark)
sudo tcpdump -i eth0 -w capture.pcap

# See just TCP handshake/connection-level activity, not full payloads
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0'

A genuinely useful, concrete diagnostic pattern: if a client reports "the connection just hangs," a tcpdump capture can immediately reveal whether a SYN packet even reaches the server at all (a network/firewall problem), whether a SYN-ACK comes back but the connection never completes (often a client-side or middlebox issue), or whether the TCP handshake completes fine but the application-level response never arrives (an application-level problem, not a network one) — each of these looks identical from the application's point of view ("it's hanging") but requires a completely different fix, and only packet-level visibility can definitively tell them apart.

---

strace — Watching a Process Talk to the Kernel#

strace shows every system call a process makes to the kernel — genuinely the deepest level of visibility into "what is this process actually doing right now."

Copy
# Attach to a running process and watch its system calls live
sudo strace -p <PID>

# Focus specifically on file-related system calls
sudo strace -e trace=open,openat,read,write -p <PID>

# Time each system call — reveals EXACTLY where a process
# is spending its time at the syscall level
sudo strace -T -p <PID>

# Trace a command from the very start
strace -f ./myapp

A powerful, genuinely diagnostic example: if a process appears completely "hung," attaching strace and seeing it stuck on a single read() or connect() system call that never returns tells you definitively it's blocked waiting on I/O (a network call, a disk read) — not stuck in an infinite CPU loop, not deadlocked on an internal lock. This single distinction (blocked on I/O vs. burning CPU vs. deadlocked) completely changes the diagnosis and the fix, and strace is often the fastest, most direct way to tell them apart when application-level logs don't make it obvious.

---

A Full Worked Incident, Start to Finish#

Tying everything in this tutorial together into one realistic, narrated investigation — this exact style of walkthrough is precisely what a strong answer to "walk me through how you'd debug a slow server" should sound like.

The report: "The checkout API is responding slowly, intermittently, for the last 20 minutes."

Diagram

Why this worked so cleanly: the investigation followed the exact 60-second triage order — CPU first (load average, confirmed via vmstat's run queue), then checked whether it was evenly distributed (mpstat, ruling out a single hot core), then identified the specific offending process (top), then went one level deeper with strace to understand why that specific process was consuming so much CPU. Each step either ruled something out or narrowed the search — never guessing, always following the data.

---

Building Your Own Cheat Sheet#

A consolidated, memorizable reference — genuinely worth having ready to reproduce from memory in an interview.

Diagram

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: uptime shows a load average of 12 on an 8-core machine, but mpstat -P ALL 1 shows all 8 cores sitting at roughly 30% CPU usage each. What does this combination suggest, and what would you check next?

Answer: This is a classic signature of processes waiting on something OTHER than CPU — likely I/O (disk or network), since CPU cores themselves aren't actually busy computing, yet the run queue (reflected in load average on Linux, which counts both CPU-runnable AND uninterruptible-sleep/I/O-waiting processes) is elevated. I'd check iostat -xz 1 next specifically for high await time (disk I/O saturation), and look for processes in D state (ps aux | awk '$8 ~ /D/') from Part 1 — this pattern points toward a disk or I/O bottleneck, not a CPU bottleneck, despite the elevated load average.

Problem 2: A service intermittently seems to "hang" for a few seconds at a time, with no errors in the application logs. How would you use strace to figure out exactly what's happening during one of those hangs?

Answer: Attach strace -T -p <PID> to the running process (the -T flag times each system call) right as, or just before, a hang is expected/observed, and watch which system call the process is blocked on when it appears to freeze. If it's stuck on a read() or recv() call, that points to it waiting on a slow network response from some dependency. If it's stuck on a connect() call, that points to a connection-establishment problem (possibly DNS resolution or a firewall issue, from Part 2). The timed output also directly shows exactly how long each individual syscall took, pinpointing precisely where the multi-second delay is actually happening at the kernel-interaction level, something application logs alone often can't reveal.

Problem 3: You need to determine whether a client's reported "connection just hangs" issue is a network problem or an application problem, without access to the application's own logs. What tool would you reach for, and what would you look for?

Answer: tcpdump, capturing traffic on the relevant interface filtered to the specific host/port in question. I'd look for whether the TCP three-way handshake (SYN, SYN-ACK, ACK, from Part 2) completes successfully — if the SYN never gets a response, it's a network/firewall/routing problem before the application is even reached; if the handshake completes but no application-level response (e.g., no HTTP response bytes) ever comes back afterward, the connection succeeded at the network level and the problem is specifically in the application itself, not the network. This distinction is invisible at the application's own "it's hanging" level and requires exactly this kind of packet-level visibility to resolve definitively.

---

Summary — The Complete Linux & Networking Series#

• The 60-second triage checklist (uptime -> dmesg -> vmstat -> mpstat -> free -> iostat -> ss -> top) is a systematic, USE-method-based order for investigating any unexplained system issue — follow it in order rather than jumping to a favorite command.
• Read load average relative to core count — the same raw number means very different things on machines with different core counts.
• iostat's await (wait time) is the practical Saturation signal for disk I/O, distinct from %util (Utilization) — exactly the USE method's distinction from the Monitoring Methodologies series, applied concretely.
• ping succeeding does not mean a service is actually working — always test the specific port/application directly.
• strace reveals exactly what system call a process is blocked on, definitively distinguishing "waiting on I/O" from "burning CPU" from "deadlocked" — a distinction application logs often can't provide on their own.
• tcpdump provides ground-truth, packet-level evidence for network issues — the only way to definitively tell apart "the network never delivered the request" from "the network worked fine but the application never responded."
• A strong incident investigation follows the data step by step — each tool either rules out or narrows toward the actual root cause, rather than guessing.

This completes the Linux & Networking Fundamentals series. See questions.md in this folder for the full interview question bank covering all three parts.