# Linux & Networking Fundamentals — Part 3: The Linux Troubleshooting Toolkit > **Series:** Linux & Networking Fundamentals (3 of 7) > **Part 1:** `01-linux-process-and-memory-internals.md` — Process & Memory Internals > **Part 2:** `02-tcp-ip-and-dns.md` — TCP/IP & DNS > **Part 3:** This file — The Linux Troubleshooting Toolkit > **Part 4:** `04-netfilter-iptables-nftables.md` — Netfilter, iptables & nftables > **Part 5:** `05-namespaces-and-virtual-networking.md` — Network Namespaces & Virtual Networking > **Part 6:** `06-systemd-deep-dive.md` — systemd Deep Dive > **Part 7:** `07-ebpf-observability-and-networking.md` — eBPF for Observability & Networking > **Questions:** `questions.md` ## Table of Contents 1. [Why a Toolkit, Not Just a List of Commands](#why-a-toolkit-not-just-a-list-of-commands) 2. [The 60-Second Triage Checklist](#the-60-second-triage-checklist) 3. [CPU Tools](#cpu-tools) 4. [Memory Tools](#memory-tools) 5. [Disk Tools](#disk-tools) 6. [Network Tools](#network-tools) 7. [Process-Level Deep Dive Tools](#process-level-deep-dive-tools) 8. [Log Tools](#log-tools) 9. [tcpdump — Watching the Actual Network Traffic](#tcpdump--watching-the-actual-network-traffic) 10. [strace — Watching a Process Talk to the Kernel](#strace--watching-a-process-talk-to-the-kernel) 11. [A Full Worked Incident, Start to Finish](#a-full-worked-incident-start-to-finish) 12. [Building Your Own Cheat Sheet](#building-your-own-cheat-sheet) 13. [Common Mistakes](#common-mistakes) 14. [Worked Practice Problems](#worked-practice-problems) 15. [Summary — The Complete Linux & Networking Series](#summary--the-complete-linux--networking-series) --- ## Why a Toolkit, Not Just a List of Commands Parts 1 and 2 explained the *concepts* — processes, memory, TCP, DNS. This part is about the actual *commands* you reach for during a real incident, organized the same way the USE method from the Monitoring Methodologies series organizes an investigation: **CPU, then Memory, then Disk, then Network, then process-level detail.** Memorizing commands in isolation is much less useful than knowing which resource each one checks and in what order to reach for them. --- ## The 60-Second Triage Checklist This is directly inspired by Brendan Gregg's well-known "Linux Performance Analysis in 60 Seconds" checklist, already referenced in the Monitoring Methodologies series — here it is in full, hands-on detail, as the backbone of this entire tutorial. ```mermaid flowchart TD A["1. uptime
(load average - overall feel)"] --> B["2. dmesg -T | tail
(any recent kernel errors,
OOM kills?)"] B --> C["3. vmstat 1
(CPU, run queue, swap
activity, at a glance)"] C --> D["4. mpstat -P ALL 1
(per-core CPU detail)"] D --> E["5. free -h
(memory overview)"] E --> F["6. iostat -xz 1
(disk I/O detail)"] F --> G["7. ss -tan
(network connection states)"] G --> H["8. top / htop
(which PROCESS is
responsible)"] ``` **Why running these in this specific order matters:** each command takes seconds to run and immediately either rules out or points toward a category of problem — by the time you've run all eight (genuinely achievable in under a minute with practice), you have a full USE-method picture across CPU, memory, disk, and network, and usually already know which specific area deserves deeper investigation. --- ## CPU Tools ```bash # Quick overview: load average over 1/5/15 minutes uptime # 14:32:01 up 10 days, 3:14, 2 users, load average: 8.42, 6.15, 4.03 ``` **Reading load average correctly is a genuinely common interview question.** Load average roughly represents the average number of processes wanting CPU time (running + waiting) over the last 1/5/15 minutes. **The critical context it needs: how many CPU cores does the machine have?** A load average of 8 is perfectly healthy on a 16-core machine (half utilized) but a serious red flag on a 4-core machine (2x oversubscribed). ```bash # See CPU usage broken down PER CORE — reveals uneven load # (e.g. one core pegged at 100% while others sit idle, often # a sign of a single-threaded bottleneck) mpstat -P ALL 1 # Live, refreshing view of overall system activity vmstat 1 # procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- # r b swpd free buff cache si so bi bo in cs us sy id wa st # 12 0 0 812340 45012 891200 0 0 45 120 3200 8500 45 12 40 3 0 ``` **The `r` (run queue) column in `vmstat` is exactly the CPU saturation metric referenced throughout the Monitoring Methodologies series** — if `r` consistently exceeds the number of CPU cores, processes are genuinely waiting for a core, not just using one efficiently. --- ## Memory Tools ```bash # Overview — always check "available," not "free" (Part 1's lesson) free -h # Per-process memory usage, sorted by usage ps aux --sort=-%mem | head -10 # The single best live view: per-process memory AND CPU together top htop ``` ```bash # Check for recent OOM kills — the single most important # memory-related log check, directly from Part 1 dmesg -T | grep -i "killed process" ``` --- ## Disk Tools ```bash # Disk space usage by filesystem/mount point df -h # Disk space usage by DIRECTORY — find what's actually consuming space du -sh /var/log/* | sort -rh | head -10 # The single most important disk I/O command: shows utilization, # queue depth, and wait time PER DISK iostat -xz 1 # Device r/s w/s rkB/s wkB/s await %util # nvme0n1 45.2 120.5 1802 15420 2.10 85.30 ``` **Reading `iostat` correctly, directly reusing the USE method from the Monitoring Methodologies series:** `%util` is Utilization (how busy the disk is), `await` (average wait time in milliseconds) is the practical Saturation signal — a high `%util` with low `await` can still be healthy (the disk is busy but keeping up); a high `await` specifically means requests are genuinely queueing and waiting, the real sign of a disk-level bottleneck. ```bash # A classic, very common "what's filling up my disk" one-liner du -h --max-depth=1 / 2>/dev/null | sort -rh | head -20 ``` --- ## Network Tools ```bash # All current connections and their TCP states (Part 2) ss -tan # Which PROCESS owns which network connection/port — genuinely # essential for "what's listening on port 8080" type questions ss -tlnp # or: sudo lsof -i :8080 # Live bandwidth usage per network interface sar -n DEV 1 # Test basic reachability ping example.com # Test whether a SPECIFIC port is actually reachable # (ping alone doesn't confirm a service is listening) nc -zv example.com 443 telnet example.com 443 ``` **Why `ping` succeeding doesn't mean a service is actually working, a genuinely common, practical distinction:** `ping` only tests basic ICMP reachability at the network layer — a host can respond to ping perfectly while the actual application listening on port 443 is completely down, or while a firewall specifically blocks that one port but allows ICMP through. **Always test the actual port/service directly**, not just basic host reachability, when diagnosing "is this thing actually up." --- ## Process-Level Deep Dive Tools ```bash # What files/sockets does this process actually have open? # (directly reuses the file descriptor discussion from Part 1) lsof -p # What's this process's memory map look like? cat /proc//status | grep -i vm # What signals is this process currently blocking/handling? cat /proc//status | grep -i sig # Live view filtered to just this process's resource usage top -p ``` --- ## Log Tools ```bash # On modern systemd-based systems - the primary log query tool journalctl -u myapp.service --since "10 minutes ago" # Follow logs live, like tail -f journalctl -u myapp.service -f # Kernel ring buffer — the FIRST place to check for OOM kills, # hardware errors, or kernel-level networking issues dmesg -T | tail -50 # Classic log file tailing, still extremely common tail -f /var/log/nginx/access.log # Search across logs for a specific pattern with context grep -B2 -A5 "ERROR" /var/log/myapp/app.log | tail -50 ``` --- ## tcpdump — Watching the Actual Network Traffic **`tcpdump`** captures raw network packets, letting you see exactly what's actually being sent and received — the closest you can get to ground truth when something seems "off" about network behavior. ```bash # Capture all traffic on a specific interface sudo tcpdump -i eth0 # Capture traffic to/from a specific host and port sudo tcpdump -i eth0 host 10.0.0.5 and port 443 # Capture and save to a file for later analysis (e.g. in Wireshark) sudo tcpdump -i eth0 -w capture.pcap # See just TCP handshake/connection-level activity, not full payloads sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0' ``` **A genuinely useful, concrete diagnostic pattern:** if a client reports "the connection just hangs," a `tcpdump` capture can immediately reveal whether a SYN packet even reaches the server at all (a network/firewall problem), whether a SYN-ACK comes back but the connection never completes (often a client-side or middlebox issue), or whether the TCP handshake completes fine but the application-level response never arrives (an application-level problem, not a network one) — each of these looks identical from the *application's* point of view ("it's hanging") but requires a completely different fix, and only packet-level visibility can definitively tell them apart. --- ## strace — Watching a Process Talk to the Kernel **`strace`** shows every system call a process makes to the kernel — genuinely the deepest level of visibility into "what is this process actually doing right now." ```bash # Attach to a running process and watch its system calls live sudo strace -p # Focus specifically on file-related system calls sudo strace -e trace=open,openat,read,write -p # Time each system call — reveals EXACTLY where a process # is spending its time at the syscall level sudo strace -T -p # Trace a command from the very start strace -f ./myapp ``` **A powerful, genuinely diagnostic example:** if a process appears completely "hung," attaching `strace` and seeing it stuck on a single `read()` or `connect()` system call that never returns tells you definitively it's blocked waiting on I/O (a network call, a disk read) — not stuck in an infinite CPU loop, not deadlocked on an internal lock. **This single distinction (blocked on I/O vs. burning CPU vs. deadlocked) completely changes the diagnosis and the fix**, and `strace` is often the fastest, most direct way to tell them apart when application-level logs don't make it obvious. --- ## A Full Worked Incident, Start to Finish Tying everything in this tutorial together into one realistic, narrated investigation — this exact style of walkthrough is precisely what a strong answer to "walk me through how you'd debug a slow server" should sound like. **The report:** "The checkout API is responding slowly, intermittently, for the last 20 minutes." ```mermaid flowchart TD A["1. uptime: load average
18 on an 8-core box —
clearly overloaded"] --> B["2. dmesg -T | tail:
no OOM kills, no kernel
errors — rules out memory
exhaustion and hardware issues"] B --> C["3. vmstat 1: 'r' column
consistently around 15-20
— confirms genuine CPU
saturation, processes
waiting for a core"] C --> D["4. mpstat -P ALL 1: ALL
cores roughly evenly loaded
— NOT a single-threaded
bottleneck on one core"] D --> E["5. top: ONE specific process
(a log-shipping agent) is
consuming 400% CPU —
far more than expected"] E --> F["6. strace -p <PID> on that
process: shows it stuck in
a tight loop repeatedly
re-reading a log file that's
growing extremely fast"] F --> G["Root cause found: a
RECENT deploy introduced a
bug causing verbose,
excessive logging, and the
log-shipping agent trying
to keep up is consuming
most of the machine's CPU,
starving the actual
checkout API"] ``` **Why this worked so cleanly:** the investigation followed the exact 60-second triage order — CPU first (load average, confirmed via `vmstat`'s run queue), then checked whether it was evenly distributed (`mpstat`, ruling out a single hot core), then identified the specific offending process (`top`), then went one level deeper with `strace` to understand *why* that specific process was consuming so much CPU. **Each step either ruled something out or narrowed the search — never guessing, always following the data.** --- ## Building Your Own Cheat Sheet A consolidated, memorizable reference — genuinely worth having ready to reproduce from memory in an interview. ```mermaid graph TD Cheat["The 60-Second Toolkit"] --> CPU["CPU: uptime, vmstat 1,
mpstat -P ALL 1"] Cheat --> Mem["Memory: free -h,
dmesg | grep -i oom"] Cheat --> Disk["Disk: iostat -xz 1,
df -h, du -sh"] Cheat --> Net["Network: ss -tan,
ss -tlnp, tcpdump"] Cheat --> Proc["Process: top/htop,
lsof -p, strace -p"] Cheat --> Logs["Logs: journalctl -f,
dmesg -T"] ``` | Question | Command | |---|---| | Is the system generally overloaded? | `uptime` | | Any recent OOM kills or kernel errors? | `dmesg -T \| tail` | | Is CPU actually saturated (run queue)? | `vmstat 1` | | Is one core hot, or evenly spread? | `mpstat -P ALL 1` | | How much memory is really free? | `free -h` | | Is disk I/O the bottleneck? | `iostat -xz 1` | | What's filling up disk space? | `du -sh /* \| sort -rh` | | What are current network connections doing? | `ss -tan` | | What's listening on a given port? | `ss -tlnp` or `lsof -i :PORT` | | Which process is actually responsible? | `top` / `htop` | | What files/sockets does a process have open? | `lsof -p PID` | | What is a process actually doing right now? | `strace -p PID` | | What's actually on the wire? | `tcpdump -i eth0` | --- ## Common Mistakes | Mistake | Why It's Wrong | Fix | |---|---|---| | Jumping straight to a favorite/familiar command instead of following a systematic order | Risks tunnel vision, missing the actual bottleneck if it's somewhere unexpected | Follow the 60-second triage order (CPU -> memory -> disk -> network -> process detail), exactly like the USE method | | Treating `ping` success as proof a service is up | Ping only tests basic network reachability, not whether the actual application/port is working | Test the specific port/service directly (`nc -zv`, `curl`), not just host-level ping | | Reading load average without knowing the core count | A load average of 8 means very different things on a 4-core vs. 32-core machine | Always divide load average by core count to judge whether it's actually high | | Skipping `dmesg`/kernel logs early in an investigation | Misses OOM kills and hardware-level errors that can explain the whole incident immediately | Check `dmesg -T \| tail` early, as part of the very first triage pass | | Guessing at a root cause without confirming with a deeper tool (`strace`, `tcpdump`) | Can lead to fixing the wrong thing, wasting time during a real incident | Use `strace`/`tcpdump` to get direct, ground-truth evidence before committing to a fix | --- ## Worked Practice Problems **Problem 1:** `uptime` shows a load average of 12 on an 8-core machine, but `mpstat -P ALL 1` shows all 8 cores sitting at roughly 30% CPU usage each. What does this combination suggest, and what would you check next? *Answer:* This is a classic signature of processes waiting on something OTHER than CPU — likely I/O (disk or network), since CPU cores themselves aren't actually busy computing, yet the run queue (reflected in load average on Linux, which counts both CPU-runnable AND uninterruptible-sleep/I/O-waiting processes) is elevated. I'd check `iostat -xz 1` next specifically for high `await` time (disk I/O saturation), and look for processes in D state (`ps aux | awk '$8 ~ /D/'`) from Part 1 — this pattern points toward a disk or I/O bottleneck, not a CPU bottleneck, despite the elevated load average. **Problem 2:** A service intermittently seems to "hang" for a few seconds at a time, with no errors in the application logs. How would you use `strace` to figure out exactly what's happening during one of those hangs? *Answer:* Attach `strace -T -p ` to the running process (the `-T` flag times each system call) right as, or just before, a hang is expected/observed, and watch which system call the process is blocked on when it appears to freeze. If it's stuck on a `read()` or `recv()` call, that points to it waiting on a slow network response from some dependency. If it's stuck on a `connect()` call, that points to a connection-establishment problem (possibly DNS resolution or a firewall issue, from Part 2). The timed output also directly shows exactly how long each individual syscall took, pinpointing precisely where the multi-second delay is actually happening at the kernel-interaction level, something application logs alone often can't reveal. **Problem 3:** You need to determine whether a client's reported "connection just hangs" issue is a network problem or an application problem, without access to the application's own logs. What tool would you reach for, and what would you look for? *Answer:* `tcpdump`, capturing traffic on the relevant interface filtered to the specific host/port in question. I'd look for whether the TCP three-way handshake (SYN, SYN-ACK, ACK, from Part 2) completes successfully — if the SYN never gets a response, it's a network/firewall/routing problem before the application is even reached; if the handshake completes but no application-level response (e.g., no HTTP response bytes) ever comes back afterward, the connection succeeded at the network level and the problem is specifically in the application itself, not the network. This distinction is invisible at the application's own "it's hanging" level and requires exactly this kind of packet-level visibility to resolve definitively. --- ## Summary — The Complete Linux & Networking Series - The **60-second triage checklist** (`uptime` -> `dmesg` -> `vmstat` -> `mpstat` -> `free` -> `iostat` -> `ss` -> `top`) is a systematic, USE-method-based order for investigating any unexplained system issue — follow it in order rather than jumping to a favorite command. - Read **load average relative to core count** — the same raw number means very different things on machines with different core counts. - `iostat`'s `await` (wait time) is the practical **Saturation** signal for disk I/O, distinct from `%util` (Utilization) — exactly the USE method's distinction from the Monitoring Methodologies series, applied concretely. - **`ping` succeeding does not mean a service is actually working** — always test the specific port/application directly. - **`strace`** reveals exactly what system call a process is blocked on, definitively distinguishing "waiting on I/O" from "burning CPU" from "deadlocked" — a distinction application logs often can't provide on their own. - **`tcpdump`** provides ground-truth, packet-level evidence for network issues — the only way to definitively tell apart "the network never delivered the request" from "the network worked fine but the application never responded." - A strong incident investigation follows the data step by step — each tool either **rules out** or **narrows toward** the actual root cause, rather than guessing. This completes the **Linux & Networking Fundamentals** series. See `questions.md` in this folder for the full interview question bank covering all three parts.