# Linux Troubleshooting Cheat Sheet — Network & Process Inspection > **Tool:** ss, netstat, lsof, strace, tcpdump, iptables, nft, dig, nslookup, iostat, vmstat > **Category:** Linux & Troubleshooting > **Verified against:** ss (iproute2, Ubuntu 24.04), lsof 4.95.0, strace 6.8, tcpdump 4.99.4, iptables 1.8.10 (nf_tables), nftables 1.0.9, dig/nslookup 9.18.39, sysstat (iostat) 12.6.1, vmstat (procps) — all flags verified via ` --help`/` -h`, `man strace`, and `man nft` run locally; netstat verified via man7.org/linux/man-pages/man8/netstat.8.html only (net-tools not installed in this environment), 2026-08-21 > **Official docs:** https://man7.org/linux/man-pages/ Finding what's listening on a port, what a process has open, what syscalls it's making, and what's actually on the wire. ## Sockets and listening ports — ss (modern) and netstat (legacy) ```bash ss -tulpn # all TCP+UDP listening sockets, with PID/program, numeric ports ss -tan state established # established TCP connections only ss -s # summary counts by protocol/state netstat -tulpn # the older equivalent of the ss command above netstat -r # kernel routing table ``` `ss` is the current tool — it reads directly from the kernel and is significantly faster on a host with many connections; `netstat` (from `net-tools`) is legacy and not installed by default on many modern distros, including this one. Know both: `ss` for anything you run yourself, `netstat` for reading someone else's old runbook or a minimal/legacy host where `ss` isn't available either. ## What's using a file, port, or directory — lsof ```bash lsof -i :8080 # what process (if any) is bound to port 8080 lsof -p 12345 # every file descriptor a specific PID has open lsof -u myuser # every open file belonging to a user lsof +D /var/log # every process with an open file under a directory ``` `lsof -i :PORT` is usually the fastest way to answer "what's already using this port" when a service fails to bind on startup — faster than cross-referencing `ss -tlpn` output by eye. ## Tracing syscalls — strace ```bash strace -p 12345 # attach to a running process and watch its syscalls live strace -f -p 12345 # + follow any child processes it forks strace -c -p 12345 # summary: syscall counts and time spent, not a live stream strace -e trace=network -p 12345 # only network-related syscalls (connect, accept, sendto, ...) strace -tt -o trace.log myprogram arg1 # run a fresh command under strace, with timestamps, to a file ``` `strace` adds real overhead to the traced process — fine for a one-off diagnostic attach, but not something to leave running against a production process under load without deciding that tradeoff deliberately first. ## Capturing packets — tcpdump ```bash tcpdump -i eth0 # capture on a specific interface tcpdump -i eth0 -n # -n: don't resolve hostnames (faster, avoids DNS noise in output) tcpdump -i eth0 port 443 tcpdump -i eth0 host 10.0.1.5 and port 443 tcpdump -i eth0 -w capture.pcap # write raw packets to a file for later analysis (e.g. in Wireshark) tcpdump -i eth0 -c 100 -X # capture exactly 100 packets, with hex+ASCII payload dump ``` Capturing on the wrong interface (`eth0` vs a container's `veth` vs `lo`) is the most common reason "tcpdump shows nothing" during an actual incident — `tcpdump -D` lists every available interface if you're not sure which one carries the traffic you're chasing. ## Blocking and inspecting traffic — iptables and nftables iptables is the legacy packet-filtering interface; nftables (`nft`) is its modern replacement. On this host `iptables -V` reports `nf_tables` as the backend, meaning both tools ultimately manage the same underlying kernel ruleset. ```bash iptables -L -n -v # list all rules in the filter table, numeric, with packet/byte counters iptables -L -n --line-numbers # same, with rule numbers (needed to target -D by position) iptables -A INPUT -p tcp --dport 22 -j ACCEPT # append a rule to the INPUT chain iptables -D INPUT 3 # delete rule #3 from INPUT iptables -P INPUT DROP # change a chain's default policy nft list ruleset # show the entire nftables ruleset, all tables/chains nft add table inet mytable # create a new table (inet = both IPv4 and IPv6) nft add chain inet mytable input '{ type filter hook input priority filter ; }' nft add rule inet mytable input tcp dport 22 accept # add a rule to the chain nft -a list ruleset # include rule handles, needed to target a specific rule for deletion ``` Don't assume every host's `iptables` is nf_tables-backed like this one — some distros still ship the legacy `iptables-legacy` backend, where `iptables` and `nft` manage genuinely separate rulesets that can't see each other. Check `iptables -V` first. ## DNS lookups — dig and nslookup ```bash dig example.com # full DNS answer, authority, and additional sections dig example.com +short # just the answer, one line dig example.com MX # query a specific record type dig @8.8.8.8 example.com # query a specific nameserver directly, bypassing local resolver config dig -x 93.184.216.34 # reverse lookup (PTR record) dig example.com +trace # trace the full delegation path from the root nameservers down nslookup example.com # quick forward lookup using the system resolver nslookup example.com 8.8.8.8 # query a specific nameserver ``` `dig` is the more capable tool for real troubleshooting — it exposes TTLs, which nameserver actually answered, and `+trace` for delegation problems. `nslookup` is faster to type for a quick sanity check but BIND's own docs point users toward `dig`/`host` instead. ## System-wide performance — iostat and vmstat ```bash iostat -x 2 # extended per-device stats (%util, await, queue depth), every 2s iostat -d -x sda 5 3 # extended stats for one device only, every 5s, 3 samples vmstat 2 5 # 5 samples of memory/swap/IO/CPU summary, 2s apart vmstat -a # active vs inactive memory instead of the default free/buff/cache split vmstat -s # cumulative event counters since boot, not a live sample ``` `iostat -x`'s `%util` column is usually the fastest way to tell "is this disk actually the bottleneck" — it approaches 100% when the device is saturated, regardless of raw throughput, which kB/s numbers alone don't tell you on their own. ## Deeper strace usage ```bash strace -e trace=%file -p 12345 # only file-related syscalls (open, stat, unlink, ...) strace -T -p 12345 # show time spent inside each syscall strace -y -p 12345 # resolve file descriptors to their paths inline in the output strace -c -e trace=%file myprogram # syscall-count summary, filtered to the file group, for a fresh command ``` `-e trace=%GROUP` (e.g. `%file`, `%network`, `%process`, `%signal`) is the fast way to cut a noisy trace down to the syscall class you actually care about — check `strace -e trace=?` (or the strace(1) man page) for the full, current group list, since group membership has changed across strace releases. ## Inspecting a running process via /proc ```bash cat /proc//status # human-readable state, memory, thread count, uid/gid cat /proc//limits # the process's actual resolved ulimits ls -l /proc//fd # every open file descriptor, as symlinks to what they point to cat /proc//cmdline | tr '\0' ' ' # exact command line it was started with (NUL-separated, tr fixes it for display) cat /proc//environ | tr '\0' '\n' # its environment variables at start time cat /proc/meminfo # system-wide memory stats (what free/vmstat parse) cat /proc/loadavg # the 1/5/15-minute load averages ``` `/proc//limits` is the one to check when a process is hitting a "too many open files" or similar resource error — it shows the limits actually in effect for that specific process, which can differ from your own shell's `ulimit -a` if the process was started by a different parent (systemd, cron, a container runtime) with its own limits configured.