# Interview Questions: Linux & Networking Fundamentals

Companion question bank for the 7-part tutorial series in this folder:
`01-linux-process-and-memory-internals.md`, `02-tcp-ip-and-dns.md`, `03-linux-troubleshooting-toolkit.md`,
`04-netfilter-iptables-nftables.md`, `05-namespaces-and-virtual-networking.md`, `06-systemd-deep-dive.md`,
`07-ebpf-observability-and-networking.md`.

Answers are short and plain — expand out loud using the diagrams and commands in the tutorials.

---

# Part 1 Questions: Process & Memory Internals

### 1. What's the difference between a process and a thread?
A process has its own private memory and file descriptors — isolated, heavier to create. A thread lives inside a process and shares that process's memory with sibling threads — lightweight, but a bug in one thread can affect the whole process.

### 2. What does "D state" (uninterruptible sleep) mean, and why is it a red flag?
The process is waiting on I/O in a way that can't even be interrupted by a signal — not even SIGKILL can end it until the I/O completes. A process stuck in D state for a long time usually points to a hung disk, a failing network filesystem, or a driver issue, not an application bug.

### 3. What's a zombie process, and is it a resource leak?
A process that's finished running but whose parent hasn't yet collected its exit status. It's not really consuming meaningful resources — a large accumulation of zombies points to a buggy parent process that spawns children but never reaps them.

### 4. What's the difference between SIGTERM and SIGKILL, and why does Kubernetes send them in that order?
SIGTERM politely asks a process to shut down and can be caught/handled to finish in-flight work cleanly. SIGKILL forcibly, immediately terminates it with zero chance to clean up. Kubernetes sends SIGTERM first (graceful shutdown window) and only escalates to SIGKILL after a grace period, so a well-behaved app can drain requests cleanly.

### 5. Why should you check "available" instead of "free" in `free -h` output?
"Free" memory looks low mostly because of reclaimable disk cache (buff/cache), which isn't a real problem and gets reclaimed instantly when needed. "Available" already accounts for that reclaimable cache and reflects true usable memory.

### 6. Why is swap activity treated as a warning sign rather than "the system self-healing"?
Swapping avoids an immediate OOM crash by moving memory to disk, but disk is 100-1000x slower than RAM — a process that starts swapping can appear to hang with confusing latency instead of failing cleanly. Any non-zero swap activity is worth investigating.

### 7. Walk through the OOM Killer mechanism.
When RAM and swap are both exhausted, the kernel scores every process by an oom_score (roughly, memory usage) and kills the highest-scoring one to free memory and keep the rest of the system alive — with zero warning to that specific process.

### 8. How does this connect to a Kubernetes pod showing "OOMKilled"?
It's the exact same kernel OOM Killer mechanism, but scoped to the container's cgroup memory limit (set via `resources.limits.memory`). Exceed that limit, and the kernel kills the process within that cgroup specifically — that's mechanically what "OOMKilled" means.

### 9. What do namespaces do, and what do cgroups do, and how do they relate to containers?
Namespaces give a process its own isolated VIEW of a resource (its own PIDs, network interfaces, filesystem view). Cgroups control HOW MUCH of a resource a process can actually USE (CPU, memory, I/O limits). A container is just a regular Linux process combined with both — there's no separate "container" object at the kernel level.

### 10. Why are containers less isolated than a true virtual machine?
Containers share the host kernel — namespaces create a convincing illusion of isolation, not physical separation, and a VM has its own separate kernel with hardware-level (hypervisor) isolation. A serious kernel vulnerability could theoretically let a process "escape" a namespace in a way a VM's isolation would prevent.

---

# Part 2 Questions: TCP/IP & DNS

### 11. What's the fundamental tradeoff between TCP and UDP?
TCP is reliable and ordered (guarantees delivery, retransmits lost packets) but has more overhead. UDP is fire-and-forget — fast and low-overhead, but no delivery or ordering guarantee at all.

### 12. Why does DNS mostly use UDP?
DNS queries are small, and if one is lost, the client just retries quickly — the overhead of a full TCP handshake for every tiny lookup would be wasteful. (DNS falls back to TCP for larger responses.)

### 13. Walk through the TCP three-way handshake and explain why it needs exactly three steps.
SYN (client: I'd like to connect) -> SYN-ACK (server: OK, let's connect) -> ACK (client: confirmed). Three steps are needed because both sides must confirm they can BOTH send and receive — after all three, both directions have been mutually verified.

### 14. What is TIME_WAIT, why does it exist, and what real problem can it cause?
After closing a connection, the side that initiated the close holds it in TIME_WAIT for 60-120 seconds to safely discard any delayed duplicate packets from that old connection. Under very high connection churn (e.g., load testing with short-lived connections), this can exhaust available local ports, making a perfectly healthy server look broken from the client's side.

### 15. Why does TLS add real latency on top of TCP?
It requires its own additional round-trips (exchanging supported ciphers, verifying the certificate, deriving a shared key) on top of the TCP handshake — this is why connection keep-alive and TLS session resumption matter for real-world performance.

### 16. Walk through the full DNS resolution journey for a brand-new, uncached domain.
Browser checks its own cache, then the OS resolver's cache, then queries a recursive resolver. If not cached there either, the recursive resolver asks a root server (who handles .com?), then the TLD server (who handles example.com?), then the authoritative server (what's the actual IP?) — and caches the answer along the way back.

### 17. What's the difference between a recursive resolver and an authoritative DNS server?
A recursive resolver does the lookup legwork on a client's behalf, walking the hierarchy and caching results. An authoritative server holds the actual, canonical DNS records for a specific domain.

### 18. What's TTL, and what's the practical tactic around it before a planned migration?
Time To Live — how long a resolver is allowed to cache a DNS record before re-checking. Before a planned cutover, lower the TTL well in advance so existing longer-TTL caches expire and get replaced with the new short TTL, meaning the actual cutover propagates quickly instead of some clients being stuck on the old IP for up to a full day.

### 19. Why doesn't a successful ping guarantee a service is actually working?
Ping only tests basic ICMP network-layer reachability — a host can respond to ping while the actual application/port is completely down, or while a firewall blocks that specific port but allows ICMP through.

---

# Part 3 Questions: The Linux Troubleshooting Toolkit

### 20. Walk through the 60-second triage checklist in order.
uptime (overall load feel) -> dmesg -T | tail (recent kernel errors/OOM kills) -> vmstat 1 (CPU, run queue, swap at a glance) -> mpstat -P ALL 1 (per-core detail) -> free -h (memory) -> iostat -xz 1 (disk I/O) -> ss -tan (network connections) -> top/htop (which process is responsible).

### 21. Why does load average need to be interpreted relative to core count?
A load average of 8 is healthy on a 16-core machine but a serious red flag on a 4-core machine — the raw number alone doesn't tell you if the system is actually oversubscribed.

### 22. In `iostat -xz`, what's the difference between `%util` and `await`, and which one is the real saturation signal?
`%util` is Utilization — how busy the disk is. `await` (average wait time) is the practical Saturation signal — a high `%util` with low `await` can still be healthy (busy but keeping up); a high `await` means requests are genuinely queueing and waiting.

### 23. `uptime` shows a high load average, but `mpstat` shows all cores at only 30% CPU usage. What does this combination suggest?
Processes are likely waiting on something OTHER than CPU — most likely disk or network I/O (Linux load average counts uninterruptible-sleep/I/O-waiting processes too, not just CPU-runnable ones). Check `iostat` for high `await`, and look for processes in D state.

### 24. What does `strace -p <PID>` actually show you, and why is it so useful?
Every system call the process makes to the kernel, live. It can definitively distinguish a process blocked on I/O (stuck in `read()`/`connect()`) from one burning CPU in a loop from one deadlocked on an internal lock — a distinction application logs often can't reveal on their own.

### 25. What does `tcpdump` let you determine that application-level "it's hanging" reports cannot?
Whether the TCP handshake (SYN/SYN-ACK/ACK) even completes at all — telling apart a pure network/firewall problem (SYN never gets a response) from an application-level problem (handshake succeeds, but no application response ever comes back). Both look identical as "it's hanging" from the outside.

### 26. What's the first thing to check in `dmesg` during almost any unexplained incident investigation, and why check it early?
Recent OOM kills (`dmesg -T | grep -i "killed process"`) and kernel-level errors — checking this early can immediately explain an entire incident (a process was killed for memory, not because of an application bug) before spending time investigating the wrong layer.

---

# Part 4 Questions: Netfilter, iptables & nftables

### 27. What is netfilter, and how does it relate to iptables and nftables?
Netfilter is the actual in-kernel hook framework — iptables and nftables are both userspace frontends that program the same underlying netfilter hooks. As of 2026, nftables is the default framework on every major distro, with `iptables` commonly implemented as a compatibility shim translating to it.

### 28. Name the five netfilter hooks in order for an inbound packet destined for the local host.
PREROUTING (before the routing decision) -> INPUT (destined for a local process) -> the local process. FORWARD and OUTPUT are the other two hooks, for routed-through and locally-generated traffic respectively.

### 29. Why does a DNAT rule belong in PREROUTING while an SNAT/MASQUERADE rule belongs in POSTROUTING?
DNAT rewrites the destination and must happen before the routing decision, since the decision depends on the correct destination. SNAT/MASQUERADE rewrites the source and must happen after routing, since the correct source to rewrite to often depends on which interface was actually chosen.

### 30. What does conntrack do, and why is a single `ESTABLISHED,RELATED` rule sufficient to allow all legitimate response traffic?
Conntrack tracks connection state in the kernel. Once a connection is tracked, its response traffic automatically matches the `ESTABLISHED,RELATED` state, so one rule covers all legitimate responses — without conntrack, every possible response would need its own explicit rule.

### 31. Why is iptables rule order critical, and what's the most common resulting mistake?
Rules are evaluated top to bottom, first-match-wins — a broad ACCEPT rule placed before a more specific DROP rule makes that DROP rule a complete, silent no-op, since it's never even evaluated.

### 32. Name two genuine architectural improvements nftables has over iptables.
Native sets/maps (no separate `ipset` package needed) and a unified `inet` address family covering both IPv4 and IPv6 in one ruleset — plus atomic ruleset loading via `nft -f`.

### 33. How does kube-proxy's iptables mode implement a Kubernetes Service, concretely?
It generates real DNAT rules in the `nat` table, rewriting the Service's stable virtual IP to one of its actual backing Pod IPs — the exact same DNAT mechanism covered generally in this chapter, just auto-generated by kube-proxy.

---

# Part 5 Questions: Network Namespaces & Virtual Networking

### 34. What is a Linux network namespace, precisely?
A kernel feature isolating the network stack — interfaces, routing table, and netfilter state — for a set of processes. It's the same kernel, with one specific global resource partitioned, not a separate kernel or VM.

### 35. What is a veth pair, and why must it always be created as a pair?
Two permanently-linked virtual Ethernet interfaces, functioning like a virtual cable — traffic entering either end always emerges from the other. They're always created together because the kernel has no concept of a single, unpaired veth interface.

### 36. What does a Linux bridge do, and can a real physical NIC be attached to one?
It acts as a virtual switch, letting many interfaces communicate as if on the same Ethernet segment. Yes — a real physical NIC can be attached alongside veth ends, which is exactly what provides bridged namespaces with real internet access.

### 37. What does VXLAN actually encapsulate, and why does that matter for MTU?
An entire Ethernet frame, inside a UDP packet, adding roughly 50 bytes of overhead. A standard 1500-byte frame plus that overhead can exceed the underlay network's own MTU, causing fragmentation or silent drops unless the overlay interface's MTU is explicitly reduced (typically to 1450).

### 38. How does Calico's routed (BGP) mode differ from a VXLAN-based overlay?
It still uses veth pairs at the Pod level, but instead of encapsulating cross-node traffic in VXLAN, it configures each node's real routing table and uses BGP to propagate routes — avoiding encapsulation overhead entirely, at the cost of requiring genuine BGP support from the underlying network.

### 39. What is "a container," concretely, from the Linux kernel's own point of view?
A process (or group of processes) running inside a specific combination of namespaces (net, pid, mnt, uts, ipc, often user) plus cgroup resource limits — there is no separate "container" object at the kernel level at all.

### 40. Why does every container in the same Kubernetes Pod share one IP address?
Every container in a Pod joins the SAME network namespace, held open by a hidden "pause" container — the namespace, not any individual application container, owns the Pod's IP, which is why restarting one container doesn't change the Pod's IP.

---

# Part 6 Questions: systemd Deep Dive

### 41. Why does editing a unit file have no effect until you run a specific command?
systemd caches unit file contents in memory — `systemctl daemon-reload` is required to make it re-read the actual files on disk before an edit takes effect.

### 42. Distinguish dependency ORDERING from dependency STRENGTH in systemd, and name the recommended default pairing.
Ordering (`After=`/`Before=`) controls WHEN a unit starts relative to another, saying nothing about whether it starts at all. Strength (`Wants=`/`Requires=`) controls WHETHER it starts at all. `Wants=`+`After=` together is the recommended default for most service relationships.

### 43. Why should a service needing real network connectivity depend on `network-online.target` rather than `network.target`?
`network.target` only means the network management service has started, not that any interface has a usable IP yet — depending on it alone can cause a service to start before DHCP completes, producing an intermittent, boot-timing-dependent failure.

### 44. What is socket activation, and what course concept does it directly parallel?
systemd creates and listens on a service's socket immediately, but defers starting the actual service process until the first real connection arrives — zero idle resource cost with no perceptible first-connection delay, the same scale-to-zero concept covered for Kubernetes runner infrastructure elsewhere in this course.

### 45. How does systemd relate to the cgroup v2 hierarchy, and what mechanism lets a container runtime manage its own cgroups?
systemd is the exclusive, kernel-enforced manager of cgroup v2 on modern distros. Cgroup delegation (`Delegate=yes`) explicitly hands a unit (e.g. `containerd.service`) authority over its own cgroup sub-tree, resolving the conflict without violating systemd's own ownership.

### 46. What does `Restart=on-failure` fail to catch, and what closes that gap?
A process that's hung/deadlocked but hasn't actually exited — `Restart=on-failure` only reacts to an actual exit. A watchdog (`WatchdogSec=` + `Restart=on-watchdog`), requiring the app to periodically call `sd_notify(WATCHDOG=1)`, catches this instead.

---

# Part 7 Questions: eBPF for Observability & Networking

### 47. What does the eBPF verifier actually guarantee, and what happens to a program that fails it?
It statically proves a program cannot crash the kernel — no unbounded loops, no out-of-bounds memory access, a bounded instruction count — before the program is ever allowed to load. A program that fails simply never runs at all; there's no partial execution.

### 48. Why is eBPF fast — what specifically makes it different from an interpreted approach?
JIT (Just-In-Time) compilation to real, native machine code for the running CPU architecture, rather than interpretation — enabling near-native execution speed even on very high event rates like every packet or every syscall.

### 49. What problem does CO-RE (Compile Once, Run Everywhere) solve, and what mechanism enables it?
Kernel struct layouts can differ between kernel versions — CO-RE lets one compiled eBPF program run correctly across different kernels by using BTF (BPF Type Format) metadata to resolve struct field offsets at load time, rather than baking them in at compile time.

### 50. Why is Cilium's eBPF-based Service routing genuinely, algorithmically faster than kube-proxy's iptables mode at scale?
kube-proxy's iptables mode is a linear, first-match-wins rule chain — O(n) lookup cost that grows with Service count. Cilium replaces this with an eBPF hash-map lookup keyed by virtual IP — a genuine O(1) operation, independent of cluster size.

### 51. How does Cilium's NetworkPolicy enforcement differ architecturally from an iptables-based CNI plugin's?
It derives a stable "identity" from a Pod's own labels and enforces policy based on that identity via an eBPF map, rather than generating IP-address-based rules that would need constant regeneration as Pod IPs churn.

### 52. What's the honest, bounded scope of "zero-instrumentation observability" via eBPF?
It covers anything crossing a kernel boundary — syscalls, network activity, file I/O — with no application code changes required. It does NOT cover genuinely internal application business logic that never crosses that boundary, which still needs real application-level instrumentation.

---

## Quick-Fire / Rapid Recall

| Q | A |
|---|---|
| Process vs thread — key tradeoff? | Isolation (process) vs lightweight shared memory (thread) |
| Which process state can't even be killed with SIGKILL? | D (uninterruptible sleep) |
| SIGTERM vs SIGKILL? | Polite, catchable request vs. immediate, forced termination |
| Correct memory column to check? | `available`, not `free` |
| What does OOMKilled in Kubernetes actually mean mechanically? | Kernel OOM Killer fired within the container's cgroup memory limit |
| What two kernel features together make a "container"? | Namespaces (what it sees) + cgroups (what it can use) |
| TCP vs UDP, one line? | Reliable/ordered but heavier vs. fast/simple but no guarantees |
| TCP handshake steps? | SYN, SYN-ACK, ACK |
| What causes TIME_WAIT port exhaustion? | Very high rate of short-lived connections |
| Does ping prove a service is up? | No — only basic network reachability |
| DNS resolution order? | Local cache -> recursive resolver -> root -> TLD -> authoritative |
| Tactic before a planned DNS cutover? | Lower the TTL well in advance |
| First 3 commands in the 60-second triage? | uptime, dmesg -T \| tail, vmstat 1 |
| Real disk saturation signal in iostat? | `await` (wait time), not just `%util` |
| Tool to see exactly what syscall a process is blocked on? | strace |
| Tool to see actual packets on the wire? | tcpdump |
| netfilter vs iptables/nftables? | The kernel hook framework vs. its two userspace frontends |
| Default packet-filtering framework on 2026-era distros? | nftables |
| DNAT belongs in which hook? | PREROUTING |
| SNAT/MASQUERADE belongs in which hook? | POSTROUTING |
| What makes ESTABLISHED,RELATED matching possible? | conntrack |
| iptables rule evaluation model? | First-match-wins, top to bottom |
| What connects two network namespaces directly? | A veth pair |
| What acts as a virtual switch for many namespaces? | A Linux bridge |
| What does VXLAN encapsulate? | An entire Ethernet frame, inside UDP |
| Typical overlay MTU after VXLAN overhead? | ~1450 (from 1500) |
| Calico's no-encapsulation alternative to VXLAN? | Routed mode via BGP |
| What is "a container" at the kernel level? | A process in a namespace+cgroup combination, no separate object |
| Command required after editing a systemd unit file? | systemctl daemon-reload |
| Wants+After vs Requires+After? | Soft-fail default vs. hard-fail dependency |
| Correct target for a service needing real connectivity? | network-online.target, not network.target |
| systemd's scale-to-zero mechanism? | Socket activation |
| Who owns the cgroup v2 hierarchy on modern distros? | systemd, exclusively |
| Mechanism letting a container runtime manage its own cgroups? | Cgroup delegation (Delegate=yes) |
| What catches a hung-but-not-exited process? | A systemd watchdog (WatchdogSec=) |
| What proves an eBPF program safe before it can load? | The verifier |
| Why is eBPF fast? | JIT compilation to native machine code |
| What solves cross-kernel-version eBPF portability? | CO-RE + BTF |
| Cilium Service lookup complexity vs iptables? | O(1) hash map vs. O(n) rule chain |
| Cilium's NetworkPolicy enforcement basis? | Pod identity (labels), not raw IPs |
| Bound of "zero-instrumentation" eBPF observability? | Only what crosses the kernel boundary |