Process & Memory Internals

Table of Contents#

1. Why an SRE Needs to Understand the Kernel
2. What a Process Actually Is
3. Process States
4. Signals — How Processes Get Told to Do Things
5. The Difference Between a Process and a Thread
6. File Descriptors — Everything Is a File
7. Memory, From the Ground Up
8. Virtual Memory — The Illusion Every Process Believes
9. The OOM Killer
10. Namespaces — How Containers Get Their Illusion of Isolation
11. cgroups — How Containers Get Resource Limits
12. Namespaces + cgroups = A Container
13. Common Mistakes
14. Worked Practice Problems
15. Summary and What's Next

---

Why an SRE Needs to Understand the Kernel#

Every tutorial in this course so far has lived one or two layers above the operating system — services, dashboards, Kubernetes objects. This tutorial goes underneath all of that, to the Linux kernel itself. The reason this matters for an SRE, specifically: when something goes wrong that dashboards can't explain, the answer is almost always down here — a process stuck waiting, memory quietly running out, a file descriptor limit hit, a container mysteriously killed. Understanding these fundamentals is what turns "I don't know, let's restart it" into "I know exactly why, and here's the fix."

Diagram

---

What a Process Actually Is#

A process is a running instance of a program — code actively being executed, with its own private memory, its own set of open files, and its own identity (a Process ID, or PID).

Simple analogy: think of a cooking recipe as the program (a static set of instructions sitting on a shelf) — a process is what happens the moment someone actually starts cooking it: a specific person, at a specific stove, with specific ingredients out on the counter right now. You can have five people cooking the same recipe simultaneously (five processes running the same program), each with their own separate ingredients and progress.

Copy
# List running processes
ps aux

# See processes in a live, auto-refreshing view
top
htop   # a friendlier, more modern alternative

# See the exact command that started a process
ps -p <PID> -o cmd

---

Process States#

Every process is, at any given moment, in one specific state — and knowing these states is genuinely diagnostic, not just trivia.

Diagram

Why "D" State Is a Real, Practical Red Flag#

A process stuck in D (uninterruptible sleep) for a long time is a strong, specific signal of a serious problem — usually a hung disk, a failing network filesystem (like NFS), or a kernel-level driver issue. The scary, practical part: you cannot kill a process in D state, not even with kill -9 — because the kernel won't hand control back to a process mid-uninterruptible-syscall. If you ever see a process wedged in D state that won't die no matter what, that's a strong signal to look at the underlying storage/disk health, not the process itself.

Copy
# Spot processes stuck in D state
ps aux | awk '$8 ~ /D/ { print }'

Zombie Processes — A Real, Commonly-Misunderstood Thing#

A zombie process isn't actually consuming meaningful resources (no CPU, almost no memory) — it's just an entry in the process table waiting for its parent to call wait() and collect its exit status. A large number of accumulating zombies usually points to a buggy parent process that spawns children but never properly reaps them — not a resource leak in the traditional sense, but a real bug worth fixing (and in extreme cases, it can exhaust the OS's maximum process table entries).

---

Signals — How Processes Get Told to Do Things#

A signal is a lightweight, asynchronous notification sent to a process — the kernel's way of saying "something happened, react to it."

Diagram

Why the Difference Between SIGTERM and SIGKILL Is a Real Operational Decision#

Diagram

Why this matters practically, and it's a very commonly asked interview question: a process that catches SIGTERM and uses it to finish in-flight work cleanly (drain existing requests, close connections properly) provides a genuinely graceful shutdown. A process that's forcibly killed with SIGKILL gets zero chance to clean up — in-flight requests are simply dropped, and any resource it held (a lock, a half-written file) can be left in a bad state. This is exactly why Kubernetes sends SIGTERM first and only escalates to SIGKILL after a configurable grace period — and why applications should be written to actually handle SIGTERM properly, not ignore it.

Copy
# Send SIGTERM (the default, polite signal)
kill <PID>

# Send SIGKILL (forced, cannot be caught or ignored)
kill -9 <PID>

# Send SIGHUP (commonly used to trigger a config reload)
kill -HUP <PID>

# List all available signal names/numbers
kill -l

---

The Difference Between a Process and a Thread#

A commonly-tested, genuinely important distinction.

Diagram

A clean, interview-ready one-liner: "Processes are isolated but heavier; threads are lightweight but share memory with each other, which means a crash or bug in one thread can bring down the whole process — a tradeoff between isolation and efficiency."

---

File Descriptors — Everything Is a File#

One of Unix's defining design philosophies: almost everything — an open file, a network socket, a pipe, even a terminal — is represented as a "file descriptor," a simple integer handle the process uses to read/write it.

Diagram

Copy
# List open file descriptors for a specific process
lsof -p <PID>

# See the CURRENT and MAX file descriptor limits for a process
cat /proc/<PID>/limits | grep "open files"

# See system-wide file descriptor usage
cat /proc/sys/fs/file-nr

"Too Many Open Files" — A Genuinely Common Production Bug#

Every process has a limit on how many file descriptors it can have open at once (ulimit -n). A very common, real production bug: a service that opens a database connection or a network socket per request but fails to close it properly — each "leaked" connection consumes one file descriptor, and eventually the process hits its limit and starts failing with EMFILE / "too many open files" errors, often looking like a mysterious, sudden outage.

Diagram

This is exactly the same "gradual, linear ramp = a leak" diagnostic pattern from the Monitoring Methodologies series — file descriptor exhaustion has the same telltale shape as a memory leak, and monitoring open file descriptor count over time (as a Saturation-style metric) is a genuinely effective early-warning signal for exactly this class of bug.

---

Memory, From the Ground Up#

Diagram

Copy
# Human-readable overview of memory usage
free -h

#               total        used        free      shared  buff/cache   available
# Mem:           31Gi        12Gi       2.1Gi       1.2Gi        17Gi        18Gi

A genuinely important, commonly-misread number: buff/cache. This is memory the kernel is using to cache recently-read disk data (so future reads are faster) — it's not actually "used" in any meaningful sense and will be instantly reclaimed the moment an application needs it. available is the number that actually matters — it accounts for reclaimable cache, telling you the true amount of memory genuinely free to use. A very common mistake: panicking because free shows very little in the "free" column, without realizing most of it is reclaimable cache, not a real problem.

---

Virtual Memory — The Illusion Every Process Believes#

Every process believes it has access to a huge, contiguous, private block of memory, starting at address 0 — this is virtual memory, and it's one of the kernel's most important illusions.

Diagram

Why this matters practically — it's what makes process isolation possible at all: because each process only ever sees its own virtual address space, Process A literally cannot access Process B's real memory, even accidentally, because Process A's view of "address 1000" and Process B's view of "address 1000" are translated by the MMU to two completely different, separate locations in physical RAM. This is the foundational mechanism underneath every claim of memory isolation between processes.

Swap — The Overflow Valve, and Why It's Dangerous#

When physical RAM fills up, the kernel can move rarely-used memory pages out to disk (swap), freeing up RAM for active use.

Diagram

Why heavy swap usage is treated as a real warning sign, not just "the system self-healing": swapping technically prevents an immediate out-of-memory crash, but it does so by trading a hard failure for severe, often confusing performance degradation — a service that starts swapping can appear to "hang" with no clear error, purely because it's waiting on slow disk I/O for memory access that used to be instant. This directly connects to the USE method's Saturation concept (si/so swap activity) from the Monitoring Methodologies series — any non-zero swap activity is worth investigating, not ignoring.

---

The OOM Killer#

When the kernel genuinely runs out of memory (RAM and swap both exhausted), it doesn't just let the system crash — it invokes the OOM (Out-Of-Memory) Killer, which forcibly kills a process to free up memory and keep the rest of the system alive.

Diagram

Copy
# Check if the OOM Killer has fired recently
dmesg | grep -i "killed process"
grep -i oom /var/log/syslog

# See a process's current OOM score (higher = more likely to be killed)
cat /proc/<PID>/oom_score

# Make a critical process LESS likely to be OOM-killed
# (range: -1000 to 1000, lower = less likely)
echo -500 > /proc/<PID>/oom_score_adj

Why this is such an important, frequently-tested concept in a Kubernetes context specifically: a container that exceeds its configured memory limit gets OOM-killed by the kernel inside its own cgroup (covered below) — this shows up in kubectl describe pod as OOMKilled, and it's one of the most common real-world causes of unexpected pod restarts. Recognizing "OOMKilled in pod status" as directly, mechanically connected to this exact kernel mechanism (not some vague Kubernetes-specific magic) is a genuine, valuable piece of understanding.

---

Namespaces — How Containers Get Their Illusion of Isolation#

This section bridges directly into container internals, first introduced in the DevSecOps series (Part 3) — here's the full mechanism underneath that earlier, higher-level discussion.

A namespace gives a process its own isolated view of a specific kind of system resource — without actually duplicating the underlying resource itself.

Diagram

The core insight worth stating explicitly, since it's exactly the nuance covered in the DevSecOps container security tutorial: a process inside a PID namespace genuinely believes it's process 1 in a completely empty system — but on the host, looking in from outside, that same process has a completely different, real PID, and the host can see (and, if not properly isolated, potentially interact with) every process across every namespace. Namespaces create a convincing illusion of isolation, not physical separation — this is exactly why a kernel-level vulnerability can potentially allow "breaking out" of a namespace, as discussed in the container security tutorial.

Copy
# See a process's namespaces
ls -la /proc/<PID>/ns/

# Run a command in a NEW, isolated set of namespaces (a simplified,
# manual version of what container runtimes like Docker/containerd
# do automatically under the hood)
unshare --pid --net --mount --uts --fork /bin/bash

---

cgroups — How Containers Get Resource Limits#

If namespaces control what a process can see, cgroups (control groups) control how much of a resource a process (or group of processes) can actually use.

Diagram

Copy
# View a container's actual cgroup memory limit (from inside the container,
# on a system using cgroup v2)
cat /sys/fs/cgroup/memory.max

# View current memory usage within that cgroup
cat /sys/fs/cgroup/memory.current

The direct, concrete tie to Kubernetes, worth naming explicitly: when you set resources.limits.memory: "512Mi" on a Kubernetes pod spec, Kubernetes is, under the hood, configuring exactly this cgroup memory limit for the container's process group. Exceed it, and the kernel's OOM Killer fires specifically within that cgroup, killing the offending process — which is exactly the mechanism behind a pod showing OOMKilled, tying this whole tutorial's threads (process states, the OOM Killer, and cgroups) into one coherent, mechanical explanation of something you'll see constantly in real operational work.

---

Namespaces + cgroups = A Container#

The single most important synthesis in this entire tutorial, worth being able to state in one clean sentence in an interview:

Diagram

"A container is not a real, separate thing at the kernel level — it's just a regular Linux process, given a restricted view of the system via namespaces, and a resource budget via cgroups. There's no special 'container' kernel object; it's these two existing, general-purpose kernel features combined." This is precisely why containers share the host kernel and are less isolated than a true virtual machine — a point already raised in the DevSecOps container security tutorial, now fully explained mechanically.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A service's memory usage graph climbs steadily for hours, and the pod eventually shows OOMKilled in kubectl describe pod. What's the exact mechanical chain of events, from the application code all the way down to the pod restarting?

Answer: The application likely has a memory leak (something allocated but never released, growing over time — same gradual-ramp shape as the file-descriptor leak example in this tutorial). As usage climbs, it eventually reaches the container's configured cgroup memory limit (set via the pod's resources.limits.memory). At that point, the kernel's OOM Killer activates specifically within that cgroup, selects the offending process (likely the only meaningful process in the container) by its oom_score, and kills it immediately with no graceful shutdown chance. Kubernetes observes the container process exited due to an OOM condition and reports it as OOMKilled in the pod status, then restarts the container per its restart policy.

Problem 2: A Kubernetes deployment currently uses the default terminationGracePeriodSeconds and doesn't handle SIGTERM in application code at all. During rolling deployments, users occasionally see dropped requests. What's happening, and how would you fix it?

Answer: Kubernetes sends SIGTERM to signal the old pod to shut down gracefully, expecting the application to finish in-flight requests and stop accepting new ones during the grace period. If the application doesn't handle SIGTERM at all, it either ignores it (continuing to accept new requests right up until SIGKILL forcibly and immediately terminates it, dropping whatever was in-flight at that exact moment) or the default behavior terminates it immediately without a clean drain. Fix: implement a SIGTERM handler that stops accepting new requests, finishes existing in-flight requests, then exits cleanly — well within the grace period — turning an abrupt, request-dropping kill into a genuinely graceful shutdown.

Problem 3: lsof -p <PID> on a long-running service shows several thousand open file descriptors to the same downstream database host, and the count keeps climbing over days. free -h still shows healthy available memory. What's the likely bug, and what failure mode is this heading toward?

Answer: A connection leak — the application is opening new database connections (each consuming one file descriptor) without properly closing them when finished, likely on some code path that skips cleanup (e.g., an exception thrown before a close() call runs). This is heading toward hitting the process's open file descriptor limit (ulimit -n), at which point new connection attempts (and potentially new file opens generally) will start failing with "too many open files" errors — an outage that will look sudden and confusing to whoever's on call, despite having been building up gradually and visibly the entire time. The fix is ensuring connections are always released deterministically (e.g., using a finally block, context manager, or connection pool with proper lifecycle management), not adding more file descriptor headroom, which only delays the same failure.

---

Summary and What's Next#

• A process is a running program with its own memory and identity; a thread lives inside a process and shares its memory — a tradeoff between crash isolation and lightweight efficiency.
• Process states are genuinely diagnostic — a process stuck in D (uninterruptible sleep) can't even be killed and points to a disk/I/O-level problem, not an application bug.
• SIGTERM requests a graceful shutdown (and can be caught/handled); SIGKILL is immediate and unstoppable — this is exactly why Kubernetes sends SIGTERM first and only escalates to SIGKILL after a grace period.
• File descriptors represent almost everything in Unix (files, sockets, pipes) — a leak of unclosed connections/files is a classic, gradually-building production bug with the exact same "linear ramp" signature as a memory leak.
• Virtual memory is what makes process isolation possible — each process only ever sees its own translated address space, never another process's real memory.
• Swap trades an immediate OOM crash for severe, confusing latency — treat any swap activity as a real warning sign, not benign self-healing.
• The OOM Killer forcibly kills the highest-scoring process when memory is exhausted — and in Kubernetes, this exact mechanism, scoped to a container's cgroup memory limit, is precisely what produces an OOMKilled pod status.
• Namespaces (what a process can see) plus cgroups (how much it can use) together are the entire mechanism behind what we call a "container" — there's no separate container object at the kernel level, which is exactly why containers share the host kernel and are less isolated than a true VM.

Continue to Part 2 (02-tcp-ip-and-dns.md) to move from what happens inside one machine to how machines actually talk to each other — TCP/IP fundamentals and DNS resolution.