# Kubernetes Deep Dive — Part 10: Troubleshooting Kubernetes, Systematically

> **Series:** Kubernetes Deep Dive (10 of 19)
> **Part 1:** `01-architecture-and-control-plane.md` — Architecture & Control Plane
> **Part 2:** `02-scheduling-and-workloads.md` — Scheduling & Workload Objects
> **Part 3:** `03-networking-and-storage.md` — Networking (CNI) & Storage (CSI)
> **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators
> **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE
> **Part 6:** `06-onprem-and-cluster-provisioning.md` — On-Prem & Self-Managed Kubernetes
> **Part 7:** `07-eks-deep-dive.md` — Amazon EKS in Production Depth
> **Part 8:** `08-gateway-api-and-envoy-gateway.md` — Gateway API & Envoy Gateway
> **Part 9:** `09-gateway-api-across-providers.md` — Gateway API Across GKE, EKS, AKS & On-Prem
> **Part 10:** This file — Troubleshooting Kubernetes, Systematically
> **Part 11:** `11-kubernetes-security-deep-dive.md` — Kubernetes Security Deep Dive (CKS-Aligned)
> **Part 12:** `12-autoscaling-hpa-vpa-keda.md` — Autoscaling: HPA, VPA, KEDA & Cluster Autoscaling
> **Part 13:** `13-multi-tenancy-and-cluster-sharing.md` — Multi-Tenancy & Cluster Sharing at Scale
> **Part 14:** `14-ai-ml-workloads-on-kubernetes.md` — Running AI/ML Workloads on Kubernetes
> **Part 15:** `15-cluster-upgrades-and-lifecycle.md` — Cluster Upgrades & Lifecycle Management
> **Part 16:** `16-capacity-planning.md` — Capacity Planning for Kubernetes Clusters
> **Part 17:** `17-cilium-ebpf-networking.md` — Cilium & eBPF Networking Deep Dive
> **Part 18:** `18-backup-and-disaster-recovery.md` — Backup & Disaster Recovery for Workloads
> **Part 19:** `19-building-custom-controllers-and-operators.md` — Building Custom Controllers and Operators
> **Questions:** `questions.md`

Assumes you're comfortable with the control plane and worker node components from Part 1, the workload
objects and scheduling mechanics from Part 2, and the networking model from Part 3 — this chapter is about
diagnosing failures *in* those systems, not introducing them.

## Table of Contents

1. [Why Troubleshooting Deserves Its Own Chapter](#why-troubleshooting-deserves-its-own-chapter)
2. [The Six-Layer Mental Model for Any Kubernetes Failure](#the-six-layer-mental-model-for-any-kubernetes-failure)
3. [The First Move, Always: `get`, `describe`, `events`](#the-first-move-always-get-describe-events)
4. [Reading Pod Phase and Container State Correctly](#reading-pod-phase-and-container-state-correctly)
5. [CrashLoopBackOff — the Full Diagnostic Tree](#crashloopbackoff--the-full-diagnostic-tree)
6. [OOMKilled — Diagnosing and Fixing Memory Failures](#oomkilled--diagnosing-and-fixing-memory-failures)
7. [ImagePullBackOff and ErrImagePull — the Full Diagnostic Tree](#imagepullbackoff-and-errimagepull--the-full-diagnostic-tree)
8. [Admission Webhook Rejections — Pods That Never Even Get Created](#admission-webhook-rejections--pods-that-never-even-get-created)
9. [Pending Pods — Diagnosing Scheduling Failures](#pending-pods--diagnosing-scheduling-failures)
10. [Running But Not Ready — Probe Failures](#running-but-not-ready--probe-failures)
11. [Choosing the Right Tool: `exec`, `kubectl debug`, and Ephemeral Containers](#choosing-the-right-tool-exec-kubectl-debug-and-ephemeral-containers)
12. [Debugging Shell-less and Distroless Containers](#debugging-shell-less-and-distroless-containers)
13. [Log Retrieval Gotchas Worth Knowing](#log-retrieval-gotchas-worth-knowing)
14. [When `kubectl` Itself Can't Help: Debugging a Broken Node](#when-kubectl-itself-cant-help-debugging-a-broken-node)
15. [Node NotReady — the Full Diagnostic Tree](#node-notready--the-full-diagnostic-tree)
16. [Cluster Networking Failures — Pod-to-Pod and Pod-to-Service](#cluster-networking-failures--pod-to-pod-and-pod-to-service)
17. [DNS Resolution Failures — the CoreDNS Diagnostic Tree](#dns-resolution-failures--the-coredns-diagnostic-tree)
18. ["It Worked Until We Added a NetworkPolicy"](#it-worked-until-we-added-a-networkpolicy)
19. [Control Plane Failures — When the API Server or etcd Is Sick](#control-plane-failures--when-the-api-server-or-etcd-is-sick)
20. [Storage Failures — PVC Pending and Mount Timeouts](#storage-failures--pvc-pending-and-mount-timeouts)
21. [Troubleshooting Jobs and CronJobs](#troubleshooting-jobs-and-cronjobs)
22. [A Full Worked Incident: A Multi-Symptom Production Outage](#a-full-worked-incident-a-multi-symptom-production-outage)
23. [A Note on `kubectl top` and metrics-server](#a-note-on-kubectl-top-and-metrics-server)
24. [Building Your Own Troubleshooting Runbook](#building-your-own-troubleshooting-runbook)
25. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
26. [Worked Practice Problems](#worked-practice-problems)
27. [Summary and What's Next](#summary-and-whats-next)

---

## Why Troubleshooting Deserves Its Own Chapter

**Troubleshooting is the single largest domain on the CKA exam — 30% of the score — yet it's the one skill
no reference doc actually teaches, because reference docs describe how things work when they're healthy,
not how to work backward from a symptom to a cause.** Parts 1 through 9 of this series built a detailed
mental model of how the pieces fit together; this chapter inverts that model into a diagnostic tool. Every
section below follows the same shape: a symptom you'd actually see, a small number of root causes ranked
by how often they're the real one, and the exact commands that tell them apart.

The throughline system from earlier parts continues here: a `checkout` namespace running `checkout-service`
talking to a `catalog-service` and a Postgres-backed `inventory-service`, all sitting behind a Service and,
in later sections, a NetworkPolicy. Every worked example below happens inside that same small system.

> [!IMPORTANT]
> The number one habit that separates fast troubleshooting from slow troubleshooting is **reading before
> acting**. `kubectl describe` and `kubectl get events` almost always tell you the category of failure in
> under 10 seconds. Jumping straight to `kubectl logs`, or worse, straight to `kubectl delete pod`, skips
> the fastest source of signal you have and often destroys the exact evidence (a crashed container's exit
> code, a scheduler's rejection reason) you need to actually find the cause.

## The Six-Layer Mental Model for Any Kubernetes Failure

**Every Kubernetes failure lives at exactly one of six layers, and knowing which layer you're in before you
start running commands cuts your search space by roughly 5x.** Practitioners who troubleshoot slowly
usually aren't missing knowledge — they're jumping between layers randomly instead of eliminating them in
order.

```mermaid
flowchart TD
    L1["Layer 1: Application code<br/>(the process inside the container<br/>crashes, hangs, or misbehaves)"]
    L2["Layer 2: Container/runtime<br/>(image, entrypoint, resource limits,<br/>containerd/CRI-O)"]
    L3["Layer 3: Pod scheduling<br/>(can the Scheduler place<br/>this pod anywhere at all?)"]
    L4["Layer 4: Node health<br/>(kubelet, disk/memory/PID<br/>pressure, node network)"]
    L5["Layer 5: Cluster networking<br/>(CNI, kube-proxy, DNS,<br/>NetworkPolicy)"]
    L6["Layer 6: Control plane<br/>(API Server, etcd,<br/>Scheduler, Controller Manager)"]

    L1 --> L2 --> L3 --> L4 --> L5 --> L6
```

**This diagram isn't a call sequence — it's a search order.** Layer 1-2 failures (a single pod
crash-looping) are by far the most common and the cheapest to rule in or out first; Layer 6 failures (the
whole control plane is unhealthy) are rare but explain symptoms across every namespace at once. A
`describe pod` in the first ten seconds usually tells you which layer you're actually in, which is why the
next section is the mandatory starting point for everything that follows.

| Layer | Symptom shape | Section below |
|---|---|---|
| 1-2: App/container | One pod, or one Deployment, is unhealthy; others are fine | CrashLoopBackOff, OOMKilled, ImagePullBackOff |
| 3: Scheduling | Pod stuck `Pending`, never gets a node | Pending Pods |
| 4: Node | Everything on one node is unhealthy; other nodes are fine | Node NotReady |
| 5: Cluster networking | Cross-service traffic fails; single pods look healthy | DNS, NetworkPolicy, CNI |
| 6: Control plane | Cluster-wide: `kubectl` itself times out, nothing schedules anywhere | API Server/etcd failures |

## The First Move, Always: `get`, `describe`, `events`

**Before touching logs, before execing into anything, run these three commands — they cost nothing and
eliminate entire categories of cause.**

```bash
kubectl get pods -n checkout -o wide
kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout
kubectl get events -n checkout --sort-by=.lastTimestamp
```

`kubectl get -o wide` shows the phase, restart count, and node in one line — restart count alone tells you
whether you're looking at a fresh failure or a pod that's been crash-looping for hours. `describe` prints
the pod spec, current container states, resource requests, and — critically — the **Events** section at the
bottom, which is the Scheduler, kubelet, and controllers narrating what they tried to do and why it failed,
in their own words. `kubectl get events` cluster- or namespace-wide catches things `describe` on one pod
misses, like a NetworkPolicy controller rejecting a rule or a PVC provisioner failing silently.

> [!TIP]
> Events expire after roughly one hour by default (`--event-ttl` on the API server, cluster-wide). If
> you're investigating something that happened earlier and `kubectl get events` comes back empty, that's
> not a dead end — it's a signal the event already aged out, and you should move straight to `kubectl logs
> --previous` and metrics/log aggregation instead of re-running `get events` expecting a different result.

## Reading Pod Phase and Container State Correctly

**A Pod has one `phase` (a coarse, cluster-wide summary) and each of its containers has its own
finer-grained `state` — conflating the two is a common source of confused troubleshooting.**

| Pod phase | Meaning |
|---|---|
| `Pending` | Accepted by the API Server, but at least one container hasn't started running yet (usually: not yet scheduled) |
| `Running` | Bound to a node and at least one container is running |
| `Succeeded` | All containers terminated successfully — normal for a completed Job, abnormal for a Deployment's pod |
| `Failed` | All containers terminated, and at least one terminated in failure |
| `Unknown` | The kubelet can't report the pod's state to the API Server — usually a node communication problem |

Underneath a `Running` phase, each container independently reports one of three states:

```mermaid
stateDiagram-v2
    [*] --> Waiting: Container created,<br/>image not yet pulled/started
    Waiting --> Running: Entrypoint started successfully
    Running --> Terminated: Process exits (any reason)
    Terminated --> Waiting: Restart policy triggers<br/>a new attempt
    Waiting --> Terminated: Image pull or config<br/>failure before start

    note right of Waiting
        describe shows a Reason here:
        ImagePullBackOff, ErrImagePull,
        CreateContainerConfigError,
        CrashLoopBackOff
    end note
```

**The Pod phase can say `Running` while the container you actually care about is `Waiting` with
`CrashLoopBackOff`** — a multi-container pod where a sidecar is healthy but the main container is looping
will show `Running` at the pod level with `1/2` ready containers. Always check the per-container state in
`describe`'s output, not just the top-line phase.

## CrashLoopBackOff — the Full Diagnostic Tree

**`CrashLoopBackOff` means the container starts, then exits, repeatedly — it is a *symptom*, never the root
cause itself, and the backoff delay (10s, doubling up to a 5-minute cap) is Kubernetes protecting itself
from hammering a broken container, not a bug.**

```mermaid
flowchart TD
    Start["Container in CrashLoopBackOff"] --> Exit["Check exit code:<br/>kubectl describe pod<br/>(lastState.terminated)"]
    Exit -->|"Exit 0"| Zero["Process exited cleanly —<br/>entrypoint finished its work<br/>and returned, e.g. a script<br/>with no long-running process"]
    Exit -->|"Exit 1"| App["Application-level error —<br/>check kubectl logs --previous"]
    Exit -->|"Exit 137"| OOM["SIGKILL — almost always OOMKilled,<br/>confirm via describe's Reason field"]
    Exit -->|"Exit 139"| Segv["SIGSEGV — a crash inside the<br/>binary/runtime itself"]
    Exit -->|"Exit 143"| Term["SIGTERM — often a failed<br/>graceful shutdown, check<br/>terminationGracePeriodSeconds"]

    App --> Logs["logs show a real stack trace<br/>or config error?"]
    Logs -->|"Yes"| FixApp["Fix the application config/code<br/>— not a cluster problem"]
    Logs -->|"No output at all"| Probe["Likely killed by a failing<br/>liveness probe before it<br/>could log anything useful"]
```

**Reading exit codes is the fastest triage step and the one most people skip.** `kubectl describe pod`'s
`Last State: Terminated` block shows both a numeric `Exit Code` and a human `Reason` — cross-referencing
the two takes seconds and immediately rules out or confirms half the tree above.

```bash
kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -A5 "Last State"
kubectl logs checkout-service-7d8f9c-x2k9p -n checkout --previous
```

> [!WARNING]
> A liveness probe misconfigured with too short a `timeoutSeconds` or too few `failureThreshold` retries
> will kill and restart a genuinely healthy but momentarily slow container, and the resulting
> `CrashLoopBackOff` has **no application error to find** — because there isn't one. A team once spent four
> hours adding retry logic to a service that was actually fine; the real fix was raising the liveness
> probe's `timeoutSeconds` from 1 to 5 to survive a GC pause under load. Always check `describe`'s Events
> for `Liveness probe failed` lines before assuming the application itself is broken.

## OOMKilled — Diagnosing and Fixing Memory Failures

**`OOMKilled` (exit code 137, `Reason: OOMKilled` in `describe`) means the kernel's cgroup memory
controller killed the container for exceeding its memory *limit* — this is enforced by the kernel, not by
Kubernetes polling and deciding, so it happens instantly and with zero graceful shutdown.**

```bash
kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -B2 -A2 OOMKilled
kubectl top pod checkout-service-7d8f9c-x2k9p -n checkout
```

| Root cause | How to confirm | Fix |
|---|---|---|
| Memory `limit` set genuinely too low for a healthy workload | `kubectl top pod` over time shows steady climb toward the limit under normal load | Raise the limit based on observed p99 usage, not a guess |
| A real memory leak in the application | Usage climbs continuously and never plateaus, even under constant load | Fix the leak — raising the limit only delays the same crash |
| A traffic spike causing a legitimate short-term memory surge | Correlates with a deploy, a batch job, or a marketing event | Add horizontal replicas or a VPA (Part 12) instead of a blanket limit increase |
| `limit` set equal to `request` with a workload that has bursty peaks | Works fine at idle, OOMKilled only under load | Widen the gap between request and limit, or move to Guaranteed QoS deliberately (Part 2) |

**Explaining this two levels deep, the way a real postmortem should:** the symptom is "checkout-service
keeps restarting under load." The immediate cause is the container hit its 256Mi memory limit. The
underlying condition is that the limit was copy-pasted from a template chart during initial rollout and
never revisited after the team added an in-memory response cache six months later — nobody owns a periodic
review of resource requests/limits against actual usage, so the mismatch sat silent until a traffic spike
finally pushed it over the edge.

> [!CAUTION]
> Raising a memory limit on a node that's already near its allocatable capacity can push the **node**
> itself into memory pressure, triggering kubelet eviction of *other* pods on that node — a fix for one
> service's OOMKill can create a cascading outage for its neighbors. Check `kubectl describe node <name>`
> for `Allocatable` headroom before raising a limit, not just the one pod's own numbers.

## ImagePullBackOff and ErrImagePull — the Full Diagnostic Tree

**`ErrImagePull` is the first failed attempt; `ImagePullBackOff` is Kubernetes retrying with the same
exponential backoff as CrashLoopBackOff — the fix is always found in `describe`'s Events, never in
`kubectl logs`, because the container never started.**

```mermaid
flowchart TD
    Start["ImagePullBackOff"] --> Check["kubectl describe pod<br/>-> read the Events reason"]
    Check -->|"manifest unknown /<br/>not found"| Typo["Wrong image name or tag —<br/>check for a typo or an<br/>unpublished tag"]
    Check -->|"unauthorized /<br/>authentication required"| Auth["Private registry, missing<br/>or wrong imagePullSecret"]
    Check -->|"i/o timeout /<br/>no route to host"| Network["Node can't reach the registry —<br/>egress firewall, proxy, or DNS"]
    Check -->|"toomanyrequests"| Rate["Registry rate limiting<br/>(common on Docker Hub's<br/>anonymous pull limits)"]

    Auth --> FixAuth["Create/attach a docker-registry<br/>Secret, reference it in<br/>imagePullSecrets"]
    Rate --> FixRate["Authenticate pulls, or mirror<br/>the image to a private<br/>registry/pull-through cache"]
```

```bash
kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -A3 Events
# Confirm the exact image string being requested:
kubectl get pod checkout-service-7d8f9c-x2k9p -n checkout -o jsonpath='{.spec.containers[*].image}'
```

**A genuinely common "worked on my machine" trap**: the image pulls fine locally because `docker login`
already authenticated your workstation to the private registry, but the cluster's nodes (or the specific
ServiceAccount the pod runs as) never got an `imagePullSecrets` reference — this is a pull-credential
problem, not an image problem, and it will pass every local test while failing 100% of the time in the
cluster.

## Admission Webhook Rejections — Pods That Never Even Get Created

**This failure looks nothing like the others in this chapter, because it happens before a Pod object even
exists — `kubectl apply` itself returns an error, and there's no pod to `describe` at all.** Part 1 covered
the admission control chain (authentication → authorization → mutating webhooks → object schema validation
→ validating webhooks); a validating webhook rejecting the request is one of the most common "why won't
this even create" failures in a cluster running policy engines like OPA Gatekeeper or Kyverno (Part 11).

```bash
kubectl apply -f checkout-deployment.yaml
# Error from server (Forbidden): error when creating "checkout-deployment.yaml":
# admission webhook "validate-resources.kyverno.svc" denied the request:
# resource limits are required on every container
```

```mermaid
flowchart TD
    Apply["kubectl apply -f pod.yaml"] --> Auth["Authentication +<br/>Authorization"]
    Auth --> Mutate["Mutating admission<br/>webhooks run first"]
    Mutate --> Schema["API Server schema<br/>validation"]
    Schema --> Validate["Validating admission<br/>webhooks run last"]
    Validate -->|"any webhook denies"| Reject["Request rejected —<br/>the error message names<br/>which webhook and why"]
    Validate -->|"all approve"| Etcd["Object persisted to etcd"]
```

**The error message almost always names the exact webhook and the exact rule it violated** — read it
literally before assuming the manifest itself is malformed. A second, more confusing variant of this
failure: the webhook's own backend (the Kyverno/Gatekeeper pod serving the webhook) is down or unreachable,
in which case the error instead reads something like `failed calling webhook ... context deadline exceeded`
or `connection refused` — that's a webhook-availability problem, not a policy violation, and the fix is
restoring the policy engine's own pods rather than editing your manifest.

```bash
kubectl get validatingwebhookconfigurations
kubectl get pods -n kyverno   # or gatekeeper-system — confirm the webhook's backing pods are healthy
```

> [!WARNING]
> A `failurePolicy: Fail` webhook that becomes unreachable (its backing pods crash, or a node hosting them
> goes down) can block **every** matching object creation cluster-wide until it recovers — including,
> potentially, the very fix a responder is trying to `kubectl apply` during an incident. Confirm a webhook's
> `failurePolicy` and namespace exemptions (`kube-system` is almost always excluded) before assuming any
> `kubectl apply` failure during an outage is unrelated to admission control.

## Pending Pods — Diagnosing Scheduling Failures

**A `Pending` pod has been accepted by the API Server but the Scheduler has not (yet, or ever) found a node
that satisfies every one of its constraints — `describe`'s Events section names the exact constraint that
failed, so this is almost never a guessing game.**

| Events message contains... | Root cause | Where to look (cross-ref) |
|---|---|---|
| `Insufficient cpu` / `Insufficient memory` | No node has enough allocatable capacity left | Node capacity, or add nodes/Karpenter (Part 7) |
| `node(s) had taint {...} that the pod didn't tolerate` | Missing a required toleration | Taints and Tolerations (Part 2) |
| `didn't match Pod's node affinity/selector` | `nodeSelector`/`nodeAffinity` rules out every node | Node Affinity (Part 2) |
| `0/5 nodes are available: 5 node(s) didn't match pod anti-affinity rules` | Pod anti-affinity has no eligible node left | Pod Affinity/Anti-Affinity (Part 2) |
| `persistentvolumeclaim "..." not found` or stuck `Pending` PVC | Storage isn't bound yet — the pod can't start until it is | Storage Failures below |
| `Too many pods` | Node's pod-count limit (default 110) reached | Add nodes, or raise `--max-pods` where policy allows |

```bash
kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -A10 Events
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory
```

> [!NOTE]
> `describe`'s scheduling failure message is a **summary across every node**, e.g. `0/5 nodes are
> available: 2 Insufficient cpu, 3 node(s) had untolerated taint`. Read the whole line — a pod can be
> failing for more than one reason across different subsets of nodes simultaneously, and fixing only the
> first cause you notice can still leave it `Pending`.

## Running But Not Ready — Probe Failures

**A pod showing `Running` with `0/1` or a fractional Ready count (e.g. `1/2`) is not a scheduling or
crash problem — it means the container process is alive but its readiness probe is failing, so the Service
deliberately excludes it from Endpoints (Part 3) rather than sending it traffic it can't yet serve.**

```bash
kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -A5 "Readiness probe failed"
kubectl get endpoints checkout-svc -n checkout
```

| Symptom | Likely cause |
|---|---|
| Readiness fails immediately at startup, forever | Probe path/port wrong, or app doesn't actually listen until a slow init step (DB migration, cache warm) completes |
| Readiness flaps — passes, fails, passes | App is intermittently overloaded; probe itself may be too aggressive (short `periodSeconds`) |
| Readiness passes but Endpoints is still empty | Label selector mismatch between the Service and the Pod — a scoping bug, not a health bug |

**The label-selector case deserves calling out on its own** because it produces a confusing symptom: the
pod is `1/1 Ready`, `kubectl exec` into it and curling the app locally works fine, yet the Service still
routes zero traffic to it. The cause is almost always that the Service's `selector` and the Pod's `labels`
were both edited independently during a refactor and silently drifted apart — `kubectl get endpoints` shows
zero or wrong addresses, which is the tell that this is a selector-matching problem, not an application
health problem at all.

## Choosing the Right Tool: `exec`, `kubectl debug`, and Ephemeral Containers

**These three tools solve different problems, and reaching for the wrong one wastes time or simply
doesn't work.**

| Tool | Use when | Limitation |
|---|---|---|
| `kubectl exec -it <pod> -- sh` | The container is running, healthy enough to accept a shell, and has one installed | Fails on a crashed container or a shell-less (distroless) image |
| `kubectl debug <pod> -it --image=busybox --target=<container>` | Container is running but lacks debugging tools; you need to inspect it from the *outside* without restarting it | Requires Kubernetes 1.25+ for stable ephemeral containers |
| `kubectl debug <pod> --copy-to=debug-pod --container=<c> -it -- sh` | The container **won't start at all** (crash-looping before you can attach) and you need to run its entrypoint under a debugger/shell instead | Creates a new pod copy — doesn't touch the live one, useful precisely because the original never has to start successfully |
| `kubectl debug node/<node> -it --image=busybox` | The node itself needs inspecting (filesystem, processes) and you have no direct SSH access | Runs in the host's namespaces — powerful and should be used carefully |

```bash
# Attach a debug container sharing the target's process namespace —
# ps aux inside it shows the real app's processes:
kubectl debug checkout-service-7d8f9c-x2k9p -n checkout \
  -it --image=nicolaka/netshoot --target=checkout-service
```

**Ephemeral containers are recorded in the API audit log and never modify the original container** — this
is precisely why they satisfy production change-control requirements that a `kubectl exec` into a
modified, debug-tooling-injected image would not.

## Debugging Shell-less and Distroless Containers

**A distroless or scratch-based production image (no shell, no package manager, no `/bin/sh`) is a
deliberate, defensible security choice — smaller attack surface, fewer CVEs to patch — but it means
`kubectl exec -it ... -- sh` fails outright with `OCI runtime exec failed: exec failed: unable to start
container process: exec: "sh": executable file not found`.**

```bash
kubectl debug checkout-service-7d8f9c-x2k9p -n checkout \
  -it --image=busybox --target=checkout-service
# Inside the debug container:
ls -l /proc/1/root/            # the target container's filesystem, read-only
cat /proc/1/environ | tr '\0' '\n'   # its actual environment variables
```

The ephemeral debug container shares the target's process namespace (via `--target`), so `/proc/1/` is the
main application's process — its filesystem, open file descriptors, and environment are all inspectable
from a fully-tooled container sitting alongside it, without ever needing a shell inside the distroless image
itself.

## Log Retrieval Gotchas Worth Knowing

**`kubectl logs` has several sharp edges that produce "logs are empty/wrong/missing" confusion often
mistaken for an application logging bug.**

| Situation | Gotcha | Fix |
|---|---|---|
| Multi-container pod | `kubectl logs <pod>` with no `-c` picks the *first* container in the spec, silently — not necessarily the one you meant | Always pass `-c <container>` explicitly on any multi-container pod |
| Container already restarted | Plain `logs` shows only the **current** attempt's output, which can be seconds old for a crash-looper | `kubectl logs <pod> --previous` retrieves the prior (crashed) attempt's logs |
| Init container | Its logs disappear from the default view once it completes and the main container starts | `kubectl logs <pod> -c <init-container-name>` still works after completion, as long as the pod object still exists |
| Very high log volume | Container runtimes rotate and cap log files (`containerLogMaxSize`); very early lines from a chatty container can already be gone by the time you look | Ship logs to an aggregator (this site's `observability` domain) for anything that needs retention beyond a live pod's lifetime |
| Pod already deleted/evicted | `kubectl logs` on a gone pod returns nothing — there is no object left to proxy the request to | Only a log aggregator (Loki, CloudWatch Logs, etc.) has it at that point; the live API can't retrieve what no longer exists |

```bash
kubectl logs checkout-service-7d8f9c-x2k9p -n checkout -c checkout-service --previous
kubectl logs checkout-service-7d8f9c-x2k9p -n checkout -c wait-for-db   # an init container, by name
kubectl logs -n checkout -l app=checkout-service --all-containers=true --prefix --tail=50
```

**The `-l`/`--all-containers`/`--prefix` combination above is worth memorizing on its own** — it tails the
last 50 lines from every container across every pod matching a label selector, each line prefixed with its
source pod/container, which is frequently faster than tailing pods one at a time when a Deployment has
several replicas and you don't yet know which one is misbehaving.

## When `kubectl` Itself Can't Help: Debugging a Broken Node

**If the kubelet on a node has crashed or the container runtime is wedged, `kubectl` commands targeting
that node's pods will hang or time out — at that point, the debugging has to happen on the node itself, one
layer below the Kubernetes API entirely.**

```mermaid
sequenceDiagram
    participant You
    participant API as API Server
    participant Kubelet
    participant CRI as containerd/CRI-O
    participant OS as Node OS

    You->>API: kubectl logs pod-x
    API->>Kubelet: proxy the log request
    Note over Kubelet: Kubelet process is hung<br/>or containerd is unresponsive
    Kubelet--xAPI: no response
    API--xYou: timeout

    You->>Kubelet: SSH to node, then:<br/>systemctl status kubelet
    You->>OS: journalctl -u kubelet -f
    You->>CRI: crictl ps (bypasses kubelet,<br/>talks to the runtime directly)
    You->>CRI: crictl logs container-id
```

**`crictl` is the single most important tool here** — it speaks the same Container Runtime Interface
protocol the kubelet uses, but works even when the kubelet process itself is the thing that's broken,
because it talks directly to containerd/CRI-O's own socket.

```bash
systemctl status kubelet
journalctl -u kubelet -n 200 --no-pager
crictl ps -a                      # every container the runtime knows about, kubelet or not
crictl logs <container-id>
crictl inspect <container-id> | grep -i oom
```

> [!TIP]
> If SSH access to a node is restricted or unavailable (common on managed EKS/GKE/AKS node pools),
> `kubectl debug node/<node-name> -it --image=busybox -- chroot /host` gets you an equivalent shell on the
> host filesystem through the Kubernetes API itself — no direct node access required.

## Node NotReady — the Full Diagnostic Tree

**A node reports `NotReady` when the API Server stops receiving its heartbeat within the node-monitor
grace period (default 40 seconds after a 10-second heartbeat interval) — every pod scheduled on it is then
treated as unreachable, and after `pod-eviction-timeout` (default 5 minutes) the Controller Manager starts
evicting and rescheduling them elsewhere.**

```mermaid
flowchart TD
    NR["Node: NotReady"] --> Check1["journalctl -u kubelet<br/>— is it even running?"]
    Check1 -->|"kubelet crashed/stopped"| KubeletDown["Check kubelet's own crash<br/>reason: OOM, bad config,<br/>certificate expiry"]
    Check1 -->|"kubelet running,<br/>logging normally"| Check2["crictl info<br/>— is the runtime healthy?"]
    Check2 -->|"runtime unresponsive"| PLEG["PLEG not healthy —<br/>containerd/CRI-O stuck,<br/>often disk I/O starvation"]
    Check2 -->|"runtime fine"| Check3["kubectl describe node<br/>— check Conditions"]
    Check3 -->|"DiskPressure"| Disk["Node disk usage above<br/>the eviction threshold<br/>(images, logs, or emptyDir)"]
    Check3 -->|"MemoryPressure"| Mem["Node memory below<br/>the eviction threshold"]
    Check3 -->|"NetworkUnavailable"| Net["CNI not initialized,<br/>or node lost network<br/>connectivity entirely"]
```

```bash
kubectl describe node ip-10-0-3-27.ec2.internal | grep -A15 Conditions
kubectl get events --field-selector involvedObject.name=ip-10-0-3-27.ec2.internal
```

**Explaining `PLEG is not healthy` two levels deep, since it's one of the more opaque messages a node
throws**: the symptom is the kubelet logging `PLEG is not healthy: pleg was last seen active Nm ago`. The
immediate cause is the kubelet's Pod Lifecycle Event Generator — the loop that polls the container runtime
for state changes — timed out waiting for a response. The underlying condition is almost always the
container runtime itself is starved, most commonly because the node's disk is under heavy I/O pressure
(log rotation misconfigured, a runaway container writing gigabytes to `emptyDir`) slow enough that even
`crictl ps` takes multiple seconds to return — fixing the disk I/O bottleneck resolves the PLEG error as a
side effect, restarting the kubelet alone usually just repeats the same failure minutes later.

> [!CAUTION]
> Restarting `containerd`/`crictl`-managed runtime on a node that's currently hosting `StatefulSet` pods
> with local storage assumptions, or any pod without a `PodDisruptionBudget`, can cause a harder outage than
> the original NotReady condition — confirm what's actually running on the node and whether it tolerates an
> unplanned restart before restarting the runtime, not just the kubelet.

## Cluster Networking Failures — Pod-to-Pod and Pod-to-Service

**When individual pods look healthy (`Running`, `1/1 Ready`) but traffic between them fails, the problem
has moved from "is this pod OK" to "can the network actually deliver a packet between these two IPs" — a
different, layered investigation.**

```mermaid
flowchart LR
    A["curl from pod A<br/>to pod B's IP directly"] -->|"fails"| B["CNI-level problem:<br/>check CNI pod logs,<br/>node routes"]
    A -->|"succeeds"| C["curl from pod A<br/>to Service ClusterIP"]
    C -->|"fails"| D["kube-proxy / Service<br/>routing problem"]
    C -->|"succeeds"| E["Traffic reaches the pod —<br/>failure is likely app-level<br/>or NetworkPolicy"]
```

**This ladder — pod IP directly, then Service ClusterIP, then DNS name — isolates the failing layer in
three commands**, because each step adds exactly one more piece of the networking stack on top of the
previous, successful step:

```bash
kubectl exec -it catalog-service-xyz -n catalog -- curl -m 3 http://10.244.1.15:8080/healthz
kubectl exec -it catalog-service-xyz -n catalog -- curl -m 3 http://checkout-svc.checkout.svc.cluster.local:80/healthz
kubectl get endpoints checkout-svc -n checkout
kubectl get pods -n kube-system -l k8s-app=kube-proxy
```

If step one (direct pod IP) fails, the fault is at the CNI layer — check the CNI daemonset's pods for
crash-loops (`kubectl get pods -n kube-system -l k8s-app=cilium` or the equivalent for your CNI) and confirm
routes exist between the two nodes hosting the pods. If step one succeeds but step two (Service ClusterIP)
fails, `kube-proxy` is the suspect — confirm its pods are healthy and, on nodes using IPVS mode,
`ipvsadm -Ln` shows the expected virtual server entries.

## DNS Resolution Failures — the CoreDNS Diagnostic Tree

**"It can't resolve the service name" is one of the most common tickets in any Kubernetes cluster, and it
has a small, well-ordered set of causes — work through them in this order, not randomly.**

```mermaid
flowchart TD
    Start["Pod can't resolve<br/>checkout-svc.checkout.svc.cluster.local"] --> S1["Are CoreDNS pods<br/>Running and Ready?"]
    S1 -->|"No"| Fix1["Fix CoreDNS pods first —<br/>nothing else matters until<br/>they're healthy"]
    S1 -->|"Yes"| S2["Does the kube-dns<br/>Service have Endpoints?"]
    S2 -->|"Empty"| Fix2["CoreDNS pods aren't matching<br/>the kube-dns Service selector"]
    S2 -->|"Has endpoints"| S3["Run nslookup from a<br/>debug pod IN the same<br/>namespace as the caller"]
    S3 -->|"NXDOMAIN"| S4["Check the exact FQDN —<br/>ndots:5 default means a<br/>short name tries 5 search<br/>domains before the real one"]
    S3 -->|"Times out, no answer"| S5["NetworkPolicy blocking<br/>UDP/TCP 53 egress to<br/>kube-system, or CNI issue"]
    S3 -->|"Resolves correctly"| Fix5["DNS is fine — the original<br/>failure is elsewhere<br/>(app-level, TLS, etc.)"]
```

```bash
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl get endpoints kube-dns -n kube-system
kubectl run dns-debug --rm -it --image=busybox:1.36 -n checkout -- nslookup checkout-svc.checkout.svc.cluster.local
kubectl get configmap coredns -n kube-system -o yaml
```

**The `ndots:5` trap is worth understanding precisely, since it's genuinely confusing the first time it
bites.** A pod's default `/etc/resolv.conf` sets `ndots:5`, meaning any name with fewer than 5 dots is
tried against every entry in the search list (`checkout.svc.cluster.local`, `svc.cluster.local`,
`cluster.local`, then the node's own search domains) **before** being tried as an absolute name — for an
external hostname like `api.stripe.com` (2 dots), that means up to 4 failed internal lookups before the
5th attempt finally succeeds externally, adding real, measurable latency to every single external call a
pod makes. This is why high-throughput services often set `dnsConfig.options` to lower `ndots`, or use a
trailing dot (`api.stripe.com.`) to force an absolute lookup.

## "It Worked Until We Added a NetworkPolicy"

**This is one of the most common self-inflicted outages in any cluster with a security team actively
hardening it — a genuinely correct NetworkPolicy silently breaks a dependency nobody realized existed,
because Kubernetes NetworkPolicies are allow-list, and once *any* policy selects a pod, every previously
implicit "allow all" traffic pattern to it disappears at once.**

```bash
kubectl get networkpolicy -n checkout
kubectl describe networkpolicy checkout-default-deny -n checkout
```

> [!WARNING]
> The instant a single `NetworkPolicy` selects a pod (via `podSelector`), that pod's ingress (or egress,
> depending on `policyTypes`) becomes **default-deny** for that direction — every other traffic source that
> used to reach it silently stops working unless an explicit rule allows it. A real incident: a team added
> a default-deny NetworkPolicy to the `checkout` namespace to satisfy a security audit, verified the
> checkout API itself still worked, and shipped it — two hours later, Prometheus stopped being able to
> scrape `/metrics` from those same pods, because nobody had written an explicit ingress rule allowing the
> monitoring namespace. The dashboards went dark with zero alerts firing about the actual outage, since the
> alerting pipeline depending on those metrics was itself the thing silently cut off.

| Symptom | Diagnostic |
|---|---|
| Traffic worked before a NetworkPolicy was added, fails after | `kubectl describe networkpolicy` — check whether the failing source's namespace/labels are covered by an `ingress.from` rule |
| Only *some* callers are blocked | Policies are additive per-pod (any matching policy's rules apply) — check for multiple overlapping policies, not just one |
| DNS itself breaks after adding an egress policy | The policy has no rule allowing egress to `kube-system` on port 53 — DNS egress must be explicitly allowed like anything else |

## Control Plane Failures — When the API Server or etcd Is Sick

**Control plane failures are the rarest layer in this chapter but the most severe, because `kubectl` itself
becomes part of what's broken — every diagnostic technique above depends on a healthy API Server to run.**

```bash
kubectl get --raw='/readyz?verbose'
kubectl get componentstatuses    # deprecated but still informative on many clusters
curl -k https://localhost:2379/health --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key
```

| Symptom | Likely cause |
|---|---|
| `kubectl` commands hang or return `connection refused` | API Server pod/process down, or its load balancer is unhealthy |
| API Server up, but writes fail with `etcdserver: request timed out` | etcd quorum lost, or etcd disk latency is far above its ~10ms expectation (Part 1) |
| Nothing schedules anywhere, cluster-wide | Scheduler or Controller Manager lost leader election — check `kubectl -n kube-system get lease kube-scheduler` |
| API Server responds but is extremely slow | Often an etcd compaction/defragmentation backlog (Part 4) — check etcd's own metrics for DB size approaching its quota |

**`/readyz?verbose` is worth memorizing precisely because it self-diagnoses**: it returns a line-by-line
pass/fail for every internal health check the API Server runs (etcd connectivity, informer sync status,
shutdown state), which is almost always faster than manually checking each dependency by hand.

## Storage Failures — PVC Pending and Mount Timeouts

**A pod stuck `Pending` with a storage-related Events message, or `Running` but stuck at
`ContainerCreating` for minutes, points at the CSI/storage layer (Part 3) rather than compute scheduling.**

```bash
kubectl get pvc -n checkout
kubectl describe pvc checkout-data -n checkout
kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -A5 "FailedMount\|FailedAttachVolume"
```

| Symptom | Root cause |
|---|---|
| PVC stuck `Pending` forever | No StorageClass matches, or the provisioner (CSI driver) pods themselves are unhealthy |
| PVC `Bound`, pod stuck `ContainerCreating` with `FailedMount` | Volume attached to a different node already (common after an unclean node failover) |
| Works on one node, fails on another | A `ReadWriteOnce` volume already attached elsewhere, or a topology-aware StorageClass zone mismatch (Part 3) |

## Troubleshooting Jobs and CronJobs

**Jobs and CronJobs (Part 2) fail in ways specific to their run-to-completion model, and the standard
pod-troubleshooting steps above only get you partway there — the Job/CronJob object itself carries
additional state worth checking first.**

```bash
kubectl describe job checkout-nightly-reconcile -n checkout
kubectl get pods -n checkout -l job-name=checkout-nightly-reconcile-29384710 --show-labels
```

| Symptom | Root cause | Where to look |
|---|---|---|
| Job shows `Failed` with reason `BackoffLimitExceeded` | Every pod attempt failed and the Job gave up after `backoffLimit` retries | Same pod-level diagnostic tree as CrashLoopBackOff — inspect the *last* failed pod's exit code and logs |
| Job's pod stuck `Running` far past expected duration | No `activeDeadlineSeconds` set, so a hung process runs indefinitely with nothing to time it out | Set `activeDeadlineSeconds` deliberately, don't rely on manual intervention |
| CronJob's schedule silently stopped firing | `startingDeadlineSeconds` exceeded during a control-plane outage window — Kubernetes counts a missed run as permanently missed, it does not queue and catch up | `kubectl get cronjob` — check `lastScheduleTime` against the actual cron expression |
| Old completed Job's pods still visible for a long time | `ttlSecondsAfterFinished` unset — Kubernetes never garbage-collects it | Set a TTL, or manually `kubectl delete job` once logs are captured elsewhere |
| Two runs of the same CronJob overlap unexpectedly | `concurrencyPolicy: Allow` (the default) lets a slow run overlap with the next scheduled one | Set `concurrencyPolicy: Forbid` or `Replace` if overlapping runs would corrupt shared state |

> [!NOTE]
> A `CronJob`'s `lastScheduleTime` field updates the moment the **CronJob controller decides to create a
> Job**, not when that Job's pod actually finishes — a CronJob showing a recent `lastScheduleTime` with no
> corresponding successful Job underneath it means the Job itself failed to create or immediately failed,
> which points you back to the standard admission-webhook or scheduling diagnostics above, not a cron
> scheduling bug.

## A Full Worked Incident: A Multi-Symptom Production Outage

**Real production incidents rarely present as one clean symptom from the sections above — they cascade.**
Walk through a realistic version end to end: `checkout-service` starts returning 503s at 14:02 on a Friday.

```mermaid
sequenceDiagram
    participant Alert as Alertmanager
    participant Eng as On-call Engineer
    participant K8s as kubectl
    participant Node as Node ip-10-0-3-27

    Alert->>Eng: checkout-svc error rate 40%
    Eng->>K8s: kubectl get pods -n checkout -o wide
    K8s-->>Eng: 3/5 pods Running, 2/5 Pending
    Eng->>K8s: kubectl describe pod (Pending ones)
    K8s-->>Eng: "0/5 nodes available: 3 Insufficient memory"
    Eng->>K8s: kubectl top nodes
    K8s-->>Eng: ip-10-0-3-27 at 97% memory
    Eng->>K8s: kubectl describe node ip-10-0-3-27
    K8s-->>Eng: Condition MemoryPressure True
    Eng->>Node: journalctl -u kubelet
    Node-->>Eng: repeated evictions of BestEffort pods
    Eng->>K8s: kubectl get pods -A --field-selector spec.nodeName=ip-10-0-3-27
    K8s-->>Eng: a batch-job Deployment with no memory limits<br/>consumed all remaining allocatable memory
```

**Reconstructing the causal chain from the bottom up, the way a real postmortem timeline should read**: a
newly deployed batch-processing Deployment had no memory `limits` set (Part 2's Guaranteed/Burstable/
BestEffort QoS classes), so it consumed memory unbounded until the node crossed its `MemoryPressure`
eviction threshold. The kubelet started evicting `BestEffort` pods to reclaim memory — which happened to
include two `checkout-service` replicas that had never had resource requests/limits configured either,
because they'd been copy-pasted from an older manifest before the team standardized on setting them. Losing
two of five replicas pushed the remaining three past their own capacity, producing the 503s that paged
on-call in the first place. **The actual fix had three parts, not one**: add memory limits to the batch job
immediately to stop the bleeding, restore `checkout-service` to its full replica count, and — the part a
rushed incident response often skips — add a `PodDisruptionBudget` and enforce resource requests/limits via
an admission policy (Part 11) so an unbounded workload can never again evict a namespace it has nothing to
do with.

## A Note on `kubectl top` and metrics-server

**`kubectl top nodes`/`kubectl top pods`, used throughout this chapter, depend entirely on the
`metrics-server` add-on — it is not part of the core control plane, and its absence or unhealthiness
produces a specific, easily-misread error.**

```bash
kubectl top pods -n checkout
# error: Metrics API not available
```

| Cause | Fix |
|---|---|
| `metrics-server` was never installed (common on a fresh kubeadm cluster — it isn't bundled by default) | Install it; every managed offering (EKS, GKE, AKS) ships it pre-installed |
| `metrics-server` pod is crash-looping | Check its own logs — a very common cause is nodes using self-signed kubelet certificates without `--kubelet-insecure-tls` set, which `metrics-server` rejects by default |
| Freshly installed, queried immediately | It needs one full scrape interval (default 15s) before any data exists — `kubectl top` genuinely has nothing to show yet, this isn't a failure |

This matters diagnostically because a `kubectl top` failure is an infrastructure gap in your observability
tooling, not a signal about the workload you were trying to inspect — don't let a missing metrics-server
get mistaken for "the pod has no resource usage," and don't let debugging metrics-server itself derail an
incident where `describe`'s `Requests`/`Limits` fields and the Node's `Allocatable` are already enough to
reason about memory pressure without live metrics at all.

## Building Your Own Troubleshooting Runbook

**A personal, memorized command sequence beats searching for the right `kubectl` flag mid-incident — build
this once, keep it somewhere you can paste from under pressure.**

- [ ] `kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded` — every
      unhealthy pod, cluster-wide, in one line
- [ ] `kubectl get events -A --sort-by=.lastTimestamp | tail -30` — the most recent 30 things the cluster
      itself is complaining about
- [ ] `kubectl top nodes` and `kubectl top pods -A --sort-by=memory` — is this a resource-pressure incident?
- [ ] `kubectl get nodes -o wide` — any node `NotReady` or with a non-zero `SchedulingDisabled`?
- [ ] `kubectl get --raw='/readyz?verbose'` — is the control plane itself healthy?
- [ ] For a specific pod: `describe` → `logs --previous` → `kubectl debug --target` → node-level `crictl`,
      in that order, stopping as soon as one step gives you the answer

## Common Mistakes and Interview Traps

| Mistake | Why it's wrong | Correct approach |
|---|---|---|
| Running `kubectl delete pod` on a crash-looping pod before reading `describe`/events | Destroys the exit code and event history that would have told you the cause in seconds | Always `describe` and `logs --previous` first — a Deployment recreates the pod anyway |
| Assuming `CrashLoopBackOff` is itself the root cause | It's a restart-policy symptom describing *that* the container keeps dying, never *why* | Read the exit code and `Reason` field to find the actual cause underneath it |
| Treating Pod `phase: Running` as "healthy" | A `Running` pod can still be `0/1 Ready` and receiving zero traffic | Check the Ready count and Endpoints, not just phase |
| Restarting the kubelet as a first troubleshooting step for `NotReady` | Masks the underlying disk/memory pressure or PLEG cause without fixing it — it usually recurs | Check `describe node` Conditions and `crictl`/`journalctl` first |
| Debugging DNS by adding more retries in application code | Papers over a real infrastructure problem (NetworkPolicy, CoreDNS health) with app-level workarounds | Run the pod-IP → ClusterIP → DNS-name diagnostic ladder first |
| Assuming a `NetworkPolicy` is misconfigured the moment traffic breaks | Often it's working exactly as written — the missing piece is realizing a dependency was never covered by an explicit allow rule | Enumerate every real traffic source to the pod before assuming the policy syntax is wrong |

## Worked Practice Problems

**Problem 1**: A pod shows `Running`, `1/1 Ready`, `0` restarts — yet a Service in front of it routes zero
traffic to it, and `curl`-ing the pod's IP directly from another pod works fine. What's the most likely
cause, and what's the one command that confirms it?

*Answer*: A label-selector mismatch between the Service's `spec.selector` and the Pod's `metadata.labels` —
the pod is genuinely healthy, but the Service's Endpoints controller never considered it a match in the
first place. `kubectl get endpoints <service-name> -n <namespace>` confirms it instantly: if the pod's IP is
absent from the list despite the pod being Ready, the selector is the problem, not the pod's health.

**Problem 2**: A node shows `NotReady`, and `journalctl -u kubelet` shows the kubelet running and logging
normally with no crash. `crictl ps` takes almost 10 seconds to return a simple list. What layer is most
likely broken, and why does the slow `crictl ps` matter diagnostically?

*Answer*: The container runtime (containerd/CRI-O) itself is under I/O or resource starvation, most likely
disk I/O pressure — the kubelet is fine and can't be blamed here since it's logging normally, but its
`PLEG` health check depends on the runtime responding quickly, and a runtime taking seconds to answer a
trivial `ps`-equivalent request will eventually trip the PLEG health check and flip the node to `NotReady`
even though nothing about the kubelet process itself is broken.

**Problem 3**: After a team adds a default-deny `NetworkPolicy` to a namespace, the application's own
traffic still works, but Prometheus scraping stops within the hour and nobody notices until a much later,
unrelated incident. What does this reveal about how NetworkPolicies should be rolled out?

*Answer*: A NetworkPolicy's effect is defined by what it *fails to allow*, not by what it explicitly denies
— testing only the traffic you expect (the application's own request path) misses every implicit dependency
that used to work by default (metrics scraping, sidecar injection webhooks, DNS). The safer rollout is to
first enumerate every real ingress/egress source for the namespace (including monitoring, service mesh
control planes, and admission webhooks), write explicit allow rules for all of them, and only then apply
the policy — or roll it out in a non-enforcing/audit mode first if the CNI supports one, rather than
applying default-deny and waiting to see what breaks.

## Summary and What's Next

Troubleshooting isn't a separate skill from understanding Kubernetes's architecture — it's that same
architectural knowledge run in reverse, starting from a symptom instead of a spec. The six-layer model
(application → container → scheduling → node → cluster networking → control plane) gives you a search
order; `describe` and `events` give you the fastest signal at almost every layer; and `crictl`/node-level
tools are the fallback for the moment `kubectl` itself becomes part of the outage.

Part 11 turns to a different lens entirely: security. Many of the diagnostic techniques in this chapter
double as security-relevant checks (a NetworkPolicy default-deny gap is both an outage risk and a hardening
gap) — Part 11 covers RBAC, Pod Security Standards, supply chain security, and runtime security in the
depth the CKS exam expects, building directly on the admission-control chain introduced back in Part 1.