Troubleshooting Kubernetes, Systematically

Assumes you're comfortable with the control plane and worker node components from Part 1, the workload objects and scheduling mechanics from Part 2, and the networking model from Part 3 — this chapter is about diagnosing failures in those systems, not introducing them.

Table of Contents#

1. Why Troubleshooting Deserves Its Own Chapter
2. The Six-Layer Mental Model for Any Kubernetes Failure
3. The First Move, Always: get, describe, events
4. Reading Pod Phase and Container State Correctly
5. CrashLoopBackOff — the Full Diagnostic Tree
6. OOMKilled — Diagnosing and Fixing Memory Failures
7. ImagePullBackOff and ErrImagePull — the Full Diagnostic Tree
8. Admission Webhook Rejections — Pods That Never Even Get Created
9. Pending Pods — Diagnosing Scheduling Failures
10. Running But Not Ready — Probe Failures
11. Choosing the Right Tool: exec, kubectl debug, and Ephemeral Containers
12. Debugging Shell-less and Distroless Containers
13. Log Retrieval Gotchas Worth Knowing
14. When kubectl Itself Can't Help: Debugging a Broken Node
15. Node NotReady — the Full Diagnostic Tree
16. Cluster Networking Failures — Pod-to-Pod and Pod-to-Service
17. DNS Resolution Failures — the CoreDNS Diagnostic Tree
18. "It Worked Until We Added a NetworkPolicy"
19. Control Plane Failures — When the API Server or etcd Is Sick
20. Storage Failures — PVC Pending and Mount Timeouts
21. Troubleshooting Jobs and CronJobs
22. A Full Worked Incident: A Multi-Symptom Production Outage
23. A Note on kubectl top and metrics-server
24. Building Your Own Troubleshooting Runbook
25. Common Mistakes and Interview Traps
26. Worked Practice Problems
27. Summary and What's Next

Why Troubleshooting Deserves Its Own Chapter#

Troubleshooting is the single largest domain on the CKA exam — 30% of the score — yet it's the one skill no reference doc actually teaches, because reference docs describe how things work when they're healthy, not how to work backward from a symptom to a cause. Parts 1 through 9 of this series built a detailed mental model of how the pieces fit together; this chapter inverts that model into a diagnostic tool. Every section below follows the same shape: a symptom you'd actually see, a small number of root causes ranked by how often they're the real one, and the exact commands that tell them apart.

The throughline system from earlier parts continues here: a checkout namespace running checkout-service talking to a catalog-service and a Postgres-backed inventory-service, all sitting behind a Service and, in later sections, a NetworkPolicy. Every worked example below happens inside that same small system.

The Six-Layer Mental Model for Any Kubernetes Failure#

Every Kubernetes failure lives at exactly one of six layers, and knowing which layer you're in before you start running commands cuts your search space by roughly 5x. Practitioners who troubleshoot slowly usually aren't missing knowledge — they're jumping between layers randomly instead of eliminating them in order.

This diagram isn't a call sequence — it's a search order. Layer 1-2 failures (a single pod crash-looping) are by far the most common and the cheapest to rule in or out first; Layer 6 failures (the whole control plane is unhealthy) are rare but explain symptoms across every namespace at once. A describe pod in the first ten seconds usually tells you which layer you're actually in, which is why the next section is the mandatory starting point for everything that follows.

The First Move, Always: get, describe, events#

Before touching logs, before execing into anything, run these three commands — they cost nothing and eliminate entire categories of cause.

kubectl get pods -n checkout -o wide
kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout
kubectl get events -n checkout --sort-by=.lastTimestamp

kubectl get -o wide shows the phase, restart count, and node in one line — restart count alone tells you whether you're looking at a fresh failure or a pod that's been crash-looping for hours. describe prints the pod spec, current container states, resource requests, and — critically — the Events section at the bottom, which is the Scheduler, kubelet, and controllers narrating what they tried to do and why it failed, in their own words. kubectl get events cluster- or namespace-wide catches things describe on one pod misses, like a NetworkPolicy controller rejecting a rule or a PVC provisioner failing silently.

Reading Pod Phase and Container State Correctly#

A Pod has one phase (a coarse, cluster-wide summary) and each of its containers has its own finer-grained state — conflating the two is a common source of confused troubleshooting.

Underneath a Running phase, each container independently reports one of three states:

The Pod phase can say Running while the container you actually care about is Waiting with CrashLoopBackOff — a multi-container pod where a sidecar is healthy but the main container is looping will show Running at the pod level with 1/2 ready containers. Always check the per-container state in describe's output, not just the top-line phase.

CrashLoopBackOff — the Full Diagnostic Tree#

CrashLoopBackOff means the container starts, then exits, repeatedly — it is a symptom, never the root cause itself, and the backoff delay (10s, doubling up to a 5-minute cap) is Kubernetes protecting itself from hammering a broken container, not a bug.

Reading exit codes is the fastest triage step and the one most people skip. kubectl describe pod's Last State: Terminated block shows both a numeric Exit Code and a human Reason — cross-referencing the two takes seconds and immediately rules out or confirms half the tree above.

kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -A5 "Last State"
kubectl logs checkout-service-7d8f9c-x2k9p -n checkout --previous

OOMKilled — Diagnosing and Fixing Memory Failures#

OOMKilled (exit code 137, Reason: OOMKilled in describe) means the kernel's cgroup memory controller killed the container for exceeding its memory limit — this is enforced by the kernel, not by Kubernetes polling and deciding, so it happens instantly and with zero graceful shutdown.

kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -B2 -A2 OOMKilled
kubectl top pod checkout-service-7d8f9c-x2k9p -n checkout

Explaining this two levels deep, the way a real postmortem should: the symptom is "checkout-service keeps restarting under load." The immediate cause is the container hit its 256Mi memory limit. The underlying condition is that the limit was copy-pasted from a template chart during initial rollout and never revisited after the team added an in-memory response cache six months later — nobody owns a periodic review of resource requests/limits against actual usage, so the mismatch sat silent until a traffic spike finally pushed it over the edge.

ImagePullBackOff and ErrImagePull — the Full Diagnostic Tree#

ErrImagePull is the first failed attempt; ImagePullBackOff is Kubernetes retrying with the same exponential backoff as CrashLoopBackOff — the fix is always found in describe's Events, never in kubectl logs, because the container never started.

kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -A3 Events
# Confirm the exact image string being requested:
kubectl get pod checkout-service-7d8f9c-x2k9p -n checkout -o jsonpath='{.spec.containers[*].image}'

A genuinely common "worked on my machine" trap: the image pulls fine locally because docker login already authenticated your workstation to the private registry, but the cluster's nodes (or the specific ServiceAccount the pod runs as) never got an imagePullSecrets reference — this is a pull-credential problem, not an image problem, and it will pass every local test while failing 100% of the time in the cluster.

Admission Webhook Rejections — Pods That Never Even Get Created#

This failure looks nothing like the others in this chapter, because it happens before a Pod object even exists — kubectl apply itself returns an error, and there's no pod to describe at all. Part 1 covered the admission control chain (authentication → authorization → mutating webhooks → object schema validation → validating webhooks); a validating webhook rejecting the request is one of the most common "why won't this even create" failures in a cluster running policy engines like OPA Gatekeeper or Kyverno (Part 11).

kubectl apply -f checkout-deployment.yaml
# Error from server (Forbidden): error when creating "checkout-deployment.yaml":
# admission webhook "validate-resources.kyverno.svc" denied the request:
# resource limits are required on every container

The error message almost always names the exact webhook and the exact rule it violated — read it literally before assuming the manifest itself is malformed. A second, more confusing variant of this failure: the webhook's own backend (the Kyverno/Gatekeeper pod serving the webhook) is down or unreachable, in which case the error instead reads something like failed calling webhook ... context deadline exceeded or connection refused — that's a webhook-availability problem, not a policy violation, and the fix is restoring the policy engine's own pods rather than editing your manifest.

kubectl get validatingwebhookconfigurations
kubectl get pods -n kyverno   # or gatekeeper-system — confirm the webhook's backing pods are healthy

Pending Pods — Diagnosing Scheduling Failures#

A Pending pod has been accepted by the API Server but the Scheduler has not (yet, or ever) found a node that satisfies every one of its constraints — describe's Events section names the exact constraint that failed, so this is almost never a guessing game.

kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -A10 Events
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory

Running But Not Ready — Probe Failures#

A pod showing Running with 0/1 or a fractional Ready count (e.g. 1/2) is not a scheduling or crash problem — it means the container process is alive but its readiness probe is failing, so the Service deliberately excludes it from Endpoints (Part 3) rather than sending it traffic it can't yet serve.

kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -A5 "Readiness probe failed"
kubectl get endpoints checkout-svc -n checkout

The label-selector case deserves calling out on its own because it produces a confusing symptom: the pod is 1/1 Ready, kubectl exec into it and curling the app locally works fine, yet the Service still routes zero traffic to it. The cause is almost always that the Service's selector and the Pod's labels were both edited independently during a refactor and silently drifted apart — kubectl get endpoints shows zero or wrong addresses, which is the tell that this is a selector-matching problem, not an application health problem at all.

Choosing the Right Tool: exec, kubectl debug, and Ephemeral Containers#

These three tools solve different problems, and reaching for the wrong one wastes time or simply doesn't work.

# Attach a debug container sharing the target's process namespace —
# ps aux inside it shows the real app's processes:
kubectl debug checkout-service-7d8f9c-x2k9p -n checkout \
  -it --image=nicolaka/netshoot --target=checkout-service

Ephemeral containers are recorded in the API audit log and never modify the original container — this is precisely why they satisfy production change-control requirements that a kubectl exec into a modified, debug-tooling-injected image would not.

Debugging Shell-less and Distroless Containers#

A distroless or scratch-based production image (no shell, no package manager, no /bin/sh) is a deliberate, defensible security choice — smaller attack surface, fewer CVEs to patch — but it means kubectl exec -it ... -- sh fails outright with OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found.

kubectl debug checkout-service-7d8f9c-x2k9p -n checkout \
  -it --image=busybox --target=checkout-service
# Inside the debug container:
ls -l /proc/1/root/            # the target container's filesystem, read-only
cat /proc/1/environ | tr '\0' '\n'   # its actual environment variables

The ephemeral debug container shares the target's process namespace (via --target), so /proc/1/ is the main application's process — its filesystem, open file descriptors, and environment are all inspectable from a fully-tooled container sitting alongside it, without ever needing a shell inside the distroless image itself.

Log Retrieval Gotchas Worth Knowing#

kubectl logs has several sharp edges that produce "logs are empty/wrong/missing" confusion often mistaken for an application logging bug.

kubectl logs checkout-service-7d8f9c-x2k9p -n checkout -c checkout-service --previous
kubectl logs checkout-service-7d8f9c-x2k9p -n checkout -c wait-for-db   # an init container, by name
kubectl logs -n checkout -l app=checkout-service --all-containers=true --prefix --tail=50

The -l/--all-containers/--prefix combination above is worth memorizing on its own — it tails the last 50 lines from every container across every pod matching a label selector, each line prefixed with its source pod/container, which is frequently faster than tailing pods one at a time when a Deployment has several replicas and you don't yet know which one is misbehaving.

When kubectl Itself Can't Help: Debugging a Broken Node#

If the kubelet on a node has crashed or the container runtime is wedged, kubectl commands targeting that node's pods will hang or time out — at that point, the debugging has to happen on the node itself, one layer below the Kubernetes API entirely.

crictl is the single most important tool here — it speaks the same Container Runtime Interface protocol the kubelet uses, but works even when the kubelet process itself is the thing that's broken, because it talks directly to containerd/CRI-O's own socket.

systemctl status kubelet
journalctl -u kubelet -n 200 --no-pager
crictl ps -a                      # every container the runtime knows about, kubelet or not
crictl logs <container-id>
crictl inspect <container-id> | grep -i oom

Node NotReady — the Full Diagnostic Tree#

A node reports NotReady when the API Server stops receiving its heartbeat within the node-monitor grace period (default 40 seconds after a 10-second heartbeat interval) — every pod scheduled on it is then treated as unreachable, and after pod-eviction-timeout (default 5 minutes) the Controller Manager starts evicting and rescheduling them elsewhere.

kubectl describe node ip-10-0-3-27.ec2.internal | grep -A15 Conditions
kubectl get events --field-selector involvedObject.name=ip-10-0-3-27.ec2.internal

Explaining PLEG is not healthy two levels deep, since it's one of the more opaque messages a node throws: the symptom is the kubelet logging PLEG is not healthy: pleg was last seen active Nm ago. The immediate cause is the kubelet's Pod Lifecycle Event Generator — the loop that polls the container runtime for state changes — timed out waiting for a response. The underlying condition is almost always the container runtime itself is starved, most commonly because the node's disk is under heavy I/O pressure (log rotation misconfigured, a runaway container writing gigabytes to emptyDir) slow enough that even crictl ps takes multiple seconds to return — fixing the disk I/O bottleneck resolves the PLEG error as a side effect, restarting the kubelet alone usually just repeats the same failure minutes later.

Cluster Networking Failures — Pod-to-Pod and Pod-to-Service#

When individual pods look healthy (Running, 1/1 Ready) but traffic between them fails, the problem has moved from "is this pod OK" to "can the network actually deliver a packet between these two IPs" — a different, layered investigation.

This ladder — pod IP directly, then Service ClusterIP, then DNS name — isolates the failing layer in three commands, because each step adds exactly one more piece of the networking stack on top of the previous, successful step:

kubectl exec -it catalog-service-xyz -n catalog -- curl -m 3 http://10.244.1.15:8080/healthz
kubectl exec -it catalog-service-xyz -n catalog -- curl -m 3 http://checkout-svc.checkout.svc.cluster.local:80/healthz
kubectl get endpoints checkout-svc -n checkout
kubectl get pods -n kube-system -l k8s-app=kube-proxy

If step one (direct pod IP) fails, the fault is at the CNI layer — check the CNI daemonset's pods for crash-loops (kubectl get pods -n kube-system -l k8s-app=cilium or the equivalent for your CNI) and confirm routes exist between the two nodes hosting the pods. If step one succeeds but step two (Service ClusterIP) fails, kube-proxy is the suspect — confirm its pods are healthy and, on nodes using IPVS mode, ipvsadm -Ln shows the expected virtual server entries.

DNS Resolution Failures — the CoreDNS Diagnostic Tree#

"It can't resolve the service name" is one of the most common tickets in any Kubernetes cluster, and it has a small, well-ordered set of causes — work through them in this order, not randomly.

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl get endpoints kube-dns -n kube-system
kubectl run dns-debug --rm -it --image=busybox:1.36 -n checkout -- nslookup checkout-svc.checkout.svc.cluster.local
kubectl get configmap coredns -n kube-system -o yaml

The ndots:5 trap is worth understanding precisely, since it's genuinely confusing the first time it bites. A pod's default /etc/resolv.conf sets ndots:5, meaning any name with fewer than 5 dots is tried against every entry in the search list (checkout.svc.cluster.local, svc.cluster.local, cluster.local, then the node's own search domains) before being tried as an absolute name — for an external hostname like api.stripe.com (2 dots), that means up to 4 failed internal lookups before the 5th attempt finally succeeds externally, adding real, measurable latency to every single external call a pod makes. This is why high-throughput services often set dnsConfig.options to lower ndots, or use a trailing dot (api.stripe.com.) to force an absolute lookup.

"It Worked Until We Added a NetworkPolicy"#

This is one of the most common self-inflicted outages in any cluster with a security team actively hardening it — a genuinely correct NetworkPolicy silently breaks a dependency nobody realized existed, because Kubernetes NetworkPolicies are allow-list, and once any policy selects a pod, every previously implicit "allow all" traffic pattern to it disappears at once.

kubectl get networkpolicy -n checkout
kubectl describe networkpolicy checkout-default-deny -n checkout

Control Plane Failures — When the API Server or etcd Is Sick#

Control plane failures are the rarest layer in this chapter but the most severe, because kubectl itself becomes part of what's broken — every diagnostic technique above depends on a healthy API Server to run.

kubectl get --raw='/readyz?verbose'
kubectl get componentstatuses    # deprecated but still informative on many clusters
curl -k https://localhost:2379/health --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key

/readyz?verbose is worth memorizing precisely because it self-diagnoses: it returns a line-by-line pass/fail for every internal health check the API Server runs (etcd connectivity, informer sync status, shutdown state), which is almost always faster than manually checking each dependency by hand.

Storage Failures — PVC Pending and Mount Timeouts#

A pod stuck Pending with a storage-related Events message, or Running but stuck at ContainerCreating for minutes, points at the CSI/storage layer (Part 3) rather than compute scheduling.

kubectl get pvc -n checkout
kubectl describe pvc checkout-data -n checkout
kubectl describe pod checkout-service-7d8f9c-x2k9p -n checkout | grep -A5 "FailedMount\|FailedAttachVolume"

Troubleshooting Jobs and CronJobs#

Jobs and CronJobs (Part 2) fail in ways specific to their run-to-completion model, and the standard pod-troubleshooting steps above only get you partway there — the Job/CronJob object itself carries additional state worth checking first.

kubectl describe job checkout-nightly-reconcile -n checkout
kubectl get pods -n checkout -l job-name=checkout-nightly-reconcile-29384710 --show-labels

A Full Worked Incident: A Multi-Symptom Production Outage#

Real production incidents rarely present as one clean symptom from the sections above — they cascade. Walk through a realistic version end to end: checkout-service starts returning 503s at 14:02 on a Friday.

Reconstructing the causal chain from the bottom up, the way a real postmortem timeline should read: a newly deployed batch-processing Deployment had no memory limits set (Part 2's Guaranteed/Burstable/ BestEffort QoS classes), so it consumed memory unbounded until the node crossed its MemoryPressure eviction threshold. The kubelet started evicting BestEffort pods to reclaim memory — which happened to include two checkout-service replicas that had never had resource requests/limits configured either, because they'd been copy-pasted from an older manifest before the team standardized on setting them. Losing two of five replicas pushed the remaining three past their own capacity, producing the 503s that paged on-call in the first place. The actual fix had three parts, not one: add memory limits to the batch job immediately to stop the bleeding, restore checkout-service to its full replica count, and — the part a rushed incident response often skips — add a PodDisruptionBudget and enforce resource requests/limits via an admission policy (Part 11) so an unbounded workload can never again evict a namespace it has nothing to do with.

A Note on kubectl top and metrics-server#

kubectl top nodes/kubectl top pods, used throughout this chapter, depend entirely on the metrics-server add-on — it is not part of the core control plane, and its absence or unhealthiness produces a specific, easily-misread error.

kubectl top pods -n checkout
# error: Metrics API not available

This matters diagnostically because a kubectl top failure is an infrastructure gap in your observability tooling, not a signal about the workload you were trying to inspect — don't let a missing metrics-server get mistaken for "the pod has no resource usage," and don't let debugging metrics-server itself derail an incident where describe's Requests/Limits fields and the Node's Allocatable are already enough to reason about memory pressure without live metrics at all.

Building Your Own Troubleshooting Runbook#

A personal, memorized command sequence beats searching for the right kubectl flag mid-incident — build this once, keep it somewhere you can paste from under pressure.

• kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded — every unhealthy pod, cluster-wide, in one line
• kubectl get events -A --sort-by=.lastTimestamp | tail -30 — the most recent 30 things the cluster itself is complaining about
• kubectl top nodes and kubectl top pods -A --sort-by=memory — is this a resource-pressure incident?
• kubectl get nodes -o wide — any node NotReady or with a non-zero SchedulingDisabled?
• kubectl get --raw='/readyz?verbose' — is the control plane itself healthy?
• For a specific pod: describe → logs --previous → kubectl debug --target → node-level crictl, in that order, stopping as soon as one step gives you the answer

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A pod shows Running, 1/1 Ready, 0 restarts — yet a Service in front of it routes zero traffic to it, and curl-ing the pod's IP directly from another pod works fine. What's the most likely cause, and what's the one command that confirms it?

Answer: A label-selector mismatch between the Service's spec.selector and the Pod's metadata.labels — the pod is genuinely healthy, but the Service's Endpoints controller never considered it a match in the first place. kubectl get endpoints <service-name> -n <namespace> confirms it instantly: if the pod's IP is absent from the list despite the pod being Ready, the selector is the problem, not the pod's health.

Problem 2: A node shows NotReady, and journalctl -u kubelet shows the kubelet running and logging normally with no crash. crictl ps takes almost 10 seconds to return a simple list. What layer is most likely broken, and why does the slow crictl ps matter diagnostically?

Answer: The container runtime (containerd/CRI-O) itself is under I/O or resource starvation, most likely disk I/O pressure — the kubelet is fine and can't be blamed here since it's logging normally, but its PLEG health check depends on the runtime responding quickly, and a runtime taking seconds to answer a trivial ps-equivalent request will eventually trip the PLEG health check and flip the node to NotReady even though nothing about the kubelet process itself is broken.

Problem 3: After a team adds a default-deny NetworkPolicy to a namespace, the application's own traffic still works, but Prometheus scraping stops within the hour and nobody notices until a much later, unrelated incident. What does this reveal about how NetworkPolicies should be rolled out?

Answer: A NetworkPolicy's effect is defined by what it fails to allow, not by what it explicitly denies — testing only the traffic you expect (the application's own request path) misses every implicit dependency that used to work by default (metrics scraping, sidecar injection webhooks, DNS). The safer rollout is to first enumerate every real ingress/egress source for the namespace (including monitoring, service mesh control planes, and admission webhooks), write explicit allow rules for all of them, and only then apply the policy — or roll it out in a non-enforcing/audit mode first if the CNI supports one, rather than applying default-deny and waiting to see what breaks.

Summary and What's Next#

Troubleshooting isn't a separate skill from understanding Kubernetes's architecture — it's that same architectural knowledge run in reverse, starting from a symptom instead of a spec. The six-layer model (application → container → scheduling → node → cluster networking → control plane) gives you a search order; describe and events give you the fastest signal at almost every layer; and crictl/node-level tools are the fallback for the moment kubectl itself becomes part of the outage.

Part 11 turns to a different lens entirely: security. Many of the diagnostic techniques in this chapter double as security-relevant checks (a NetworkPolicy default-deny gap is both an outage risk and a hardening gap) — Part 11 covers RBAC, Pod Security Standards, supply chain security, and runtime security in the depth the CKS exam expects, building directly on the admission-control chain introduced back in Part 1.