Architecture & Control Plane

Table of Contents#

1. What Problem Kubernetes Actually Solves
2. The Big Picture: Control Plane vs Worker Nodes
3. The API Server — The Front Door to Everything
4. etcd — Kubernetes's Memory
5. The Controller Manager and the Reconciliation Loop
6. The Scheduler, at a High Level
7. Worker Node Components
8. The Kubelet — The Node's Local Agent
9. kube-proxy — Making Services Actually Work
10. The Container Runtime
11. A Full Worked Journey: kubectl apply to a Running Pod
12. Declarative vs Imperative — The Core Philosophy
13. Common Mistakes
14. Worked Practice Problems
15. Summary and What's Next

---

What Problem Kubernetes Actually Solves#

Before any component-by-component detail, it's worth being able to answer the single most common opening Kubernetes interview question in one clean breath: what problem does Kubernetes actually solve?

Diagram

The single-sentence answer worth memorizing: "Kubernetes is a system that continuously works to make the actual state of your infrastructure match the desired state you've declared — placing containers on machines, restarting them when they fail, routing traffic to them, and scaling them — without a human manually doing any of that."

---

The Big Picture: Control Plane vs Worker Nodes#

Every Kubernetes cluster splits into two fundamentally different kinds of machines, each with a distinct job.

Diagram

Simple analogy: the Control Plane is like a restaurant's head office — it decides the menu (desired state), tracks inventory (current state), and issues instructions. Worker nodes are the actual kitchens where food (containers) really gets cooked and served. The head office never cooks anything itself — it only ever tells kitchens what to do and watches what's actually happening.

---

The API Server — The Front Door to Everything#

The API Server (kube-apiserver) is the single, central entry point for absolutely everything in Kubernetes — every kubectl command, every internal component, every automated controller talks to Kubernetes exclusively through this one component.

Diagram

A genuinely important architectural fact, worth stating explicitly: NOTHING in Kubernetes talks directly to etcd except the API Server. Every other component — the scheduler, controllers, kubelets — only ever reads and writes cluster state by calling the API Server, which is the sole gatekeeper to the actual stored data. This single-entry-point design is what makes authentication, authorization (RBAC, from the DevSecOps series), and validation possible to enforce consistently across the entire cluster.

Copy
# Every single kubectl command is really just an HTTP request
# to the API server — you can see this directly:
kubectl get pods -v=8 2>&1 | grep "GET https"
# GET https://<api-server>/api/v1/namespaces/default/pods

---

etcd — Kubernetes's Memory#

etcd is a distributed, consistent key-value store — and it is, quite literally, the entire source of truth for a Kubernetes cluster's state. Every object (every Pod, Deployment, Service, Secret) is stored here.

Diagram

Why etcd is a genuinely critical, high-stakes component — worth stressing explicitly in an interview: if etcd is lost or corrupted with no backup, the cluster effectively has amnesia — it has no memory of what should be running, where, or how it was configured. This directly connects to the Disaster Recovery topic (topic 11) in this course: etcd backups are one of the single most critical, non-negotiable disaster-recovery practices for any self-managed Kubernetes cluster.

Copy
# Take a backup of etcd (run on a control plane node)
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify the snapshot
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot.db --write-out=table

Why etcd specifically needs strong consistency (CP, not AP, using the CAP theorem vocabulary from the Reliability & Architecture Patterns series): if two control plane replicas ever disagreed about whether a Pod exists, the cluster's behavior would become genuinely unpredictable. This is exactly why etcd uses the Raft consensus algorithm, requiring a majority (quorum) of its members to agree before any write is considered committed — directly the same quorum principle (W + R > N) covered in that earlier tutorial.

Diagram

Why etcd clusters always use an odd number of nodes, a genuinely common, sharp interview question: a 4-node cluster still only tolerates 1 failure (needs 3 of 4 to agree — same as a 3-node cluster needing 2 of 3), but costs an entire extra node for zero additional fault tolerance. An odd number is always the efficient choice.

---

The Controller Manager and the Reconciliation Loop#

This is arguably the single most important conceptual idea in all of Kubernetes — genuinely worth spending real time to understand deeply, since almost everything else in the system is built on top of this one pattern.

Diagram

This loop — observe, compare, act, repeat forever — is called a controller, and it's the fundamental unit of automation in Kubernetes. A Controller Manager process runs dozens of these loops simultaneously, each one responsible for one specific type of object (a Deployment Controller, a ReplicaSet Controller, a Node Controller, and many more).

Simple analogy: think of a home thermostat. It doesn't "turn on the heat once" — it continuously checks the current temperature against your desired setting, and takes action (heat on/off) whenever there's a gap, forever, without you doing anything. Every Kubernetes controller works exactly this way, just applied to cluster objects instead of temperature.

Diagram

The direct, practical payoff of this design, worth stating explicitly: this is exactly why Kubernetes self-heals. If a node dies and takes a pod with it, you don't need any human or script to notice and react — the relevant controller notices the gap between desired (3 replicas) and actual (2 replicas) on its very next reconciliation pass (which happens continuously, many times a second) and simply creates a replacement, automatically, with zero human involvement.

---

The Scheduler, at a High Level#

The Scheduler (kube-scheduler) has one specific job: when a new Pod is created with no node assigned yet, decide which node it should actually run on. (The full mechanics — filtering, scoring, affinity rules — get their own deep dive in Part 2.)

Diagram

A genuinely important point worth stating explicitly: the Scheduler only ever DECIDES and RECORDS which node a pod should run on — it never actually starts a container itself. That job belongs entirely to the kubelet on the chosen node, covered next.

---

Worker Node Components#

Every worker node runs three essential components, each with one specific job.

Diagram

---

The Kubelet — The Node's Local Agent#

The kubelet is the only Kubernetes component running on a worker node that talks directly to the API Server. It's responsible for making sure the containers assigned to its node are actually running, healthy, and match their spec.

Diagram

Liveness vs. readiness probes — a genuinely common, specific interview distinction:

Diagram

Why this distinction matters practically, and it's a classic interview trap: a container that's temporarily overwhelmed and slow (but not actually broken) should fail its readiness probe (stop receiving new traffic until it catches up) but should absolutely NOT fail its liveness probe — killing and restarting a container that's just temporarily busy makes the problem worse, not better, by throwing away whatever progress it had made and adding restart overhead on top of an already-struggling situation.

Copy
apiVersion: v1
kind: Pod
metadata:
  name: checkout
spec:
  containers:
    - name: app
      image: checkout:1.2.3
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 10
        periodSeconds: 10
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        periodSeconds: 5
        failureThreshold: 2

---

kube-proxy — Making Services Actually Work#

kube-proxy runs on every node and is responsible for implementing the actual networking rules that make a Kubernetes Service (a stable, virtual IP that load-balances across a changing set of pods) actually work.

Diagram

In plain terms: without kube-proxy, a Service's stable virtual IP would just be an abstract idea with nothing actually making it work — kube-proxy is the component that turns "traffic to this Service" into "actual network rules routing to real, currently-healthy pod IPs," updated automatically every time pods come and go. (The full mechanics of Services get their own deep dive in Part 3.)

---

The Container Runtime#

The actual layer that runs containers — the lowest-level component in this whole stack, sitting directly on top of the Linux kernel primitives (namespaces and cgroups) covered in the Linux & Networking Fundamentals series.

Diagram

Why the CRI (Container Runtime Interface) matters, worth knowing by name: Kubernetes doesn't hardcode a dependency on any one specific container runtime — it talks to whatever runtime is installed through this standard interface. This is exactly why Kubernetes could cleanly deprecate direct Docker support (a well-known, sometimes misunderstood industry event) without actually breaking anything for end users — Docker-built images still work fine, because the image format (OCI-compliant) is separate from the runtime that executes containers, and any CRI-compliant runtime (like containerd, which Docker itself is built on top of) can run them.

---

A Full Worked Journey: kubectl apply to a Running Pod#

Tying every component in this tutorial together into one complete, step-by-step story — genuinely one of the most valuable things to be able to narrate fluently in an interview.

Diagram

A strong interview answer walks through this exact sequence, naming every component and its specific, narrow responsibility — this single narrative demonstrates the entire architecture in one coherent story, rather than a list of disconnected component definitions.

---

Declarative vs Imperative — The Core Philosophy#

A final, foundational concept worth stating explicitly, since it explains why Kubernetes is designed the way it is.

Diagram

A clean, memorable interview line: "Kubernetes is fundamentally declarative — you describe what you want, not the steps to get there, and the reconciliation loop pattern is the engine that continuously, automatically closes the gap between what you asked for and what's actually running, which is exactly what makes the whole system self-healing without any human in the loop."

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A Deployment specifies replicas: 5, but kubectl get pods shows only 3 running, with no error events visible. Walk through which components are involved in eventually fixing this, and how.

Answer: The Deployment Controller (part of the Controller Manager), continuously reconciling, compares the desired state (5 replicas) against the actual observed state (3 pods) via the API Server, and on its next reconciliation pass creates 2 new Pod objects to close the gap — with no node assigned yet. The Scheduler, watching for unscheduled pods, picks a suitable node for each of the 2 new pods and records that decision via the API Server. Each chosen node's kubelet, watching for pods assigned to its own node, sees the new assignment and instructs the container runtime to actually start the containers. This entire chain happens automatically, with zero human intervention, purely as a consequence of the reconciliation loop pattern.

Problem 2: A container is under heavy, legitimate load and its response times have temporarily climbed above its liveness probe's timeout threshold, causing kubelet to repeatedly restart it — making the underlying overload problem even worse. What's misconfigured, and what's the fix?

Answer: The liveness probe is being used to judge something it shouldn't — genuine, temporary business load isn't the same as "this container is broken/deadlocked," which is what liveness probes should be reserved for. The fix: loosen the liveness probe's timeout/failure threshold so temporary slowness under real load doesn't trigger a restart, and rely on the readiness probe instead to temporarily pull the pod out of Service rotation during genuine overload — letting it finish its existing work and recover on its own, rather than repeatedly restarting it and discarding progress.

Problem 3: Someone argues a 4-node etcd cluster is "safer" than a 3-node one because "more nodes means more redundancy." Explain why this reasoning is flawed.

Answer: etcd requires a strict majority (quorum) to agree before any write commits. A 3-node cluster needs 2 of 3 to agree and can tolerate exactly 1 node failing. A 4-node cluster needs 3 of 4 to agree — and can STILL only tolerate exactly 1 node failing (losing 2 of 4 breaks the majority requirement just as it would with 3 nodes). The 4th node adds real cost (more compute, more storage, more network overhead for consensus) without improving fault tolerance at all compared to the 3-node setup — which is exactly why etcd clusters are always sized with an odd number of members.

---

Summary and What's Next#

• Kubernetes's core job: continuously make the actual state of the cluster match the desired state you declare, automatically — this is the entire point of the system.
• The cluster splits into the Control Plane (API Server, etcd, Scheduler, Controller Manager — the "brain") and Worker Nodes (kubelet, kube-proxy, container runtime — where containers actually run).
• The API Server is the sole gatekeeper to all cluster state — nothing else talks directly to etcd, which is the cluster's entire source of truth and requires a quorum-based majority (always an odd number of members) to commit any write.
• The reconciliation loop (observe -> compare -> act, forever) is the single most important pattern in Kubernetes — it's the mechanism behind every controller and the entire reason the system self-heals without human intervention.
• The Scheduler only decides where a pod should run; the kubelet on that specific node is what actually starts and monitors it.
• Liveness probes answer "should this be restarted"; readiness probes answer "should this receive traffic right now" — conflating the two is a classic, damaging misconfiguration.
• kube-proxy turns a Service's stable virtual IP into real, working network rules across every node.
• Kubernetes's declarative philosophy (describe the desired end state, not the steps to get there) is precisely what makes kubectl apply idempotent and the whole system self-healing.

Continue to Part 2 (02-scheduling-and-workloads.md) for a full deep dive into exactly how the Scheduler makes its placement decisions, and the different workload objects (Deployments, StatefulSets, DaemonSets, Jobs) built on top of this foundation.