# Kubernetes Deep Dive — Part 1: Architecture & Control Plane
> **Series:** Kubernetes Deep Dive (1 of 9)
> **Part 1:** This file — Architecture & Control Plane
> **Part 2:** `02-scheduling-and-workloads.md` — Scheduling & Workload Objects
> **Part 3:** `03-networking-and-storage.md` — Networking (CNI) & Storage (CSI)
> **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators
> **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE
> **Part 6:** `06-onprem-and-cluster-provisioning.md` — On-Prem & Self-Managed Kubernetes
> **Part 7:** `07-eks-deep-dive.md` — Amazon EKS in Production Depth
> **Part 8:** `08-gateway-api-and-envoy-gateway.md` — Gateway API & Envoy Gateway
> **Part 9:** `09-gateway-api-across-providers.md` — Gateway API Across GKE, EKS, AKS & On-Prem
> **Questions:** `questions.md`
## Table of Contents
1. [What Problem Kubernetes Actually Solves](#what-problem-kubernetes-actually-solves)
2. [The Big Picture: Control Plane vs Worker Nodes](#the-big-picture-control-plane-vs-worker-nodes)
3. [The API Server — The Front Door to Everything](#the-api-server--the-front-door-to-everything)
4. [The Admission Control Chain: Authentication, Authorization, Admission](#the-admission-control-chain-authentication-authorization-admission)
5. [Mutating and Validating Admission Webhooks, In Depth](#mutating-and-validating-admission-webhooks-in-depth)
6. [kube-apiserver Flags That Matter Operationally](#kube-apiserver-flags-that-matter-operationally)
7. [etcd — Kubernetes's Memory](#etcd--kubernetess-memory)
8. [etcd's Raft Consensus, In Depth](#etcds-raft-consensus-in-depth)
9. [etcd Compaction, Defragmentation, and Alarms](#etcd-compaction-defragmentation-and-alarms)
10. [The Controller Manager and the Reconciliation Loop](#the-controller-manager-and-the-reconciliation-loop)
11. [Leader Election — How Control Plane Components Avoid Split-Brain](#leader-election--how-control-plane-components-avoid-split-brain)
12. [The Scheduler, at a High Level](#the-scheduler-at-a-high-level)
13. [Worker Node Components](#worker-node-components)
14. [The Kubelet — The Node's Local Agent](#the-kubelet--the-nodes-local-agent)
15. [Node Heartbeats and Node Conditions](#node-heartbeats-and-node-conditions)
16. [kube-proxy — Making Services Actually Work](#kube-proxy--making-services-actually-work)
17. [The Container Runtime](#the-container-runtime)
18. [API Server High Availability](#api-server-high-availability)
19. [Extending the API: Aggregated API Servers and CRDs at the Wire Level](#extending-the-api-aggregated-api-servers-and-crds-at-the-wire-level)
20. [Control Plane Component Ports and Communication, Reference](#control-plane-component-ports-and-communication-reference)
21. [A Full Worked Journey: `kubectl apply` to a Running Pod](#a-full-worked-journey-kubectl-apply-to-a-running-pod)
22. [Kubernetes API Versioning and Deprecation Policy](#kubernetes-api-versioning-and-deprecation-policy)
23. [The API Server's Own Health Endpoints](#the-api-servers-own-health-endpoints)
24. [Declarative vs Imperative — The Core Philosophy](#declarative-vs-imperative--the-core-philosophy)
25. [Common Mistakes](#common-mistakes)
26. [Worked Practice Problems](#worked-practice-problems)
27. [Summary and What's Next](#summary-and-whats-next)
---
## What Problem Kubernetes Actually Solves
Before any component-by-component detail, it's worth being able to answer the single most common opening Kubernetes interview question in one clean breath: **what problem does Kubernetes actually solve?**
```mermaid
graph TD
Problem["Running containers at
scale, reliably, by hand"] --> P1["Which server should this
container run on?"]
Problem --> P2["What happens when a
server dies?"]
Problem --> P3["How does one container
find and talk to another?"]
Problem --> P4["How do I roll out a new
version without downtime?"]
Problem --> P5["How do I scale up/down
automatically?"]
K8s["Kubernetes automates
ALL of these decisions,
continuously, based on a
DESIRED STATE you declare"] -.-> Problem
```
**The single-sentence answer worth memorizing:** "Kubernetes is a system that continuously works to make the *actual* state of your infrastructure match the *desired* state you've declared — placing containers on machines, restarting them when they fail, routing traffic to them, and scaling them — without a human manually doing any of that."
---
## The Big Picture: Control Plane vs Worker Nodes
Every Kubernetes cluster splits into two fundamentally different kinds of machines, each with a distinct job.
```mermaid
graph TD
subgraph "Control Plane (the BRAIN)"
API["API Server"]
ETCD["etcd"]
Sched["Scheduler"]
CM["Controller Manager"]
end
subgraph "Worker Node 1"
Kubelet1["kubelet"]
Proxy1["kube-proxy"]
Runtime1["Container Runtime"]
Pod1["Pods"]
end
subgraph "Worker Node 2"
Kubelet2["kubelet"]
Proxy2["kube-proxy"]
Runtime2["Container Runtime"]
Pod2["Pods"]
end
API <--> Kubelet1
API <--> Kubelet2
API <--> ETCD
```
**Simple analogy:** the Control Plane is like a restaurant's head office — it decides the menu (desired state), tracks inventory (current state), and issues instructions. Worker nodes are the actual kitchens where food (containers) really gets cooked and served. The head office never cooks anything itself — it only ever tells kitchens what to do and watches what's actually happening.
---
## The API Server — The Front Door to Everything
The **API Server** (`kube-apiserver`) is the single, central entry point for absolutely everything in Kubernetes — every `kubectl` command, every internal component, every automated controller talks to Kubernetes exclusively through this one component.
```mermaid
graph TD
kubectl["kubectl (you)"] --> API["API Server"]
Controllers["Controllers"] --> API
Kubelet["kubelet (on every node)"] --> API
Scheduler["Scheduler"] --> API
API --> ETCD["etcd
(the only thing that
talks directly to etcd)"]
```
**A genuinely important architectural fact, worth stating explicitly: NOTHING in Kubernetes talks directly to `etcd` except the API Server.** Every other component — the scheduler, controllers, kubelets — only ever reads and writes cluster state by calling the API Server, which is the sole gatekeeper to the actual stored data. This single-entry-point design is what makes authentication, authorization (RBAC, from the DevSecOps series), and validation possible to enforce consistently across the entire cluster.
```bash
# Every single kubectl command is really just an HTTP request
# to the API server — you can see this directly:
kubectl get pods -v=8 2>&1 | grep "GET https"
# GET https:///api/v1/namespaces/default/pods
```
---
## The Admission Control Chain: Authentication, Authorization, Admission
Every request hitting the API Server passes through three genuinely distinct stages, in a strict order — a frequently-tested, precise sequence worth knowing by name, not just "there's some security checking."
```mermaid
flowchart LR
Req["Incoming request
(kubectl, a controller,
an external client)"] --> Authn["1. AUTHENTICATION:
WHO is making this
request? (client cert,
bearer token, OIDC)"]
Authn --> Authz["2. AUTHORIZATION:
Is this identity ALLOWED
to do this action on this
resource? (RBAC)"]
Authz --> Admission["3. ADMISSION CONTROL:
Should this SPECIFIC
request be allowed/
modified, given cluster
policy? (webhooks, quotas,
defaults)"]
Admission --> ETCD["Only NOW: written
to etcd"]
```
**Why the ordering matters, worth stating precisely:** authentication and authorization answer "who are you, and are you generally allowed to do this kind of thing" — coarse-grained, identity-based questions the DevSecOps series' RBAC material already covers in depth. **Admission control is a genuinely different, later stage**: by the time a request reaches it, the caller is already known and already authorized in principle — admission control instead asks fine-grained, *content*-based questions about this specific request ("does this Pod spec violate our Pod Security Standard," "does this Deployment omit required labels," "does this Namespace already have too many Pods for its ResourceQuota").
**Built-in admission controllers worth knowing by name**, since "admission control" is often assumed to mean only custom webhooks, when in fact most clusters rely heavily on built-in ones:
| Admission Controller | What It Does |
|---|---|
| `NamespaceLifecycle` | Rejects creating objects in a Namespace that's being deleted |
| `LimitRanger` | Applies default resource requests/limits (Part 2) when a Pod spec omits them |
| `ResourceQuota` | Rejects requests that would exceed a Namespace's configured quota |
| `PodSecurity` | Enforces Pod Security Standards (privileged/baseline/restricted) — the modern replacement for the deprecated PodSecurityPolicy |
| `DefaultStorageClass` | Assigns the cluster's default StorageClass to a PVC that doesn't specify one (Part 3) |
| `MutatingAdmissionWebhook` | Runs any registered custom mutating webhooks (next section) |
| `ValidatingAdmissionWebhook` | Runs any registered custom validating webhooks (next section) |
---
## Mutating and Validating Admission Webhooks, In Depth
Beyond the built-in admission controllers, Kubernetes lets you register your **own** admission logic as an HTTP webhook — this is the exact mechanism service meshes (Part 4), policy engines like OPA Gatekeeper or Kyverno, and many operators use to enforce custom rules or automatically inject configuration.
```mermaid
sequenceDiagram
participant API as API Server
participant Mut as Mutating Webhook
(e.g. sidecar injector)
participant Val as Validating Webhook
(e.g. policy engine)
participant ETCD as etcd
API->>API: Authn + Authz pass
API->>Mut: 1. Runs ALL mutating
webhooks FIRST - each can
MODIFY the object
Mut-->>API: Modified object
(e.g. sidecar container
injected)
API->>Val: 2. THEN runs ALL validating
webhooks - each can only
ALLOW or REJECT, never modify
Val-->>API: Allow or deny
API->>ETCD: Only if allowed:
final object persisted
```
**The mutating-before-validating ordering is deliberate and worth stating explicitly:** if validation ran first, a webhook meant to reject non-compliant Pods might reject a Pod that a *later* mutating webhook would have fixed automatically (e.g. injecting a missing required label). Running all mutations first means validation always evaluates the *final*, fully-mutated object — the actual object that will actually be persisted and run.
```yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: require-resource-limits
webhooks:
- name: require-resource-limits.example.com
clientConfig:
service:
name: policy-webhook
namespace: platform
path: "/validate"
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
failurePolicy: Fail
admissionReviewVersions: ["v1"]
```
**`failurePolicy` is a genuinely important, easy-to-get-wrong field worth knowing precisely:** `Fail` means if the webhook itself is unreachable (network issue, the webhook's own pod is down), the API Server **rejects the request** — safe by default, but means a broken webhook can block ALL matching operations cluster-wide, including emergency changes. `Ignore` means an unreachable webhook is silently skipped, letting the request through unvalidated — safer for availability, but means policy enforcement has a real, silent failure mode. **A strong interview answer names this as a genuine tradeoff, not a "just pick Fail" default**: critical security policy webhooks often justify `Fail` (worth the availability risk), while a nice-to-have labeling/defaulting webhook is often better as `Ignore` (don't let it become a single point of cluster-wide failure).
---
## kube-apiserver Flags That Matter Operationally
Most `kube-apiserver` configuration is invisible day-to-day on managed Kubernetes (EKS/AKS/GKE own it entirely, per Part 5), but understanding the flags that matter is genuinely useful both for self-managed clusters (Part 6) and for reasoning about *why* a managed cluster behaves the way it does.
```bash
kube-apiserver \
--etcd-servers=https://127.0.0.1:2379 \
--service-cluster-ip-range=10.96.0.0/12 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ResourceQuota,PodSecurity \
--audit-log-path=/var/log/kubernetes/audit.log \
--audit-log-maxage=30 \
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \
--authorization-mode=Node,RBAC \
--max-requests-inflight=400 \
--max-mutating-requests-inflight=200
```
| Flag | Why it matters operationally |
|---|---|
| `--service-cluster-ip-range` | The CIDR block Services get their virtual IPs from (Part 3) — sized wrong at cluster creation, and you can run out of Service IPs with no easy fix later |
| `--audit-log-path` / `--audit-policy-file` | Enables the API Server's own audit log — the exact mechanism EKS's control-plane audit logging (Part 7) surfaces through CloudWatch; a critical forensic signal during a security incident, off by default |
| `--authorization-mode=Node,RBAC` | `Node` authorizes kubelets to only access objects related to their own node (a real security boundary); `RBAC` layers standard role-based access on top |
| `--max-requests-inflight` / `--max-mutating-requests-inflight` | Caps concurrent API requests to protect the API Server itself from being overwhelmed — a genuinely real production concern at high request volume, directly connecting to the rate-limiting patterns in the Capacity Planning & Performance series |
| `--tls-min-version` | Enforces a minimum TLS version for all client connections — a genuinely common compliance/hardening requirement (DevSecOps series) |
| `--anonymous-auth` | Whether unauthenticated requests are allowed at all — should be `false` on any production cluster, self-managed or otherwise |
**Why `--authorization-mode=Node` specifically matters, a concrete security point:** without it, a compromised kubelet credential (stolen from one specific node) could, under RBAC alone, potentially be scoped broadly enough to read secrets or pod specs belonging to *other* nodes — the Node authorizer adds an additional, node-scoped restriction on top of RBAC specifically for kubelet identities, limiting a single compromised node's blast radius to its own workloads.
---
## etcd — Kubernetes's Memory
**etcd** is a distributed, consistent key-value store — and it is, quite literally, the entire source of truth for a Kubernetes cluster's state. Every object (every Pod, Deployment, Service, Secret) is stored here.
```mermaid
graph LR
ETCD["etcd"] --> K1["/registry/pods/default/checkout-abc123"]
ETCD --> K2["/registry/deployments/default/checkout"]
ETCD --> K3["/registry/services/default/checkout-svc"]
```
**Why etcd is a genuinely critical, high-stakes component — worth stressing explicitly in an interview:** if etcd is lost or corrupted with no backup, the cluster effectively has **amnesia** — it has no memory of what should be running, where, or how it was configured. This directly connects to the Disaster Recovery topic (topic 11) in this course: **etcd backups are one of the single most critical, non-negotiable disaster-recovery practices for any self-managed Kubernetes cluster.**
```bash
# Take a backup of etcd (run on a control plane node)
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
# Verify the snapshot
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot.db --write-out=table
```
**Why etcd specifically needs strong consistency (CP, not AP, using the CAP theorem vocabulary from the Reliability & Architecture Patterns series):** if two control plane replicas ever disagreed about whether a Pod exists, the cluster's behavior would become genuinely unpredictable. This is exactly why etcd uses the **Raft consensus algorithm**, requiring a majority (quorum) of its members to agree before any write is considered committed — directly the same quorum principle (`W + R > N`) covered in that earlier tutorial.
```mermaid
graph TD
ETCD3["etcd cluster:
typically 3 or 5 nodes
(always an ODD number)"] --> Quorum["Requires a MAJORITY
to agree before any
write commits"]
Quorum --> Why["Why odd numbers?
3 nodes: tolerates 1
failure (need 2 of 3)
5 nodes: tolerates 2
failures (need 3 of 5)
An EVEN number wastes
a node without improving
fault tolerance"]
```
**Why etcd clusters always use an odd number of nodes, a genuinely common, sharp interview question:** a 4-node cluster still only tolerates 1 failure (needs 3 of 4 to agree — same as a 3-node cluster needing 2 of 3), but costs an entire extra node for zero additional fault tolerance. An odd number is always the efficient choice.
---
## etcd's Raft Consensus, In Depth
The quorum requirement above is a *consequence* of etcd's underlying consensus algorithm, **Raft** — worth understanding at the mechanism level, not just "it needs a majority."
```mermaid
sequenceDiagram
participant Leader as etcd Leader
participant F1 as Follower 1
participant F2 as Follower 2
Leader->>Leader: Receives a write
(from the API Server)
Leader->>F1: Replicate log entry
Leader->>F2: Replicate log entry
F1-->>Leader: Acknowledge
F2-->>Leader: Acknowledge
Leader->>Leader: Majority (2 of 3,
including itself)
acknowledged - entry
is now COMMITTED
Leader->>F1: Notify: commit index
advanced
Leader->>F2: Notify: commit index
advanced
```
**The core Raft roles, worth naming precisely:** at any moment, exactly one etcd member is the **Leader** (the only one that accepts writes), and the rest are **Followers** (passively replicating the Leader's log). If the Leader fails or becomes unreachable, remaining members hold a **leader election** — each Follower waits a randomized timeout, and the first to time out becomes a **Candidate**, requesting votes from the others; whichever Candidate gets a majority of votes becomes the new Leader.
```mermaid
graph TD
Normal["Normal operation:
1 Leader, N-1 Followers,
Leader handles ALL writes"] --> Fail["Leader fails / network
partition isolates it"]
Fail --> Timeout["Followers' randomized
election timeouts expire -
the FIRST one to time out
becomes a Candidate"]
Timeout --> Vote["Candidate requests votes
from remaining members"]
Vote --> NewLeader{"Gets a MAJORITY
of votes?"}
NewLeader -->|Yes| Elected["Becomes the new Leader -
cluster resumes accepting
writes"]
NewLeader -->|No - split vote| Retry["Election times out,
a NEW round starts with
a new randomized timeout"]
```
**Why the randomized election timeout matters, a genuinely subtle but real design detail worth knowing:** if every Follower used the *same* fixed timeout, they'd all become Candidates simultaneously after a Leader failure, splitting the vote repeatedly with no majority winner — the randomization means, in practice, one Follower almost always times out meaningfully before the others, giving it a clean shot at winning the election before a second Candidate even enters the race.
**The direct, practical consequence for cluster operators, worth stating explicitly:** during a Leader election (which is typically sub-second, but real), **etcd cannot accept new writes** — meaning the Kubernetes API Server cannot write any new/updated cluster state during that brief window. Reads of already-committed data can often still be served, but write-heavy operations (creating pods, scaling deployments) will briefly queue or fail. This is exactly why etcd's own recommended deployment topology emphasizes low-latency, stable networking between members — frequent, unnecessary leader elections caused by network flakiness directly degrade the entire cluster's ability to accept changes, not just etcd itself.
---
## etcd Compaction, Defragmentation, and Alarms
Two genuinely real, operationally-important etcd maintenance concepts, easy to overlook until they cause a production incident.
```mermaid
graph TD
Writes["Every write to etcd
creates a NEW revision -
etcd keeps ALL historical
revisions by default"] --> Growth["Without maintenance, the
underlying data file
grows UNBOUNDED over time"]
Growth --> Compact["COMPACTION: explicitly
discards old revisions
before a given revision
number, freeing LOGICAL
space"]
Growth --> Defrag["DEFRAGMENTATION: reclaims
the underlying storage
engine's PHYSICAL disk
space after compaction -
a SEPARATE step"]
```
**Why compaction and defragmentation are two distinct steps, a real, commonly-missed operational detail:** compaction tells etcd's internal MVCC (multi-version concurrency control) store "these old revisions are no longer needed" — but etcd's underlying storage engine (`bbolt`, a B+tree-based store) doesn't automatically shrink the actual file on disk just because logical space was freed; it just marks that space as reusable for *future* writes. **Defragmentation is the separate operation that actually returns freed space to the filesystem**, reducing the on-disk file size. Skipping defragmentation on a long-running, write-heavy cluster is a genuinely common cause of etcd's data file silently growing until it hits its default storage quota (commonly 2GB) — at which point etcd stops accepting writes entirely.
```bash
# Compact etcd history up to a specific revision
ETCDCTL_API=3 etcdctl compact $(etcdctl endpoint status --write-out="json" | grep -o '"revision":[0-9]*' | grep -o '[0-9]*')
# Defragment (run per-member, one at a time, never all simultaneously —
# defrag briefly blocks that member, and doing all at once risks a
# temporary full quorum outage)
ETCDCTL_API=3 etcdctl defrag --endpoints=https://127.0.0.1:2379
# Check for active alarms (e.g. NOSPACE — quota exceeded)
ETCDCTL_API=3 etcdctl alarm list
```
**The `NOSPACE` alarm, worth knowing as a real, concrete failure mode:** when etcd's storage quota is exceeded, it raises a `NOSPACE` alarm and **rejects all writes cluster-wide** until the alarm is explicitly disarmed — meaning `kubectl apply` for anything, cluster-wide, starts failing. Recovery requires compacting history, defragmenting to actually reclaim disk space, and then explicitly clearing the alarm (`etcdctl alarm disarm`) — a genuinely realistic, high-severity incident scenario for any team running self-managed Kubernetes without regular etcd maintenance automated.
---
## The Controller Manager and the Reconciliation Loop
This is arguably **the single most important conceptual idea in all of Kubernetes** — genuinely worth spending real time to understand deeply, since almost everything else in the system is built on top of this one pattern.
```mermaid
flowchart TD
A["1. OBSERVE: what's the
CURRENT actual state?
(e.g. 2 pods running)"] --> B["2. COMPARE: what's the
DESIRED state?
(e.g. Deployment says
replicas: 3)"]
B --> C{"Do they match?"}
C -->|Yes| A
C -->|No| D["3. ACT: take action to
close the gap
(e.g. create 1 more pod)"]
D --> A
```
**This loop — observe, compare, act, repeat forever — is called a controller, and it's the fundamental unit of automation in Kubernetes.** A **Controller Manager** process runs dozens of these loops simultaneously, each one responsible for one specific type of object (a Deployment Controller, a ReplicaSet Controller, a Node Controller, and many more).
**Simple analogy:** think of a home thermostat. It doesn't "turn on the heat once" — it continuously checks the current temperature against your desired setting, and takes action (heat on/off) whenever there's a gap, forever, without you doing anything. Every Kubernetes controller works exactly this way, just applied to cluster objects instead of temperature.
```mermaid
graph TD
Deploy["Deployment says:
'I want 3 replicas of
checkout-service'"] --> Controller["Deployment Controller
watches this continuously"]
Controller --> Check{"Are there ACTUALLY
3 running?"}
Check -->|"Only 2 exist
(one crashed)"| Create["Create 1 more Pod"]
Check -->|"4 exist (a bug,
or manual scale-up
outside the Deployment)"| Delete["Delete 1 Pod"]
Check -->|"Exactly 3 exist"| DoNothing["Do nothing —
already matches"]
```
**The direct, practical payoff of this design, worth stating explicitly:** this is exactly *why* Kubernetes self-heals. If a node dies and takes a pod with it, you don't need any human or script to notice and react — the relevant controller notices the gap between desired (3 replicas) and actual (2 replicas) on its very next reconciliation pass (which happens continuously, many times a second) and simply creates a replacement, automatically, with zero human involvement.
---
## Leader Election — How Control Plane Components Avoid Split-Brain
Production clusters run the Controller Manager and Scheduler as **multiple replicas** for high availability (directly the same "don't run a single point of failure" principle from the Reliability & Architecture Patterns series) — but only ONE replica of each should actually be *active* at a time, or you'd get duplicate, conflicting reconciliation actions. This is solved with the exact same **leader election** pattern etcd itself uses internally (previous section), applied one layer up.
```mermaid
graph TD
CM1["Controller Manager
replica 1"] --> Lock["Lease object in etcd
(via the API Server) -
'controller-manager-lock'"]
CM2["Controller Manager
replica 2"] --> Lock
CM3["Controller Manager
replica 3"] --> Lock
Lock --> Winner["Whichever replica
successfully acquires and
RENEWS the lease becomes
the ACTIVE leader"]
Winner --> Standby["The other replicas sit
idle in STANDBY, doing
nothing, continuously
watching for the lease to
become available"]
```
**The mechanism, concretely:** each replica repeatedly attempts to create or update a `Lease` object (a standard Kubernetes API object, stored in etcd like everything else) with its own identity and a short expiry, using an atomic compare-and-swap operation. Whichever replica succeeds becomes the leader and must continuously **renew** that lease before it expires — if the leader crashes or is partitioned away, it stops renewing, the lease expires, and one of the standby replicas acquires it and takes over.
```bash
# See which replica currently holds leadership for a given component
kubectl get lease -n kube-system kube-controller-manager -o yaml
# holderIdentity:
```
**Why this matters concretely for cluster operators, a genuinely important operational fact:** during the brief window between a leader crashing and a standby acquiring the lease (bounded by the lease's expiry duration, commonly a handful of seconds), **no reconciliation happens at all** for whatever component just lost leadership — new Deployments won't get their ReplicaSets created, failed pods won't be replaced, until a new leader takes over. This is a real, bounded gap, not an instantaneous failover, and is exactly why the lease duration is a deliberate tradeoff: too short risks unnecessary leadership churn during brief network blips; too long extends the reconciliation gap during a genuine failure.
---
## The Scheduler, at a High Level
The **Scheduler** (`kube-scheduler`) has one specific job: when a new Pod is created with no node assigned yet, decide **which node it should actually run on.** (The full mechanics — filtering, scoring, affinity rules — get their own deep dive in Part 2.)
```mermaid
sequenceDiagram
participant API as API Server
participant Sched as Scheduler
participant Node as Chosen Node's Kubelet
API->>Sched: New Pod exists, no
node assigned yet
Sched->>Sched: Evaluate ALL nodes -
which ones COULD run this pod,
and which is the BEST choice?
Sched->>API: "Assign this pod to Node 2"
API->>Node: Node 2's kubelet sees
a pod assigned to it
Node->>Node: Actually starts the
container(s)
```
**A genuinely important point worth stating explicitly: the Scheduler only ever DECIDES and RECORDS which node a pod should run on — it never actually starts a container itself.** That job belongs entirely to the kubelet on the chosen node, covered next.
---
## Worker Node Components
Every worker node runs three essential components, each with one specific job.
```mermaid
graph TD
Node["Worker Node"] --> Kubelet["kubelet: the node's local
AGENT - talks to the API
server, actually manages
containers on THIS node"]
Node --> Proxy["kube-proxy: implements
Kubernetes SERVICE
networking on THIS node"]
Node --> Runtime["Container Runtime
(containerd, CRI-O): actually
runs the containers
themselves"]
```
---
## The Kubelet — The Node's Local Agent
The **kubelet** is the only Kubernetes component running on a worker node that talks directly to the API Server. It's responsible for making sure the containers assigned to its node are actually running, healthy, and match their spec.
```mermaid
flowchart TD
A["kubelet continuously WATCHES
the API server for pods
assigned to ITS node"] --> B["For each assigned pod,
tells the container runtime
to start the containers"]
B --> C["Continuously runs health
checks (liveness/readiness
probes) against those
containers"]
C --> D["Reports status back to
the API server
(is it Running? Ready?
Has it crashed?)"]
```
**Liveness vs. readiness probes — a genuinely common, specific interview distinction:**
```mermaid
graph TD
Live["Liveness probe:
'Is this container ALIVE,
or should it be RESTARTED?'"] --> LiveAction["Fails -> kubelet KILLS
and restarts the container"]
Ready["Readiness probe:
'Is this container READY
to receive TRAFFIC right now?'"] --> ReadyAction["Fails -> Pod is REMOVED
from Service endpoints
(traffic stops routing to
it) but the container
KEEPS running, untouched"]
```
**Why this distinction matters practically, and it's a classic interview trap:** a container that's temporarily overwhelmed and slow (but not actually broken) should fail its **readiness** probe (stop receiving new traffic until it catches up) but should absolutely NOT fail its **liveness** probe — killing and restarting a container that's just temporarily busy makes the problem *worse*, not better, by throwing away whatever progress it had made and adding restart overhead on top of an already-struggling situation.
```yaml
apiVersion: v1
kind: Pod
metadata:
name: checkout
spec:
containers:
- name: app
image: checkout:1.2.3
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
failureThreshold: 2
```
---
## Node Heartbeats and Node Conditions
Beyond managing individual pods, the kubelet has a second, continuous job worth understanding: reporting the health of the **node itself** back to the control plane, on a regular heartbeat.
```mermaid
sequenceDiagram
participant Kubelet
participant API as API Server
participant NC as Node Controller
loop Every ~10 seconds (default)
Kubelet->>API: Update Node status -
conditions, capacity,
allocatable resources
end
NC->>API: Watches Node objects
NC->>NC: If a Node hasn't reported
in ~40s (default) - mark
condition Ready=Unknown
NC->>NC: If still no report after
a further grace period
(default 5 minutes) -
start evicting that node's
pods
```
**Node Conditions worth knowing by name**, since "the node is unhealthy" is actually several distinct, separately-tracked signals:
| Condition | Meaning |
|---|---|
| `Ready` | The node's kubelet is healthy and able to accept new pods |
| `MemoryPressure` | Available memory is low enough that the kubelet may start evicting pods |
| `DiskPressure` | Available disk space is low enough that the kubelet may start evicting pods |
| `PIDPressure` | Too many processes running, approaching the OS's process ID limit |
| `NetworkUnavailable` | The node's network hasn't been correctly configured (commonly a transient state right after a node joins) |
**Why the two-stage timeout (Unknown, then eviction after a grace period) matters, a genuinely important reliability nuance:** a node briefly missing its heartbeat due to a short network blip shouldn't immediately trigger evicting and rescheduling every pod on it — that would be wasteful, disruptive churn for what might resolve itself in seconds. The Node Controller's default behavior deliberately waits through a longer grace period (`pod-eviction-timeout`, default 5 minutes) before concluding the node is genuinely gone and starting pod eviction — directly the same "don't overreact to a transient blip" principle behind readiness-probe failure thresholds earlier in this Part, applied at the node level instead of the container level.
```bash
# See a node's current conditions directly
kubectl describe node worker-3 | grep -A 10 Conditions
```
---
## kube-proxy — Making Services Actually Work
**kube-proxy** runs on every node and is responsible for implementing the actual networking rules that make a Kubernetes **Service** (a stable, virtual IP that load-balances across a changing set of pods) actually work.
```mermaid
graph TD
Service["Service: checkout-svc
Virtual IP: 10.96.0.5"] --> Proxy["kube-proxy programs
networking rules (iptables
or IPVS) on EVERY node"]
Proxy --> Rule["Rule: 'traffic to
10.96.0.5 -> randomly pick
one of the currently
healthy pod IPs backing
this Service'"]
```
**In plain terms:** without kube-proxy, a Service's stable virtual IP would just be an abstract idea with nothing actually making it work — kube-proxy is the component that turns "traffic to this Service" into "actual network rules routing to real, currently-healthy pod IPs," updated automatically every time pods come and go. (The full mechanics of Services get their own deep dive in Part 3.)
---
## The Container Runtime
The actual layer that runs containers — the lowest-level component in this whole stack, sitting directly on top of the Linux kernel primitives (namespaces and cgroups) covered in the Linux & Networking Fundamentals series.
```mermaid
graph TD
Kubelet["kubelet"] -->|"CRI (Container Runtime
Interface) - a STANDARD
API"| Runtime["Container Runtime
(containerd, CRI-O)"]
Runtime --> Kernel["Linux kernel: namespaces
+ cgroups
(Linux & Networking
Fundamentals series, Part 1)"]
```
**Why the CRI (Container Runtime Interface) matters, worth knowing by name:** Kubernetes doesn't hardcode a dependency on any one specific container runtime — it talks to whatever runtime is installed through this standard interface. This is exactly why Kubernetes could cleanly deprecate direct Docker support (a well-known, sometimes misunderstood industry event) without actually breaking anything for end users — Docker-built images still work fine, because the image *format* (OCI-compliant) is separate from the runtime that executes containers, and any CRI-compliant runtime (like `containerd`, which Docker itself is built on top of) can run them.
---
## API Server High Availability
A single `kube-apiserver` instance would be an obvious single point of failure — real clusters (and every managed offering from Part 5) run **multiple API server replicas**, fronted by a load balancer.
```mermaid
graph TD
Client["kubectl / any client"] --> LB["Load Balancer
(the cluster's actual
API endpoint)"]
LB --> API1["API Server replica 1"]
LB --> API2["API Server replica 2"]
LB --> API3["API Server replica 3"]
API1 --> ETCD["etcd cluster"]
API2 --> ETCD
API3 --> ETCD
```
**A genuinely important architectural point worth stating precisely: unlike the Controller Manager and Scheduler, API Server replicas do NOT use leader election — every replica is simultaneously active and can independently serve any request.** This is possible specifically because the API Server itself is largely **stateless** — it doesn't hold cluster state in memory as its own source of truth; it reads and writes through to etcd for everything. Any replica can serve any read or write, and etcd's own consensus (previous sections) is what actually guarantees consistency, not coordination between the API Server replicas themselves.
**Why this distinction between "stateless, all-active" (API Server) and "stateful reconciliation, single-active-via-leader-election" (Controller Manager, Scheduler) is a strong, precise interview answer:** it demonstrates understanding that HA isn't one uniform pattern applied identically to every control plane component — the right HA mechanism depends on whether the component actually needs single-writer semantics (reconciliation loops absolutely do, to avoid duplicate/conflicting actions) or can safely be handled by any replica independently (the API Server, because etcd is the actual arbiter of consistency).
```bash
# On a self-managed cluster, you can see multiple API server endpoints
# directly in kubeconfig if configured behind a load balancer:
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
```
**On managed Kubernetes specifically (Part 5, Part 7):** this entire API server HA topology — how many replicas, how they're load-balanced, how they scale under load — is exactly the part of "the control plane" that EKS/AKS/GKE fully own and abstract away. You interact with a single, stable API endpoint URL and never need to reason about the replica count or load-balancing mechanism behind it, which is precisely the "managed control plane" value proposition named at the start of Part 5.
**The watch cache — why most reads don't actually hit etcd, a genuinely important performance detail:** every API server replica maintains an in-memory **watch cache** of recently-seen objects, kept continuously up to date by watching etcd for changes. The vast majority of `kubectl get`/`list`/`watch` requests (the overwhelming majority of real API traffic, since controllers and kubelets are constantly watching for changes) are served directly from this in-memory cache, not by querying etcd on every single read. This is exactly why etcd — despite being "the entire source of truth" — doesn't become a read bottleneck even in large, high-churn clusters: **etcd primarily absorbs writes and the watch cache absorbs reads**, a division of labor worth stating explicitly when asked how Kubernetes scales its own control plane.
---
## Extending the API: Aggregated API Servers and CRDs at the Wire Level
Kubernetes's API surface isn't fixed — it's genuinely extensible, and understanding the mechanism (not just "CRDs exist," which gets full operator-pattern treatment in Part 4) belongs here as an architectural fact about the API Server itself.
```mermaid
graph TD
Client["kubectl get pods"] --> API["kube-apiserver"]
API --> Native["Native resource
(Pod, Service...) -
handled DIRECTLY by
kube-apiserver itself"]
Client2["kubectl get
customresource"] --> API2["kube-apiserver"]
API2 --> CRDPath{"Is this a CRD, or an
aggregated API?"}
CRDPath -->|"CRD (Custom
Resource Definition)"| CRDStore["Stored directly in
etcd, same as native
resources - kube-apiserver
handles it natively once
the CRD is registered"]
CRDPath -->|"Aggregated API
(APIService object)"| Aggregate["Request PROXIED to a
SEPARATE, independent
API server (e.g.
metrics-server) - NOT
stored in the main
cluster's etcd at all"]
```
**Why these are two genuinely different extension mechanisms, worth distinguishing precisely — a common point of confusion even among experienced practitioners:**
| | Custom Resource Definitions (CRDs) | Aggregated API Servers |
|---|---|---|
| Where data is stored | The main cluster's own etcd, alongside native resources | Wherever the aggregated API server chooses — often NOT etcd at all |
| Who serves the request | `kube-apiserver` itself, natively, once the CRD is registered | A completely separate API server process, `kube-apiserver` just proxies to it |
| Typical use case | Defining new, durable, declarative resource types (the foundation of the Operator pattern, Part 4) | Serving data that doesn't fit the "durable declarative object" model — `metrics-server`'s live CPU/memory metrics are the canonical example, since they're transient, not something you'd want to accumulate forever in etcd |
| Registration object | `CustomResourceDefinition` | `APIService` |
```bash
# See registered aggregated API services on a cluster
kubectl get apiservices | grep -v "^NAME\|Local"
# v1beta1.metrics.k8s.io kube-system/metrics-server True
```
**Why `metrics-server` specifically is the canonical aggregated-API example, worth explaining precisely:** live CPU/memory usage changes constantly and has no lasting value as durable cluster state — storing a continuous stream of point-in-time metrics in etcd (a system deliberately optimized for consistency and durability of relatively low-write-volume configuration data, not high-frequency time-series data) would be a genuine architectural mismatch. `metrics-server` instead runs as its own independent process, computes metrics from kubelet's own resource-usage reporting, and is only *reachable* through the main API server's proxy — `kubectl top pods` and the Horizontal Pod Autoscaler (Part 2) both query it through this exact aggregation mechanism, without metrics-server data ever touching the cluster's own etcd.
---
## Control Plane Component Ports and Communication, Reference
A concrete, worth-having reference table — genuinely useful both for understanding the architecture diagrams throughout this Part and for real troubleshooting (firewall rules, security group configuration on self-managed clusters per Part 6).
| Component | Default Port | Protocol | Who Talks to It |
|---|---|---|---|
| `kube-apiserver` | 6443 | HTTPS | Everyone — kubectl, kubelets, controllers, scheduler |
| `etcd` (client) | 2379 | HTTPS | `kube-apiserver` only |
| `etcd` (peer) | 2380 | HTTPS | Other etcd members (Raft replication) |
| `kubelet` (API) | 10250 | HTTPS | `kube-apiserver` (exec, logs, port-forward) |
| `kube-scheduler` (metrics) | 10259 | HTTPS | Monitoring/metrics scrapers |
| `kube-controller-manager` (metrics) | 10257 | HTTPS | Monitoring/metrics scrapers |
| `kube-proxy` (metrics) | 10249 | HTTP | Monitoring/metrics scrapers |
```mermaid
graph TD
subgraph "Control Plane Network Boundary"
API["kube-apiserver :6443"]
ETCDC["etcd :2379 (client)"]
ETCDP["etcd :2380 (peer)"]
Sched["kube-scheduler :10259"]
CM["kube-controller-manager
:10257"]
end
subgraph "Worker Node Network Boundary"
Kubelet["kubelet :10250"]
Proxy["kube-proxy :10249"]
end
API -->|"Only component
allowed to reach etcd"| ETCDC
ETCDC <-->|"Raft replication
between members"| ETCDP
API -->|"exec/logs/port-forward"| Kubelet
Sched -->|"watches/writes via API"| API
CM -->|"watches/writes via API"| API
Kubelet -->|"reports status via API"| API
```
**Why knowing port 10250 specifically matters as a real security fact, worth stating explicitly:** the kubelet API is a genuinely powerful surface — it's what lets the API Server execute commands inside a running container (`kubectl exec`) or stream logs. An improperly secured kubelet API (missing authentication/authorization, a real misconfiguration seen in the wild) is a well-known, serious attack vector, directly connecting to the container and Kubernetes security material in the DevSecOps series (`05-devsecops/03-container-and-kubernetes-security.md`) — always verify kubelet authentication is enforced (`--anonymous-auth=false`, `--authorization-mode=Webhook`), never left at insecure defaults.
---
## A Full Worked Journey: `kubectl apply` to a Running Pod
Tying every component in this tutorial together into one complete, step-by-step story — genuinely one of the most valuable things to be able to narrate fluently in an interview.
```mermaid
sequenceDiagram
participant You as You (kubectl)
participant API as API Server
participant ETCD as etcd
participant DC as Deployment Controller
participant Sched as Scheduler
participant Kubelet as kubelet (chosen node)
participant Runtime as Container Runtime
You->>API: kubectl apply -f deployment.yaml
(desired: 3 replicas)
API->>API: Authenticate + authorize
(RBAC) + validate
API->>ETCD: Store the Deployment object
DC->>API: Watching for Deployments...
sees the new one
DC->>API: Creates 3 Pod objects
(no node assigned yet)
API->>ETCD: Stores the 3 Pod objects
Sched->>API: Watching for unscheduled
Pods... sees 3 of them
Sched->>Sched: Decides best node
for each Pod
Sched->>API: Updates each Pod with
its assigned node
Kubelet->>API: Watching for Pods
assigned to ITS node...
sees one
Kubelet->>Runtime: "Start this container"
Runtime->>Runtime: Pulls image, creates
namespaces + cgroups,
starts the process
Kubelet->>API: Reports: Pod is
Running and Ready
```
**A strong interview answer walks through this exact sequence, naming every component and its specific, narrow responsibility — this single narrative demonstrates the entire architecture in one coherent story, rather than a list of disconnected component definitions.**
---
## Kubernetes API Versioning and Deprecation Policy
A final architectural fact worth knowing precisely, since it directly affects how you plan cluster upgrades (Part 5, Part 7) and write manifests that don't silently break.
```mermaid
graph TD
Alpha["alpha (e.g. v1alpha1):
may contain bugs, may be
REMOVED at any time
without notice, disabled
by default"] --> Beta["beta (e.g. v1beta1):
well-tested, enabled by
default, but the API
SHAPE may still change
in incompatible ways"]
Beta --> Stable["stable (e.g. v1):
appears in released
software for MANY
versions, a formal
deprecation policy applies
before removal"]
```
**The formal deprecation policy for stable (GA) APIs, worth knowing as a concrete, citable rule:** once an API reaches `v1` (stable), Kubernetes's own deprecation policy guarantees it remains supported for a minimum period measured in **API minor version releases**, not a vague "eventually" — giving operators genuine, predictable planning time before a migration is required. Beta APIs get a shorter, but still formal, guaranteed support window; alpha APIs have no such guarantee at all.
```bash
# Check a cluster for any deprecated API usage before upgrading —
# genuinely essential pre-upgrade due diligence
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
# Or use the community-standard tool built specifically for this
pluto detect-helm -o wide
```
**Why this matters concretely, a real, common upgrade-incident pattern worth naming:** a team that upgrades a cluster's Kubernetes minor version without first checking for deprecated API usage can find a previously-working manifest suddenly rejected outright post-upgrade, because the API version it referenced (e.g. an old `extensions/v1beta1` Ingress, removed in Kubernetes 1.22) simply no longer exists on the new control plane. **This is precisely why "check for deprecated API usage" is a mandatory, non-skippable step in any real Kubernetes upgrade runbook** — directly connecting to the node-upgrade and control-plane-upgrade discussion in Part 5 and Part 7: the upgrade isn't just about node compatibility, it's equally about API compatibility for every manifest, Helm chart, and controller running in the cluster.
---
## The API Server's Own Health Endpoints
The pod-level liveness/readiness distinction covered earlier in this Part applies to the control plane's own components too — the API Server itself exposes several distinct health endpoints, worth knowing apart from application-level probes.
```mermaid
graph TD
API["kube-apiserver"] --> Healthz["/healthz - overall health
(legacy, still widely used)"]
API --> Livez["/livez - IS the process
alive? (newer, more
precise than /healthz)"]
API --> Readyz["/readyz - is it READY to
serve requests RIGHT NOW?
(e.g. NOT during initial
startup before informers
have synced)"]
```
```bash
# Query the API server's own readiness, with per-check detail
kubectl get --raw='/readyz?verbose'
# [+]ping ok
# [+]log ok
# [+]etcd ok
# [+]poststarthook/start-kube-apiserver-admission-initializer ok
# readyz check passed
```
**Why `/readyz?verbose` is worth knowing specifically as a real troubleshooting tool:** when a self-managed API server is behaving strangely right after startup or an upgrade, this endpoint breaks down readiness into its individual constituent checks — including, notably, an explicit `etcd` check, meaning a genuinely fast way to confirm "is the API server's own etcd connectivity actually healthy right now" without needing to separately query etcd directly.
---
## Declarative vs Imperative — The Core Philosophy
A final, foundational concept worth stating explicitly, since it explains *why* Kubernetes is designed the way it is.
```mermaid
graph TD
Imperative["IMPERATIVE: 'run these
exact COMMANDS, in this
exact ORDER, to get to
the state I want'"] --> ImpProb["❌ Brittle - if a step
fails partway, or the
starting state was
different than assumed,
the result is unpredictable"]
Declarative["DECLARATIVE (Kubernetes'
approach): 'here's the STATE
I want — YOU figure out
how to get there, and KEEP
me there, continuously'"] --> DecGood["✅ Self-healing, idempotent
- you can apply the SAME
manifest a thousand times
safely, and it converges to
the exact same result
through the reconciliation
loop, regardless of the
starting state"]
```
**A clean, memorable interview line:** "Kubernetes is fundamentally declarative — you describe what you want, not the steps to get there, and the reconciliation loop pattern is the engine that continuously, automatically closes the gap between what you asked for and what's actually running, which is exactly what makes the whole system self-healing without any human in the loop."
---
## Common Mistakes
| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Assuming any component besides the API Server talks directly to etcd | Breaks the whole security/consistency model — the API Server is the sole gatekeeper | Understand the API Server as the single, mandatory front door to all cluster state |
| Running an even-numbered etcd cluster (e.g. 4 nodes) | Wastes a node for zero additional fault tolerance compared to an odd number | Always use an odd number of etcd members (commonly 3 or 5) |
| Configuring only a liveness probe, with no readiness probe (or vice versa) | Conflates "is this container broken" with "is this container ready for traffic right now" — very different questions with very different correct responses | Configure both, deliberately, with different criteria appropriate to each |
| Treating a temporarily slow/busy container's liveness probe as a signal to restart it | Restarting a container that's just busy, not broken, discards progress and adds restart overhead on top of an already-struggling situation | Let readiness probes handle "temporarily not ready for traffic"; reserve liveness failures for genuinely broken/deadlocked containers |
| Believing the Scheduler actually starts containers | Confuses the Scheduler's role (deciding WHERE) with the kubelet's role (actually running it THERE) | Know the precise, narrow responsibility of each component |
| No etcd backup strategy for a self-managed cluster | A lost/corrupted etcd means the cluster has no memory of its own desired state at all | Treat etcd snapshots as a non-negotiable, regularly-tested backup practice (full depth in the Disaster Recovery topic) |
| Setting a critical policy webhook's `failurePolicy` to `Ignore` | A webhook that's down silently stops enforcing policy instead of blocking the request — security-critical policies can be bypassed simply by the webhook being briefly unreachable | Use `Fail` for genuinely critical policy webhooks, and ensure the webhook itself is highly available; reserve `Ignore` for non-critical, nice-to-have webhooks |
| Never running etcd defragmentation on a long-running, write-heavy cluster | Compaction frees logical space but doesn't shrink the on-disk file — the data file can silently grow until it hits the storage quota and etcd stops accepting ALL writes | Schedule regular, one-member-at-a-time defragmentation as part of routine cluster maintenance |
| Assuming control plane leader election means zero reconciliation gap during a failover | There's a real, bounded gap between a leader crashing and a standby acquiring the lease — nothing reconciles for that component during that window | Understand the lease-duration tradeoff and size it deliberately, not as an afterthought |
| Upgrading a cluster's Kubernetes minor version without checking for deprecated API usage first | A manifest, Helm chart, or controller referencing a removed API version starts failing outright the moment the control plane no longer serves it | Run a deprecated-API-usage check (`kubectl get --raw /metrics`, or a tool like `pluto`) as a mandatory, non-skippable pre-upgrade step |
| Leaving kubelet's API port (10250) with anonymous access enabled on a self-managed cluster | A well-known, serious real-world attack vector — lets an unauthenticated caller exec into containers or read logs directly | Enforce `--anonymous-auth=false` and `--authorization-mode=Webhook` on every kubelet, verify it explicitly, don't assume secure defaults |
---
## Worked Practice Problems
**Problem 1:** A Deployment specifies `replicas: 5`, but `kubectl get pods` shows only 3 running, with no error events visible. Walk through which components are involved in eventually fixing this, and how.
*Answer:* The Deployment Controller (part of the Controller Manager), continuously reconciling, compares the desired state (5 replicas) against the actual observed state (3 pods) via the API Server, and on its next reconciliation pass creates 2 new Pod objects to close the gap — with no node assigned yet. The Scheduler, watching for unscheduled pods, picks a suitable node for each of the 2 new pods and records that decision via the API Server. Each chosen node's kubelet, watching for pods assigned to its own node, sees the new assignment and instructs the container runtime to actually start the containers. This entire chain happens automatically, with zero human intervention, purely as a consequence of the reconciliation loop pattern.
**Problem 2:** A container is under heavy, legitimate load and its response times have temporarily climbed above its liveness probe's timeout threshold, causing kubelet to repeatedly restart it — making the underlying overload problem even worse. What's misconfigured, and what's the fix?
*Answer:* The liveness probe is being used to judge something it shouldn't — genuine, temporary business load isn't the same as "this container is broken/deadlocked," which is what liveness probes should be reserved for. The fix: loosen the liveness probe's timeout/failure threshold so temporary slowness under real load doesn't trigger a restart, and rely on the readiness probe instead to temporarily pull the pod out of Service rotation during genuine overload — letting it finish its existing work and recover on its own, rather than repeatedly restarting it and discarding progress.
**Problem 3:** Someone argues a 4-node etcd cluster is "safer" than a 3-node one because "more nodes means more redundancy." Explain why this reasoning is flawed.
*Answer:* etcd requires a strict majority (quorum) to agree before any write commits. A 3-node cluster needs 2 of 3 to agree and can tolerate exactly 1 node failing. A 4-node cluster needs 3 of 4 to agree — and can STILL only tolerate exactly 1 node failing (losing 2 of 4 breaks the majority requirement just as it would with 3 nodes). The 4th node adds real cost (more compute, more storage, more network overhead for consensus) without improving fault tolerance at all compared to the 3-node setup — which is exactly why etcd clusters are always sized with an odd number of members.
**Problem 4:** A self-managed cluster's etcd data directory has silently grown to fill available disk space over several months, and `kubectl apply` for anything, cluster-wide, has started failing with no application-level explanation. What's happening, and what's the fix?
*Answer:* This is almost certainly etcd's `NOSPACE` alarm — its configured storage quota has been exceeded, likely because compaction and defragmentation were never scheduled as routine maintenance, so old revisions accumulated (or were compacted but never defragmented, leaving the on-disk file large despite freed logical space). The fix: compact etcd's history up to a recent revision, defragment each etcd member one at a time (never simultaneously, since defrag briefly blocks the member being defragmented and doing all members at once risks a temporary full quorum outage), then explicitly disarm the `NOSPACE` alarm with `etcdctl alarm disarm` — etcd continues rejecting all writes until the alarm is explicitly cleared, even after space has been freed. Going forward, this points to a real operational gap: compaction and defragmentation should be automated on a schedule, not handled reactively after an outage.
**Problem 5:** After a routine Kubernetes minor-version upgrade, a CI/CD pipeline that had been reliably applying a set of Ingress manifests for over a year suddenly starts failing every deployment with a "resource not found" error, with no changes made to the manifests themselves. What's the most likely explanation, and how should this have been caught before the upgrade?
*Answer:* The most likely explanation is that the manifests reference a deprecated, now-removed API version (a classic real example: `extensions/v1beta1` Ingress objects, formally removed in Kubernetes 1.22) — the upgrade moved the control plane past the point where that API version is served at all, so requests referencing it now fail outright rather than being silently translated. This should have been caught with a deprecated-API-usage scan (`kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis`, or a dedicated tool like `pluto`) run as a mandatory step in the upgrade runbook, before the upgrade — Kubernetes's formal deprecation policy guarantees advance notice measured in API minor versions specifically so this kind of check is possible ahead of time, not just discoverable after the fact. The concrete fix here: update the Ingress manifests to the current stable `networking.k8s.io/v1` API version and re-apply.
---
## Summary and What's Next
- Kubernetes's core job: continuously make the **actual state** of the cluster match the **desired state** you declare, automatically — this is the entire point of the system.
- The cluster splits into the **Control Plane** (API Server, etcd, Scheduler, Controller Manager — the "brain") and **Worker Nodes** (kubelet, kube-proxy, container runtime — where containers actually run).
- The **API Server** is the sole gatekeeper to all cluster state — nothing else talks directly to **etcd**, which is the cluster's entire source of truth and requires a quorum-based majority (always an odd number of members) to commit any write.
- Every request passes through **authentication → authorization → admission control**, in that strict order — mutating webhooks run before validating webhooks, so validation always sees the final, fully-mutated object.
- **etcd's Raft consensus** requires a majority to commit a write and elects a single Leader via randomized-timeout elections — during a leader election, etcd (and therefore the whole cluster) briefly cannot accept writes. **Compaction and defragmentation** are separate, both-required maintenance steps to keep the data file from growing unbounded.
- The **reconciliation loop** (observe -> compare -> act, forever) is the single most important pattern in Kubernetes — it's the mechanism behind every controller and the entire reason the system self-heals without human intervention.
- Control plane components run multiple replicas for HA, coordinated via **leader election** using a `Lease` object — only one replica is ever active, with a real, bounded reconciliation gap during failover. The **API Server itself is the exception**: stateless, all replicas active simultaneously, no leader election needed.
- The API is genuinely **extensible** two distinct ways: **CRDs** (native, etcd-backed, the Operator pattern's foundation) and **aggregated API servers** (proxied to an independent process, used for non-durable data like live metrics).
- APIs move through **alpha → beta → stable**, with a formal deprecation policy protecting stable APIs — always scan for deprecated API usage before any minor-version upgrade.
- The **watch cache** means the vast majority of reads never actually hit etcd — etcd absorbs writes, the in-memory cache absorbs reads, which is precisely how the control plane scales to large, high-churn clusters without etcd becoming a bottleneck.
- **Node Conditions** (`Ready`, `MemoryPressure`, `DiskPressure`, `PIDPressure`, `NetworkUnavailable`) are separately-tracked health signals reported on a heartbeat, with a deliberate two-stage timeout (mark Unknown, then evict after a grace period) to avoid overreacting to transient blips.
- The kubelet's own API (port 10250) is a genuinely powerful, security-critical surface — always verify anonymous access is disabled and webhook authorization is enforced, never assume secure defaults on a self-managed cluster.
- Admission control's own three-stage chain — **authentication, authorization, admission** — runs in that strict order for every single request, with mutating webhooks always evaluated before validating ones.
- **CRDs and aggregated API servers** are the two genuinely distinct extension mechanisms — CRDs store data in the cluster's own etcd like a native resource, aggregated APIs proxy to an entirely separate process for non-durable data like live metrics.
- The **API Server's own health endpoints** (`/healthz`, `/livez`, `/readyz`) mirror the pod-level liveness/readiness distinction one layer up — `/readyz?verbose` breaks readiness down into individual checks, including an explicit etcd-connectivity check, genuinely useful during real troubleshooting.
- The **Scheduler** only decides *where* a pod should run; the **kubelet** on that specific node is what actually starts and monitors it.
- **Liveness probes** answer "should this be restarted"; **readiness probes** answer "should this receive traffic right now" — conflating the two is a classic, damaging misconfiguration.
- **kube-proxy** turns a Service's stable virtual IP into real, working network rules across every node.
- Kubernetes's **declarative philosophy** (describe the desired end state, not the steps to get there) is precisely what makes `kubectl apply` idempotent and the whole system self-healing.
**Continue to Part 2** (`02-scheduling-and-workloads.md`) for a full deep dive into exactly how the Scheduler makes its placement decisions, and the different workload objects (Deployments, StatefulSets, DaemonSets, Jobs) built on top of this foundation.