Architecture & Control Plane

Table of Contents#

1. What Problem Kubernetes Actually Solves
2. The Big Picture: Control Plane vs Worker Nodes
3. The API Server — The Front Door to Everything
4. The Admission Control Chain: Authentication, Authorization, Admission
5. Mutating and Validating Admission Webhooks, In Depth
6. kube-apiserver Flags That Matter Operationally
7. etcd — Kubernetes's Memory
8. etcd's Raft Consensus, In Depth
9. etcd Compaction, Defragmentation, and Alarms
10. The Controller Manager and the Reconciliation Loop
11. Leader Election — How Control Plane Components Avoid Split-Brain
12. The Scheduler, at a High Level
13. Worker Node Components
14. The Kubelet — The Node's Local Agent
15. Node Heartbeats and Node Conditions
16. kube-proxy — Making Services Actually Work
17. The Container Runtime
18. API Server High Availability
19. Extending the API: Aggregated API Servers and CRDs at the Wire Level
20. Control Plane Component Ports and Communication, Reference
21. A Full Worked Journey: kubectl apply to a Running Pod
22. Kubernetes API Versioning and Deprecation Policy
23. The API Server's Own Health Endpoints
24. Declarative vs Imperative — The Core Philosophy
25. Common Mistakes
26. Worked Practice Problems
27. Summary and What's Next

What Problem Kubernetes Actually Solves#

Before any component-by-component detail, it's worth being able to answer the single most common opening Kubernetes interview question in one clean breath: what problem does Kubernetes actually solve?

The single-sentence answer worth memorizing: "Kubernetes is a system that continuously works to make the actual state of your infrastructure match the desired state you've declared — placing containers on machines, restarting them when they fail, routing traffic to them, and scaling them — without a human manually doing any of that."

The Big Picture: Control Plane vs Worker Nodes#

Every Kubernetes cluster splits into two fundamentally different kinds of machines, each with a distinct job.

Simple analogy: the Control Plane is like a restaurant's head office — it decides the menu (desired state), tracks inventory (current state), and issues instructions. Worker nodes are the actual kitchens where food (containers) really gets cooked and served. The head office never cooks anything itself — it only ever tells kitchens what to do and watches what's actually happening.

The API Server — The Front Door to Everything#

The API Server (kube-apiserver) is the single, central entry point for absolutely everything in Kubernetes — every kubectl command, every internal component, every automated controller talks to Kubernetes exclusively through this one component.

A genuinely important architectural fact, worth stating explicitly: NOTHING in Kubernetes talks directly to etcd except the API Server. Every other component — the scheduler, controllers, kubelets — only ever reads and writes cluster state by calling the API Server, which is the sole gatekeeper to the actual stored data. This single-entry-point design is what makes authentication, authorization (RBAC, from the DevSecOps series), and validation possible to enforce consistently across the entire cluster.

# Every single kubectl command is really just an HTTP request
# to the API server — you can see this directly:
kubectl get pods -v=8 2>&1 | grep "GET https"
# GET https://<api-server>/api/v1/namespaces/default/pods

The Admission Control Chain: Authentication, Authorization, Admission#

Every request hitting the API Server passes through three genuinely distinct stages, in a strict order — a frequently-tested, precise sequence worth knowing by name, not just "there's some security checking."

Why the ordering matters, worth stating precisely: authentication and authorization answer "who are you, and are you generally allowed to do this kind of thing" — coarse-grained, identity-based questions the DevSecOps series' RBAC material already covers in depth. Admission control is a genuinely different, later stage: by the time a request reaches it, the caller is already known and already authorized in principle — admission control instead asks fine-grained, content-based questions about this specific request ("does this Pod spec violate our Pod Security Standard," "does this Deployment omit required labels," "does this Namespace already have too many Pods for its ResourceQuota").

Built-in admission controllers worth knowing by name, since "admission control" is often assumed to mean only custom webhooks, when in fact most clusters rely heavily on built-in ones:

Mutating and Validating Admission Webhooks, In Depth#

Beyond the built-in admission controllers, Kubernetes lets you register your own admission logic as an HTTP webhook — this is the exact mechanism service meshes (Part 4), policy engines like OPA Gatekeeper or Kyverno, and many operators use to enforce custom rules or automatically inject configuration.

The mutating-before-validating ordering is deliberate and worth stating explicitly: if validation ran first, a webhook meant to reject non-compliant Pods might reject a Pod that a later mutating webhook would have fixed automatically (e.g. injecting a missing required label). Running all mutations first means validation always evaluates the final, fully-mutated object — the actual object that will actually be persisted and run.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: require-resource-limits
webhooks:
  - name: require-resource-limits.example.com
    clientConfig:
      service:
        name: policy-webhook
        namespace: platform
        path: "/validate"
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
    failurePolicy: Fail
    admissionReviewVersions: ["v1"]

failurePolicy is a genuinely important, easy-to-get-wrong field worth knowing precisely: Fail means if the webhook itself is unreachable (network issue, the webhook's own pod is down), the API Server rejects the request — safe by default, but means a broken webhook can block ALL matching operations cluster-wide, including emergency changes. Ignore means an unreachable webhook is silently skipped, letting the request through unvalidated — safer for availability, but means policy enforcement has a real, silent failure mode. A strong interview answer names this as a genuine tradeoff, not a "just pick Fail" default: critical security policy webhooks often justify Fail (worth the availability risk), while a nice-to-have labeling/defaulting webhook is often better as Ignore (don't let it become a single point of cluster-wide failure).

kube-apiserver Flags That Matter Operationally#

Most kube-apiserver configuration is invisible day-to-day on managed Kubernetes (EKS/AKS/GKE own it entirely, per Part 5), but understanding the flags that matter is genuinely useful both for self-managed clusters (Part 6) and for reasoning about why a managed cluster behaves the way it does.

kube-apiserver \
  --etcd-servers=https://127.0.0.1:2379 \
  --service-cluster-ip-range=10.96.0.0/12 \
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ResourceQuota,PodSecurity \
  --audit-log-path=/var/log/kubernetes/audit.log \
  --audit-log-maxage=30 \
  --audit-policy-file=/etc/kubernetes/audit-policy.yaml \
  --authorization-mode=Node,RBAC \
  --max-requests-inflight=400 \
  --max-mutating-requests-inflight=200

Why --authorization-mode=Node specifically matters, a concrete security point: without it, a compromised kubelet credential (stolen from one specific node) could, under RBAC alone, potentially be scoped broadly enough to read secrets or pod specs belonging to other nodes — the Node authorizer adds an additional, node-scoped restriction on top of RBAC specifically for kubelet identities, limiting a single compromised node's blast radius to its own workloads.

etcd — Kubernetes's Memory#

etcd is a distributed, consistent key-value store — and it is, quite literally, the entire source of truth for a Kubernetes cluster's state. Every object (every Pod, Deployment, Service, Secret) is stored here.

Why etcd is a genuinely critical, high-stakes component — worth stressing explicitly in an interview: if etcd is lost or corrupted with no backup, the cluster effectively has amnesia — it has no memory of what should be running, where, or how it was configured. This directly connects to the Disaster Recovery topic (topic 11) in this course: etcd backups are one of the single most critical, non-negotiable disaster-recovery practices for any self-managed Kubernetes cluster.

# Take a backup of etcd (run on a control plane node)
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Verify the snapshot
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot.db --write-out=table

Why etcd specifically needs strong consistency (CP, not AP, using the CAP theorem vocabulary from the Reliability & Architecture Patterns series): if two control plane replicas ever disagreed about whether a Pod exists, the cluster's behavior would become genuinely unpredictable. This is exactly why etcd uses the Raft consensus algorithm, requiring a majority (quorum) of its members to agree before any write is considered committed — directly the same quorum principle (W + R > N) covered in that earlier tutorial.

Why etcd clusters always use an odd number of nodes, a genuinely common, sharp interview question: a 4-node cluster still only tolerates 1 failure (needs 3 of 4 to agree — same as a 3-node cluster needing 2 of 3), but costs an entire extra node for zero additional fault tolerance. An odd number is always the efficient choice.

etcd's Raft Consensus, In Depth#

The quorum requirement above is a consequence of etcd's underlying consensus algorithm, Raft — worth understanding at the mechanism level, not just "it needs a majority."

The core Raft roles, worth naming precisely: at any moment, exactly one etcd member is the Leader (the only one that accepts writes), and the rest are Followers (passively replicating the Leader's log). If the Leader fails or becomes unreachable, remaining members hold a leader election — each Follower waits a randomized timeout, and the first to time out becomes a Candidate, requesting votes from the others; whichever Candidate gets a majority of votes becomes the new Leader.

Why the randomized election timeout matters, a genuinely subtle but real design detail worth knowing: if every Follower used the same fixed timeout, they'd all become Candidates simultaneously after a Leader failure, splitting the vote repeatedly with no majority winner — the randomization means, in practice, one Follower almost always times out meaningfully before the others, giving it a clean shot at winning the election before a second Candidate even enters the race.

The direct, practical consequence for cluster operators, worth stating explicitly: during a Leader election (which is typically sub-second, but real), etcd cannot accept new writes — meaning the Kubernetes API Server cannot write any new/updated cluster state during that brief window. Reads of already-committed data can often still be served, but write-heavy operations (creating pods, scaling deployments) will briefly queue or fail. This is exactly why etcd's own recommended deployment topology emphasizes low-latency, stable networking between members — frequent, unnecessary leader elections caused by network flakiness directly degrade the entire cluster's ability to accept changes, not just etcd itself.

etcd Compaction, Defragmentation, and Alarms#

Two genuinely real, operationally-important etcd maintenance concepts, easy to overlook until they cause a production incident.

Why compaction and defragmentation are two distinct steps, a real, commonly-missed operational detail: compaction tells etcd's internal MVCC (multi-version concurrency control) store "these old revisions are no longer needed" — but etcd's underlying storage engine (bbolt, a B+tree-based store) doesn't automatically shrink the actual file on disk just because logical space was freed; it just marks that space as reusable for future writes. Defragmentation is the separate operation that actually returns freed space to the filesystem, reducing the on-disk file size. Skipping defragmentation on a long-running, write-heavy cluster is a genuinely common cause of etcd's data file silently growing until it hits its default storage quota (commonly 2GB) — at which point etcd stops accepting writes entirely.

# Compact etcd history up to a specific revision
ETCDCTL_API=3 etcdctl compact $(etcdctl endpoint status --write-out="json" | grep -o '"revision":[0-9]*' | grep -o '[0-9]*')

# Defragment (run per-member, one at a time, never all simultaneously —
# defrag briefly blocks that member, and doing all at once risks a
# temporary full quorum outage)
ETCDCTL_API=3 etcdctl defrag --endpoints=https://127.0.0.1:2379

# Check for active alarms (e.g. NOSPACE — quota exceeded)
ETCDCTL_API=3 etcdctl alarm list

The NOSPACE alarm, worth knowing as a real, concrete failure mode: when etcd's storage quota is exceeded, it raises a NOSPACE alarm and rejects all writes cluster-wide until the alarm is explicitly disarmed — meaning kubectl apply for anything, cluster-wide, starts failing. Recovery requires compacting history, defragmenting to actually reclaim disk space, and then explicitly clearing the alarm (etcdctl alarm disarm) — a genuinely realistic, high-severity incident scenario for any team running self-managed Kubernetes without regular etcd maintenance automated.

The Controller Manager and the Reconciliation Loop#

This is arguably the single most important conceptual idea in all of Kubernetes — genuinely worth spending real time to understand deeply, since almost everything else in the system is built on top of this one pattern.

This loop — observe, compare, act, repeat forever — is called a controller, and it's the fundamental unit of automation in Kubernetes. A Controller Manager process runs dozens of these loops simultaneously, each one responsible for one specific type of object (a Deployment Controller, a ReplicaSet Controller, a Node Controller, and many more).

Simple analogy: think of a home thermostat. It doesn't "turn on the heat once" — it continuously checks the current temperature against your desired setting, and takes action (heat on/off) whenever there's a gap, forever, without you doing anything. Every Kubernetes controller works exactly this way, just applied to cluster objects instead of temperature.

The direct, practical payoff of this design, worth stating explicitly: this is exactly why Kubernetes self-heals. If a node dies and takes a pod with it, you don't need any human or script to notice and react — the relevant controller notices the gap between desired (3 replicas) and actual (2 replicas) on its very next reconciliation pass (which happens continuously, many times a second) and simply creates a replacement, automatically, with zero human involvement.

Leader Election — How Control Plane Components Avoid Split-Brain#

Production clusters run the Controller Manager and Scheduler as multiple replicas for high availability (directly the same "don't run a single point of failure" principle from the Reliability & Architecture Patterns series) — but only ONE replica of each should actually be active at a time, or you'd get duplicate, conflicting reconciliation actions. This is solved with the exact same leader election pattern etcd itself uses internally (previous section), applied one layer up.

The mechanism, concretely: each replica repeatedly attempts to create or update a Lease object (a standard Kubernetes API object, stored in etcd like everything else) with its own identity and a short expiry, using an atomic compare-and-swap operation. Whichever replica succeeds becomes the leader and must continuously renew that lease before it expires — if the leader crashes or is partitioned away, it stops renewing, the lease expires, and one of the standby replicas acquires it and takes over.

# See which replica currently holds leadership for a given component
kubectl get lease -n kube-system kube-controller-manager -o yaml
# holderIdentity: <pod-name-of-the-current-leader>

Why this matters concretely for cluster operators, a genuinely important operational fact: during the brief window between a leader crashing and a standby acquiring the lease (bounded by the lease's expiry duration, commonly a handful of seconds), no reconciliation happens at all for whatever component just lost leadership — new Deployments won't get their ReplicaSets created, failed pods won't be replaced, until a new leader takes over. This is a real, bounded gap, not an instantaneous failover, and is exactly why the lease duration is a deliberate tradeoff: too short risks unnecessary leadership churn during brief network blips; too long extends the reconciliation gap during a genuine failure.

The Scheduler, at a High Level#

The Scheduler (kube-scheduler) has one specific job: when a new Pod is created with no node assigned yet, decide which node it should actually run on. (The full mechanics — filtering, scoring, affinity rules — get their own deep dive in Part 2.)

A genuinely important point worth stating explicitly: the Scheduler only ever DECIDES and RECORDS which node a pod should run on — it never actually starts a container itself. That job belongs entirely to the kubelet on the chosen node, covered next.

Worker Node Components#

Every worker node runs three essential components, each with one specific job.

The Kubelet — The Node's Local Agent#

The kubelet is the only Kubernetes component running on a worker node that talks directly to the API Server. It's responsible for making sure the containers assigned to its node are actually running, healthy, and match their spec.

Liveness vs. readiness probes — a genuinely common, specific interview distinction:

Why this distinction matters practically, and it's a classic interview trap: a container that's temporarily overwhelmed and slow (but not actually broken) should fail its readiness probe (stop receiving new traffic until it catches up) but should absolutely NOT fail its liveness probe — killing and restarting a container that's just temporarily busy makes the problem worse, not better, by throwing away whatever progress it had made and adding restart overhead on top of an already-struggling situation.

apiVersion: v1
kind: Pod
metadata:
  name: checkout
spec:
  containers:
    - name: app
      image: checkout:1.2.3
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 10
        periodSeconds: 10
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        periodSeconds: 5
        failureThreshold: 2

Node Heartbeats and Node Conditions#

Beyond managing individual pods, the kubelet has a second, continuous job worth understanding: reporting the health of the node itself back to the control plane, on a regular heartbeat.

Node Conditions worth knowing by name, since "the node is unhealthy" is actually several distinct, separately-tracked signals:

Why the two-stage timeout (Unknown, then eviction after a grace period) matters, a genuinely important reliability nuance: a node briefly missing its heartbeat due to a short network blip shouldn't immediately trigger evicting and rescheduling every pod on it — that would be wasteful, disruptive churn for what might resolve itself in seconds. The Node Controller's default behavior deliberately waits through a longer grace period (pod-eviction-timeout, default 5 minutes) before concluding the node is genuinely gone and starting pod eviction — directly the same "don't overreact to a transient blip" principle behind readiness-probe failure thresholds earlier in this Part, applied at the node level instead of the container level.

# See a node's current conditions directly
kubectl describe node worker-3 | grep -A 10 Conditions

kube-proxy — Making Services Actually Work#

kube-proxy runs on every node and is responsible for implementing the actual networking rules that make a Kubernetes Service (a stable, virtual IP that load-balances across a changing set of pods) actually work.

In plain terms: without kube-proxy, a Service's stable virtual IP would just be an abstract idea with nothing actually making it work — kube-proxy is the component that turns "traffic to this Service" into "actual network rules routing to real, currently-healthy pod IPs," updated automatically every time pods come and go. (The full mechanics of Services get their own deep dive in Part 3.)

The Container Runtime#

The actual layer that runs containers — the lowest-level component in this whole stack, sitting directly on top of the Linux kernel primitives (namespaces and cgroups) covered in the Linux & Networking Fundamentals series.

Why the CRI (Container Runtime Interface) matters, worth knowing by name: Kubernetes doesn't hardcode a dependency on any one specific container runtime — it talks to whatever runtime is installed through this standard interface. This is exactly why Kubernetes could cleanly deprecate direct Docker support (a well-known, sometimes misunderstood industry event) without actually breaking anything for end users — Docker-built images still work fine, because the image format (OCI-compliant) is separate from the runtime that executes containers, and any CRI-compliant runtime (like containerd, which Docker itself is built on top of) can run them.

API Server High Availability#

A single kube-apiserver instance would be an obvious single point of failure — real clusters (and every managed offering from Part 5) run multiple API server replicas, fronted by a load balancer.

A genuinely important architectural point worth stating precisely: unlike the Controller Manager and Scheduler, API Server replicas do NOT use leader election — every replica is simultaneously active and can independently serve any request. This is possible specifically because the API Server itself is largely stateless — it doesn't hold cluster state in memory as its own source of truth; it reads and writes through to etcd for everything. Any replica can serve any read or write, and etcd's own consensus (previous sections) is what actually guarantees consistency, not coordination between the API Server replicas themselves.

Why this distinction between "stateless, all-active" (API Server) and "stateful reconciliation, single-active-via-leader-election" (Controller Manager, Scheduler) is a strong, precise interview answer: it demonstrates understanding that HA isn't one uniform pattern applied identically to every control plane component — the right HA mechanism depends on whether the component actually needs single-writer semantics (reconciliation loops absolutely do, to avoid duplicate/conflicting actions) or can safely be handled by any replica independently (the API Server, because etcd is the actual arbiter of consistency).

# On a self-managed cluster, you can see multiple API server endpoints
# directly in kubeconfig if configured behind a load balancer:
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'

On managed Kubernetes specifically (Part 5, Part 7): this entire API server HA topology — how many replicas, how they're load-balanced, how they scale under load — is exactly the part of "the control plane" that EKS/AKS/GKE fully own and abstract away. You interact with a single, stable API endpoint URL and never need to reason about the replica count or load-balancing mechanism behind it, which is precisely the "managed control plane" value proposition named at the start of Part 5.

The watch cache — why most reads don't actually hit etcd, a genuinely important performance detail: every API server replica maintains an in-memory watch cache of recently-seen objects, kept continuously up to date by watching etcd for changes. The vast majority of kubectl get/list/watch requests (the overwhelming majority of real API traffic, since controllers and kubelets are constantly watching for changes) are served directly from this in-memory cache, not by querying etcd on every single read. This is exactly why etcd — despite being "the entire source of truth" — doesn't become a read bottleneck even in large, high-churn clusters: etcd primarily absorbs writes and the watch cache absorbs reads, a division of labor worth stating explicitly when asked how Kubernetes scales its own control plane.

Extending the API: Aggregated API Servers and CRDs at the Wire Level#

Kubernetes's API surface isn't fixed — it's genuinely extensible, and understanding the mechanism (not just "CRDs exist," which gets full operator-pattern treatment in Part 4) belongs here as an architectural fact about the API Server itself.

Why these are two genuinely different extension mechanisms, worth distinguishing precisely — a common point of confusion even among experienced practitioners:

# See registered aggregated API services on a cluster
kubectl get apiservices | grep -v "^NAME\|Local"
# v1beta1.metrics.k8s.io   kube-system/metrics-server   True

Why metrics-server specifically is the canonical aggregated-API example, worth explaining precisely: live CPU/memory usage changes constantly and has no lasting value as durable cluster state — storing a continuous stream of point-in-time metrics in etcd (a system deliberately optimized for consistency and durability of relatively low-write-volume configuration data, not high-frequency time-series data) would be a genuine architectural mismatch. metrics-server instead runs as its own independent process, computes metrics from kubelet's own resource-usage reporting, and is only reachable through the main API server's proxy — kubectl top pods and the Horizontal Pod Autoscaler (Part 2) both query it through this exact aggregation mechanism, without metrics-server data ever touching the cluster's own etcd.

Control Plane Component Ports and Communication, Reference#

A concrete, worth-having reference table — genuinely useful both for understanding the architecture diagrams throughout this Part and for real troubleshooting (firewall rules, security group configuration on self-managed clusters per Part 6).

Why knowing port 10250 specifically matters as a real security fact, worth stating explicitly: the kubelet API is a genuinely powerful surface — it's what lets the API Server execute commands inside a running container (kubectl exec) or stream logs. An improperly secured kubelet API (missing authentication/authorization, a real misconfiguration seen in the wild) is a well-known, serious attack vector, directly connecting to the container and Kubernetes security material in the DevSecOps series (05-devsecops/03-container-and-kubernetes-security.md) — always verify kubelet authentication is enforced (--anonymous-auth=false, --authorization-mode=Webhook), never left at insecure defaults.

A Full Worked Journey: kubectl apply to a Running Pod#

Tying every component in this tutorial together into one complete, step-by-step story — genuinely one of the most valuable things to be able to narrate fluently in an interview.

A strong interview answer walks through this exact sequence, naming every component and its specific, narrow responsibility — this single narrative demonstrates the entire architecture in one coherent story, rather than a list of disconnected component definitions.

Kubernetes API Versioning and Deprecation Policy#

A final architectural fact worth knowing precisely, since it directly affects how you plan cluster upgrades (Part 5, Part 7) and write manifests that don't silently break.

The formal deprecation policy for stable (GA) APIs, worth knowing as a concrete, citable rule: once an API reaches v1 (stable), Kubernetes's own deprecation policy guarantees it remains supported for a minimum period measured in API minor version releases, not a vague "eventually" — giving operators genuine, predictable planning time before a migration is required. Beta APIs get a shorter, but still formal, guaranteed support window; alpha APIs have no such guarantee at all.

# Check a cluster for any deprecated API usage before upgrading —
# genuinely essential pre-upgrade due diligence
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

# Or use the community-standard tool built specifically for this
pluto detect-helm -o wide

Why this matters concretely, a real, common upgrade-incident pattern worth naming: a team that upgrades a cluster's Kubernetes minor version without first checking for deprecated API usage can find a previously-working manifest suddenly rejected outright post-upgrade, because the API version it referenced (e.g. an old extensions/v1beta1 Ingress, removed in Kubernetes 1.22) simply no longer exists on the new control plane. This is precisely why "check for deprecated API usage" is a mandatory, non-skippable step in any real Kubernetes upgrade runbook — directly connecting to the node-upgrade and control-plane-upgrade discussion in Part 5 and Part 7: the upgrade isn't just about node compatibility, it's equally about API compatibility for every manifest, Helm chart, and controller running in the cluster.

The API Server's Own Health Endpoints#

The pod-level liveness/readiness distinction covered earlier in this Part applies to the control plane's own components too — the API Server itself exposes several distinct health endpoints, worth knowing apart from application-level probes.

# Query the API server's own readiness, with per-check detail
kubectl get --raw='/readyz?verbose'
# [+]ping ok
# [+]log ok
# [+]etcd ok
# [+]poststarthook/start-kube-apiserver-admission-initializer ok
# readyz check passed

Why /readyz?verbose is worth knowing specifically as a real troubleshooting tool: when a self-managed API server is behaving strangely right after startup or an upgrade, this endpoint breaks down readiness into its individual constituent checks — including, notably, an explicit etcd check, meaning a genuinely fast way to confirm "is the API server's own etcd connectivity actually healthy right now" without needing to separately query etcd directly.

Declarative vs Imperative — The Core Philosophy#

A final, foundational concept worth stating explicitly, since it explains why Kubernetes is designed the way it is.

A clean, memorable interview line: "Kubernetes is fundamentally declarative — you describe what you want, not the steps to get there, and the reconciliation loop pattern is the engine that continuously, automatically closes the gap between what you asked for and what's actually running, which is exactly what makes the whole system self-healing without any human in the loop."

Common Mistakes#

Worked Practice Problems#

Problem 1: A Deployment specifies replicas: 5, but kubectl get pods shows only 3 running, with no error events visible. Walk through which components are involved in eventually fixing this, and how.

Answer: The Deployment Controller (part of the Controller Manager), continuously reconciling, compares the desired state (5 replicas) against the actual observed state (3 pods) via the API Server, and on its next reconciliation pass creates 2 new Pod objects to close the gap — with no node assigned yet. The Scheduler, watching for unscheduled pods, picks a suitable node for each of the 2 new pods and records that decision via the API Server. Each chosen node's kubelet, watching for pods assigned to its own node, sees the new assignment and instructs the container runtime to actually start the containers. This entire chain happens automatically, with zero human intervention, purely as a consequence of the reconciliation loop pattern.

Problem 2: A container is under heavy, legitimate load and its response times have temporarily climbed above its liveness probe's timeout threshold, causing kubelet to repeatedly restart it — making the underlying overload problem even worse. What's misconfigured, and what's the fix?

Answer: The liveness probe is being used to judge something it shouldn't — genuine, temporary business load isn't the same as "this container is broken/deadlocked," which is what liveness probes should be reserved for. The fix: loosen the liveness probe's timeout/failure threshold so temporary slowness under real load doesn't trigger a restart, and rely on the readiness probe instead to temporarily pull the pod out of Service rotation during genuine overload — letting it finish its existing work and recover on its own, rather than repeatedly restarting it and discarding progress.

Problem 3: Someone argues a 4-node etcd cluster is "safer" than a 3-node one because "more nodes means more redundancy." Explain why this reasoning is flawed.

Answer: etcd requires a strict majority (quorum) to agree before any write commits. A 3-node cluster needs 2 of 3 to agree and can tolerate exactly 1 node failing. A 4-node cluster needs 3 of 4 to agree — and can STILL only tolerate exactly 1 node failing (losing 2 of 4 breaks the majority requirement just as it would with 3 nodes). The 4th node adds real cost (more compute, more storage, more network overhead for consensus) without improving fault tolerance at all compared to the 3-node setup — which is exactly why etcd clusters are always sized with an odd number of members.

Problem 4: A self-managed cluster's etcd data directory has silently grown to fill available disk space over several months, and kubectl apply for anything, cluster-wide, has started failing with no application-level explanation. What's happening, and what's the fix?

Answer: This is almost certainly etcd's NOSPACE alarm — its configured storage quota has been exceeded, likely because compaction and defragmentation were never scheduled as routine maintenance, so old revisions accumulated (or were compacted but never defragmented, leaving the on-disk file large despite freed logical space). The fix: compact etcd's history up to a recent revision, defragment each etcd member one at a time (never simultaneously, since defrag briefly blocks the member being defragmented and doing all members at once risks a temporary full quorum outage), then explicitly disarm the NOSPACE alarm with etcdctl alarm disarm — etcd continues rejecting all writes until the alarm is explicitly cleared, even after space has been freed. Going forward, this points to a real operational gap: compaction and defragmentation should be automated on a schedule, not handled reactively after an outage.

Problem 5: After a routine Kubernetes minor-version upgrade, a CI/CD pipeline that had been reliably applying a set of Ingress manifests for over a year suddenly starts failing every deployment with a "resource not found" error, with no changes made to the manifests themselves. What's the most likely explanation, and how should this have been caught before the upgrade?

Answer: The most likely explanation is that the manifests reference a deprecated, now-removed API version (a classic real example: extensions/v1beta1 Ingress objects, formally removed in Kubernetes 1.22) — the upgrade moved the control plane past the point where that API version is served at all, so requests referencing it now fail outright rather than being silently translated. This should have been caught with a deprecated-API-usage scan (kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis, or a dedicated tool like pluto) run as a mandatory step in the upgrade runbook, before the upgrade — Kubernetes's formal deprecation policy guarantees advance notice measured in API minor versions specifically so this kind of check is possible ahead of time, not just discoverable after the fact. The concrete fix here: update the Ingress manifests to the current stable networking.k8s.io/v1 API version and re-apply.

Summary and What's Next#

• Kubernetes's core job: continuously make the actual state of the cluster match the desired state you declare, automatically — this is the entire point of the system.
• The cluster splits into the Control Plane (API Server, etcd, Scheduler, Controller Manager — the "brain") and Worker Nodes (kubelet, kube-proxy, container runtime — where containers actually run).
• The API Server is the sole gatekeeper to all cluster state — nothing else talks directly to etcd, which is the cluster's entire source of truth and requires a quorum-based majority (always an odd number of members) to commit any write.
• Every request passes through authentication → authorization → admission control, in that strict order — mutating webhooks run before validating webhooks, so validation always sees the final, fully-mutated object.
• etcd's Raft consensus requires a majority to commit a write and elects a single Leader via randomized-timeout elections — during a leader election, etcd (and therefore the whole cluster) briefly cannot accept writes. Compaction and defragmentation are separate, both-required maintenance steps to keep the data file from growing unbounded.
• The reconciliation loop (observe -> compare -> act, forever) is the single most important pattern in Kubernetes — it's the mechanism behind every controller and the entire reason the system self-heals without human intervention.
• Control plane components run multiple replicas for HA, coordinated via leader election using a Lease object — only one replica is ever active, with a real, bounded reconciliation gap during failover. The API Server itself is the exception: stateless, all replicas active simultaneously, no leader election needed.
• The API is genuinely extensible two distinct ways: CRDs (native, etcd-backed, the Operator pattern's foundation) and aggregated API servers (proxied to an independent process, used for non-durable data like live metrics).
• APIs move through alpha → beta → stable, with a formal deprecation policy protecting stable APIs — always scan for deprecated API usage before any minor-version upgrade.
• The watch cache means the vast majority of reads never actually hit etcd — etcd absorbs writes, the in-memory cache absorbs reads, which is precisely how the control plane scales to large, high-churn clusters without etcd becoming a bottleneck.
• Node Conditions (Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable) are separately-tracked health signals reported on a heartbeat, with a deliberate two-stage timeout (mark Unknown, then evict after a grace period) to avoid overreacting to transient blips.
• The kubelet's own API (port 10250) is a genuinely powerful, security-critical surface — always verify anonymous access is disabled and webhook authorization is enforced, never assume secure defaults on a self-managed cluster.
• Admission control's own three-stage chain — authentication, authorization, admission — runs in that strict order for every single request, with mutating webhooks always evaluated before validating ones.
• CRDs and aggregated API servers are the two genuinely distinct extension mechanisms — CRDs store data in the cluster's own etcd like a native resource, aggregated APIs proxy to an entirely separate process for non-durable data like live metrics.
• The API Server's own health endpoints (/healthz, /livez, /readyz) mirror the pod-level liveness/readiness distinction one layer up — /readyz?verbose breaks readiness down into individual checks, including an explicit etcd-connectivity check, genuinely useful during real troubleshooting.
• The Scheduler only decides where a pod should run; the kubelet on that specific node is what actually starts and monitors it.
• Liveness probes answer "should this be restarted"; readiness probes answer "should this receive traffic right now" — conflating the two is a classic, damaging misconfiguration.
• kube-proxy turns a Service's stable virtual IP into real, working network rules across every node.
• Kubernetes's declarative philosophy (describe the desired end state, not the steps to get there) is precisely what makes kubectl apply idempotent and the whole system self-healing.

Continue to Part 2 (02-scheduling-and-workloads.md) for a full deep dive into exactly how the Scheduler makes its placement decisions, and the different workload objects (Deployments, StatefulSets, DaemonSets, Jobs) built on top of this foundation.