# Kubernetes Deep Dive — Part 6: On-Prem & Self-Managed Kubernetes > **Series:** Kubernetes Deep Dive (6 of 9) > **Part 1:** `01-architecture-and-control-plane.md` — Architecture & Control Plane > **Part 2:** `02-scheduling-and-workloads.md` — Scheduling & Workload Objects > **Part 3:** `03-networking-and-storage.md` — Networking (CNI) & Storage (CSI) > **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators > **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE > **Part 6:** This file — On-Prem & Self-Managed Kubernetes > **Part 7:** `07-eks-deep-dive.md` — Amazon EKS in Production Depth > **Part 8:** `08-gateway-api-and-envoy-gateway.md` — Gateway API & Envoy Gateway > **Part 9:** `09-gateway-api-across-providers.md` — Gateway API Across GKE, EKS, AKS & On-Prem > **Questions:** `questions.md` ## Table of Contents 1. [Why Anyone Still Runs Kubernetes Themselves](#why-anyone-still-runs-kubernetes-themselves) 2. [kubeadm — The Official Bootstrapping Tool](#kubeadm--the-official-bootstrapping-tool) 3. [kubeadm HA Control Plane Setup](#kubeadm-ha-control-plane-setup) 4. [What kubeadm Deliberately Doesn't Do](#what-kubeadm-deliberately-doesnt-do) 5. [Lightweight Distributions: k3s, k0s, and MicroK8s](#lightweight-distributions-k3s-k0s-and-microk8s) 6. [Cluster API — Using Kubernetes to Manage Kubernetes](#cluster-api--using-kubernetes-to-manage-kubernetes) 7. [Cluster API In Depth: ClusterClass and Fleet Templating](#cluster-api-in-depth-clusterclass-and-fleet-templating) 8. [Rancher — Multi-Cluster Management](#rancher--multi-cluster-management) 9. [Rancher Fleet — GitOps for Multi-Cluster](#rancher-fleet--gitops-for-multi-cluster) 10. [Talos Linux — The Immutable, API-Only OS](#talos-linux--the-immutable-api-only-os) 11. [The Bare-Metal-Specific Gaps](#the-bare-metal-specific-gaps) 12. [Bare-Metal Node Provisioning: PXE Boot and Tinkerbell](#bare-metal-node-provisioning-pxe-boot-and-tinkerbell) 13. [MetalLB — Solving LoadBalancer Services On-Prem](#metallb--solving-loadbalancer-services-on-prem) 14. [MetalLB Layer 2 vs BGP Mode, In Depth](#metallb-layer-2-vs-bgp-mode-in-depth) 15. [Bare-Metal Storage](#bare-metal-storage) 16. [Air-Gapped Kubernetes](#air-gapped-kubernetes) 17. [Air-Gapped Cluster Setup, Step by Step](#air-gapped-cluster-setup-step-by-step) 18. [Node Maintenance and Hardware Failure On-Prem](#node-maintenance-and-hardware-failure-on-prem) 19. [A Full Worked Example: Building a Complete Bare-Metal Production Stack](#a-full-worked-example-building-a-complete-bare-metal-production-stack) 20. [A Full Worked Comparison](#a-full-worked-comparison) 21. [Part 6 CLI Cheat Sheet](#part-6-cli-cheat-sheet) 22. [Common Mistakes](#common-mistakes) 23. [Worked Practice Problems](#worked-practice-problems) 24. [Summary and What's Next](#summary-and-whats-next) --- ## Why Anyone Still Runs Kubernetes Themselves Given Part 5's case for managed Kubernetes, it's worth directly addressing: why does self-managed, on-premise Kubernetes remain genuinely common, not a legacy relic? This Part answers that question with every tool needed to actually run it well — bootstrapping, fleet lifecycle, bare-metal-specific gaps, and hardware maintenance. ```mermaid graph TD Reasons["Real Reasons Organizations
Self-Manage Kubernetes"] --> R1["Data residency/regulatory
requirements — data
legally CANNOT leave a
specific physical location
or country"] Reasons --> R2["Cost at genuine scale —
owned hardware can be
cheaper than cloud compute
for large, STEADY,
predictable workloads"] Reasons --> R3["Existing data center
investment — hardware
already owned, depreciated,
and staffed for"] Reasons --> R4["Latency-sensitive edge/
industrial workloads —
physically CANNOT be
in a distant cloud region"] Reasons --> R5["Air-gapped/highly
regulated environments
(defense, certain financial
systems) with NO internet
connectivity permitted
at all"] ``` **A strong, senior-level interview line, directly connecting to the RPO/RTO cost-curve discussion from the Disaster Recovery series: "The decision between managed and self-managed Kubernetes is a genuine cost/control tradeoff, exactly like choosing a DR strategy — self-managing trades ongoing operational toil (Part 5's core argument for managed) for control over cost at scale, data residency, and latency that a managed offering genuinely can't provide for certain workloads."** **The cost crossover point, worth understanding as a real, concrete shape rather than an abstract claim:** ```mermaid graph TD Small["SMALL/variable scale:
cloud's pay-as-you-go +
zero upfront capital +
zero data-center staffing
usually wins decisively"] --> SmallNote["Managed Kubernetes is
almost always the right
default here"] Large["LARGE, STEADY, predictable
scale: owned hardware's
per-unit cost, amortized
over years, can genuinely
undercut equivalent cloud
compute"] --> LargeNote["This is where self-managed
on-prem becomes a real,
defensible cost argument -
NOT a universal one"] ``` **Why this crossover point is worth naming honestly, rather than either extreme ("cloud is always cheaper" or "on-prem is always cheaper"):** the crossover depends on genuinely specific factors — existing data center lease/depreciation schedules, staffing already in place, the workload's actual predictability (steady-state vs. genuinely variable), and the real, fully-loaded cost of the engineering time this entire Part's tooling (kubeadm, CAPI, MetalLB, Rook/Ceph, and the ongoing operational burden of running all of it) actually requires. **A strong, honest answer names the crossover as workload- and organization-specific, never a blanket rule** — this is precisely why the reasons named at the top of this section (data residency, latency, existing investment, air-gapped requirements) are usually the deciding factors in practice, with raw cost-at-scale as a real but secondary consideration that only tips the balance for genuinely large, steady, predictable workloads. --- ## kubeadm — The Official Bootstrapping Tool **kubeadm** is the Kubernetes project's own, official tool for bootstrapping a cluster — genuinely worth knowing hands-on, since it's the closest thing to "vanilla," from-first-principles Kubernetes setup. ```bash # On the FIRST control plane node sudo kubeadm init --pod-network-cidr=10.244.0.0/16 # Configure kubectl for the current user mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config # Install a CNI plugin (Part 3) — kubeadm does NOT do this for you kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml # On EVERY worker node — join using the token kubeadm init printed sudo kubeadm join :6443 --token \ --discovery-token-ca-cert-hash sha256: ``` ```mermaid graph TD Init["kubeadm init
(on control plane node)"] --> Bootstrap["Bootstraps: API server,
etcd, scheduler, controller
manager — ALL the
components from Part 1"] Bootstrap --> Join["kubeadm join
(on EACH worker node)"] Join --> Cluster["A REAL, working
Kubernetes cluster —
built from the SAME
components covered
throughout this ENTIRE
series"] ``` **Why this is such a genuinely valuable exercise even for engineers who'll mostly use managed Kubernetes day to day: running `kubeadm init` yourself, once, makes every abstract component from Part 1 (the API server, etcd, the scheduler) suddenly concrete and tangible — you SEE them as actual running processes/pods, not just diagram boxes.** Many strong Kubernetes engineers deliberately do this at least once specifically to build that intuition. A genuinely practical follow-up worth trying immediately after: `kubectl get pods -n kube-system` on a freshly-kubeadm-initialized cluster shows the API server, etcd, scheduler, and controller-manager all running as regular, ordinary Pods — a real, concrete confirmation that Part 1's control-plane components aren't some special, hidden kind of process, just Kubernetes running Kubernetes. --- ## kubeadm HA Control Plane Setup The single-node example above is fine for learning, but genuinely production-worthy self-managed clusters need a highly-available control plane — worth understanding the real mechanics, directly extending Part 1's etcd/API-server-HA discussion to the self-managed context. ```mermaid graph TD LB["External Load Balancer
(HAProxy, keepalived, or a
hardware LB) - the STABLE
endpoint kubectl/kubelets
actually talk to"] --> CP1["Control Plane Node 1"] LB --> CP2["Control Plane Node 2"] LB --> CP3["Control Plane Node 3"] CP1 <--> CP2 CP2 <--> CP3 CP1 <--> CP3 ``` ```bash # First control plane node - note --control-plane-endpoint, # pointing at the LOAD BALANCER, not this node's own IP sudo kubeadm init \ --control-plane-endpoint "k8s-lb.internal:6443" \ --upload-certs \ --pod-network-cidr=10.244.0.0/16 # Additional control plane nodes join as CONTROL PLANE members, # not as workers - note the --control-plane flag sudo kubeadm join k8s-lb.internal:6443 \ --token \ --discovery-token-ca-cert-hash sha256: \ --control-plane \ --certificate-key ``` **Why the external load balancer is genuinely mandatory, not optional, for a real HA setup — worth stating precisely:** without it, every `kubectl`/kubelet client would need to hardcode one specific control plane node's IP, defeating the entire point of having multiple control plane replicas (Part 1's API-server-HA discussion) — the load balancer is what gives clients a single, stable endpoint that transparently routes to whichever control plane nodes are currently healthy, exactly mirroring the managed-provider behavior from Part 5 that you'd otherwise take for granted. `kube-vip` (introduced later in this Part, alongside MetalLB) is a genuinely common alternative to a dedicated external load balancer specifically for this control-plane-endpoint role. **Stacked vs. external etcd topology, a genuinely important, real architectural decision worth naming precisely:** ```mermaid graph TD Stacked["STACKED etcd (kubeadm's
default): each control plane
node runs its OWN etcd
member LOCALLY, alongside
its API server"] --> StackedNote["Simpler to operate - fewer
total machines - but a
control plane node failure
ALSO takes down an etcd
member simultaneously"] External["EXTERNAL etcd: a
SEPARATE, dedicated cluster
of etcd-only machines,
referenced by the control
plane nodes"] --> ExternalNote["More machines to operate,
but genuinely DECOUPLES
control-plane-node failures
from etcd-member failures -
preferred for the LARGEST,
most critical deployments"] ``` **Why this distinction matters concretely, worth stating as a real tradeoff:** with stacked etcd, losing 2 of 3 control plane nodes simultaneously (a real, if uncommon, failure mode — a rack power event, for instance) takes down both API server availability AND etcd quorum (Part 1) at once. External etcd means a control-plane-node failure and an etcd-member failure are genuinely independent events, at the cost of operating more machines — the same "simplicity vs. blast-radius isolation" tradeoff that recurs throughout this course, here applied to the control plane's own internal topology. Most real production kubeadm deployments use stacked etcd specifically because the operational simplicity is worth it at typical HA scale (3-5 control plane nodes) — external etcd is reserved for genuinely large, mission-critical deployments where the additional blast-radius isolation clearly justifies the extra machines and operational surface. **Worth confirming HA status directly, a real, practical diagnostic worth knowing:** `kubectl get pods -n kube-system -l component=etcd -o wide` shows every etcd member and which node it's running on — a quick, concrete way to verify a cluster's actual etcd topology matches what was intended, rather than assuming. --- ## What kubeadm Deliberately Doesn't Do A genuinely important, frequently-tested nuance — kubeadm is explicitly scoped to be a bootstrapping tool ONLY, not a full lifecycle management platform. ```mermaid graph TD Does["kubeadm DOES:"] --> D1["Bootstrap a NEW cluster's
control plane"] Does --> D2["Join NEW nodes to an
EXISTING cluster"] Does --> D3["Handle IN-PLACE version
upgrades of the control
plane"] DoesNot["kubeadm does NOT:"] --> N1["Provision the underlying
VMs/hardware itself"] DoesNot --> N2["Install a CNI plugin
(you must choose and
apply one yourself)"] DoesNot --> N3["Handle ongoing DAY-2
operations: monitoring,
backup automation, node
OS patching"] ``` **Why this matters practically, worth stating explicitly: kubeadm gets you to "a working cluster exists," but everything covered elsewhere in this course — etcd backups (Part 1/4), monitoring (Monitoring Methodologies series), CNI choice and NetworkPolicy enforcement (Part 3, DevSecOps series) — remains entirely your own responsibility to design and implement, with zero built-in guidance or automation from kubeadm itself.** This is precisely the gap the higher-level tools covered next (Cluster API, Rancher) exist to fill. **A useful, quick mental model worth stating explicitly: kubeadm is to a Kubernetes cluster roughly what `docker run` is to a full production container deployment** — it gets the core thing genuinely working, correctly, but every operational concern beyond that initial bootstrap (monitoring, backup automation, fleet consistency, day-2 lifecycle) is deliberately left as a separate, composable decision, not bundled in by default. --- ## Lightweight Distributions: k3s, k0s, and MicroK8s For resource-constrained environments (edge computing, IoT, development/testing, small on-prem deployments), full upstream Kubernetes can be genuinely heavier than needed — lightweight distributions trim it down. ```mermaid graph TD K3s["k3s (by Rancher/SUSE):
a SINGLE BINARY, under
100MB, with SQLite as a
lightweight etcd alternative
for small clusters, batteries-
included (built-in CNI,
Ingress controller, local
storage provisioner)"] --> K3sUse["Extremely popular for
EDGE/IoT deployments,
development environments,
and small on-prem clusters"] K0s["k0s (by Mirantis): similarly
lightweight, single-binary,
ZERO host OS dependencies
at all (doesn't even need
systemd)"] --> K0sUse["Popular for similarly
constrained/embedded
use cases, with an even
MORE minimal host
footprint"] MicroK8s["MicroK8s (by Canonical):
distributed as a SNAP
package, with an add-on
MODEL ('microk8s enable
dns storage ingress') for
opting INTO extra components
rather than bundling them
all by default"] --> MicroK8sUse["Popular specifically in
Ubuntu-centric shops,
and for local developer
workstations"] ``` ```bash # Installing k3s is genuinely this simple — one command, # a real, working cluster in under a minute curl -sfL https://get.k3s.io | sh - # Check it's running sudo k3s kubectl get nodes # MicroK8s's distinctive add-on model sudo snap install microk8s --classic microk8s enable dns storage ingress microk8s kubectl get nodes ``` **Why the simplicity of that one-line install is worth highlighting explicitly: it's a dramatic contrast to kubeadm's multi-step process, specifically because k3s ships with sane, working defaults for the pieces kubeadm deliberately leaves to you (CNI, storage, ingress) — a genuine, real tradeoff of "opinionated and simple" versus "unopinionated and flexible."** Every one of k3s's bundled defaults (its own lightweight CNI, its `local-path-provisioner` storage class, its built-in Traefik Ingress Controller) can be individually disabled and swapped for a different choice at install time — it's an opinionated starting point, not a hard architectural lock-in. **Why MicroK8s's add-on model is worth knowing as a genuinely distinct middle ground, precisely between k3s's "batteries-included by default" and kubeadm's "nothing included at all":** rather than either extreme, MicroK8s ships a genuinely minimal core and lets you explicitly opt into exactly the components a given use case needs (`dns`, `storage`, `ingress`, `metrics-server`, and more, each a single named add-on) — a real, practical middle point on the same "opinionated vs. flexible" spectrum every lightweight distribution sits somewhere along. All three remain genuinely conformant, real Kubernetes underneath their respective packaging choices — the exact same CNCF conformance guarantee already discussed for the major managed providers in Part 5 applies here too, so workload manifests remain fully portable regardless of which lightweight distribution actually runs them. | Distribution | Default etcd alternative | Package format | Distinctive strength | |---|---|---|---| | k3s | SQLite (or embedded etcd for HA) | Single binary | Broadest ecosystem adoption, genuinely production-proven at edge scale | | k0s | SQLite or embedded etcd | Single binary, zero OS deps | Smallest possible host footprint, no systemd requirement | | MicroK8s | Embedded dqlite | Snap package | Explicit, granular add-on model; strong Ubuntu/desktop-dev story | --- ## Cluster API — Using Kubernetes to Manage Kubernetes A genuinely elegant, conceptually important tool worth understanding deeply — **Cluster API (CAPI)** is a CNCF project that manages the **lifecycle of Kubernetes clusters themselves** using Kubernetes's own declarative, reconciliation-loop model (Part 1) as the management mechanism. ```mermaid graph TD Mgmt["A small 'Management
Cluster' (itself a
Kubernetes cluster)"] --> CRDs["Runs CUSTOM RESOURCES
(Part 4) representing
OTHER clusters:
'Cluster', 'Machine',
'MachineDeployment'"] CRDs --> Controllers["CAPI Controllers
(Part 4's Operator pattern)
watch these objects and
reconcile REAL infrastructure
to match"] Controllers --> Workload1["Workload Cluster 1
(actually created/managed)"] Controllers --> Workload2["Workload Cluster 2
(actually created/managed)"] ``` ```yaml # A simplified CAPI Cluster object — declaring a DESIRED # cluster, exactly like declaring a Deployment (Part 2) apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: production-cluster spec: clusterNetwork: pods: cidrBlocks: ["10.244.0.0/16"] infrastructureRef: kind: VSphereCluster # or AWSCluster, MetalCluster, etc. name: production-cluster ``` **Why this is genuinely one of the most conceptually elegant tools worth citing in an interview, and it directly closes the loop with Part 1's reconciliation-loop discussion: "Cluster API applies the EXACT same declarative, reconciliation-loop pattern used for Pods and Deployments — the fundamental idea from Part 1 — one level UP, to entire clusters. You declare 'I want a cluster that looks like this,' and CAPI's controllers continuously work to make that true, exactly the same way a Deployment Controller keeps your pod count correct."** CAPI supports a wide range of "infrastructure providers" (AWS, Azure, GCP, vSphere, bare metal, and more), making it a genuinely provider-agnostic way to manage fleets of clusters declaratively. **Worth stating precisely: CAPI can even manage clusters running ON managed cloud providers themselves (an `AWSManagedControlPlane` for EKS, for instance)** — meaning the same declarative, GitOps-friendly lifecycle model can span self-managed on-prem clusters AND managed cloud clusters under one consistent tooling layer, a genuinely real answer to "how do we manage a truly hybrid fleet consistently" beyond just the on-prem case this Part focuses on. --- ## Cluster API In Depth: ClusterClass and Fleet Templating Beyond the single `Cluster` object shown above, CAPI's real production value shows up at fleet scale — worth understanding the templating mechanism that makes managing dozens of similar clusters genuinely tractable. ```mermaid graph TD ClusterClass["ClusterClass: a REUSABLE
TEMPLATE defining a
cluster 'shape' - control
plane config, worker node
pool config, networking
defaults - defined ONCE"] --> Cluster1["Cluster: dev-team-a
(references the
ClusterClass, overrides
just a FEW values -
e.g. node count)"] ClusterClass --> Cluster2["Cluster: dev-team-b
(same ClusterClass,
different overrides)"] ClusterClass --> Cluster3["Cluster: staging
(same ClusterClass,
different overrides)"] ``` ```yaml apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: dev-team-a spec: topology: class: standard-cluster-class # references a ClusterClass, defined once version: v1.31.0 controlPlane: replicas: 3 workers: machineDeployments: - class: default-worker name: md-0 replicas: 5 ``` **Why `ClusterClass` is worth knowing as a genuinely significant capability, directly extending the "same idea, one level up" framing already used for CAPI itself:** this is the exact same relationship a Deployment's pod template has to its individual Pods (Part 2) — define the shape ONCE, in a `ClusterClass`, and every actual `Cluster` object becomes a lightweight reference plus a small set of overrides, rather than a fully-repeated, independently-maintained cluster specification. **The direct, practical payoff, worth stating explicitly:** rolling out a Kubernetes version upgrade or a control-plane configuration change across dozens of clusters becomes a single `ClusterClass` edit, propagated by CAPI's own reconciliation loops to every referencing cluster — rather than a dozens-of-times-repeated manual change, directly extending the "fleet consistency through templating" theme. `ClusterClass` and Rancher Fleet, covered later in this Part, are genuinely complementary, not competing — `ClusterClass` templates the CLUSTERS themselves, while Fleet templates what gets DEPLOYED onto already-existing clusters; a mature multi-cluster platform commonly uses both together. **A genuinely real production use case worth citing:** a platform team offering "Kubernetes as a service" internally to many application teams, where every team's cluster should follow the same organizational baseline (security policies, monitoring agents, networking defaults) but with legitimate per-team sizing differences — `ClusterClass` is precisely the mechanism that keeps that baseline consistent and centrally upgradable, without each team's cluster drifting independently over time. --- ## Rancher — Multi-Cluster Management **Rancher** (also from SUSE, the same organization behind k3s) addresses a different, related problem: **managing MANY clusters — potentially a mix of on-prem, managed cloud, and edge clusters — from one unified control plane and UI.** ```mermaid graph TD Rancher["Rancher Management
Server"] --> C1["On-prem cluster
(kubeadm or k3s)"] Rancher --> C2["EKS cluster
(Part 5)"] Rancher --> C3["AKS cluster
(Part 5)"] Rancher --> C4["Edge cluster
(k3s)"] ``` **Why this matters practically, worth stating explicitly: real organizations, especially larger ones, rarely run just ONE Kubernetes cluster, on just ONE platform — Rancher's value proposition is providing a single, consistent RBAC, policy, and observability layer across a genuinely heterogeneous fleet, rather than requiring separate tooling and separate logins per cluster/provider.** This is a distinct, complementary concern from CAPI's cluster-lifecycle focus — some organizations use both together. Rancher itself runs AS a Kubernetes application (typically on its own small "local" cluster), managing every other cluster in the fleet through standard Kubernetes API calls against each — no proprietary agent-based protocol, just the same API surface covered throughout this entire series. **Rancher Fleet, worth knowing as Rancher's own answer to "how do I deploy the SAME application consistently across every cluster in this fleet":** ```mermaid graph TD GitRepo["Git repo: application
manifests / Helm charts"] --> Fleet["Fleet (Rancher's built-in
GitOps engine)"] Fleet --> C1["Cluster 1: reconciled to
match Git"] Fleet --> C2["Cluster 2: reconciled to
match Git"] Fleet --> C3["Cluster 3: reconciled to
match Git"] ``` ```yaml # A Fleet GitRepo object - watches a Git repo, deploys its # contents to every cluster matching the target selector apiVersion: fleet.cattle.io/v1alpha1 kind: GitRepo metadata: name: platform-baseline namespace: fleet-default spec: repo: https://github.com/example/platform-baseline targets: - clusterSelector: matchLabels: {env: production} ``` **Why Fleet matters concretely, directly extending the GitOps discussion referenced throughout this course (fully covered in the Automation, CI/CD & GitOps series):** it applies the exact same "Git as the single source of truth, continuously reconciled" model already covered for single-cluster GitOps (ArgoCD/Flux) but natively fans it out across an entire Rancher-managed fleet using cluster label selectors — a genuinely practical mechanism for keeping a consistent baseline (monitoring agents, security policies, platform-team-owned components) applied uniformly across every cluster in a heterogeneous fleet, without manually repeating the deployment per cluster. A team already standardized on ArgoCD for single-cluster GitOps doesn't necessarily need to switch to Fleet — ArgoCD's own `ApplicationSet` resource solves a genuinely similar multi-cluster fan-out problem; Fleet's distinctive advantage is its native, built-in integration specifically with Rancher's own cluster inventory and RBAC model. --- ## Talos Linux — The Immutable, API-Only OS A genuinely modern, distinctive approach worth knowing about — **Talos Linux** is a purpose-built operating system specifically designed to run Kubernetes, with one radical, deliberate design choice: **there is no SSH access at all.** ```mermaid graph TD Traditional["Traditional node OS
(Ubuntu, etc.): SSH access,
a general-purpose package
manager, a large attack
surface of tools an
attacker COULD use if
they gain access"] --> TraditionalNote["Familiar, but a genuinely
large attack surface —
directly connects to the
DevSecOps series'
minimal-image principle,
now applied at the OS level"] Talos["Talos Linux: NO SSH,
NO shell, managed
ENTIRELY through a
declarative API and a
dedicated CLI (talosctl) —
immutable, minimal by
design"] --> TalosNote["Every node operation
(config change, upgrade)
happens declaratively,
API-driven — genuinely
closes an entire category
of attack surface"] ``` ```bash # Talos is configured and managed entirely through its own # API and CLI — never SSH talosctl apply-config --nodes 192.168.1.10 --file worker.yaml talosctl upgrade --nodes 192.168.1.10 --image ghcr.io/siderolabs/installer:v1.6.0 ``` **Why this is worth citing as a genuinely forward-looking, distinctive example, directly extending the "minimal attack surface" principle from the DevSecOps series' distroless container discussion to the HOST OS level itself: "Talos applies exactly the same minimal-attack-surface philosophy as distroless container images — but to the node's OPERATING SYSTEM, not just the container. No SSH, no shell, no general-purpose package manager means an attacker who somehow gains access has dramatically less to actually DO, and every legitimate operation happens through an auditable, declarative API instead of ad hoc shell commands."** Talos's own cluster bootstrap process (`talosctl bootstrap`) is conceptually the same declarative HA pattern already covered for kubeadm — control plane nodes behind a stable endpoint, an explicit etcd topology — just expressed through Talos's own API instead of `kubeadm init`/`join`. **A fuller node-OS comparison, worth having as a single reference — Talos isn't the only Kubernetes-optimized OS choice:** | OS | Philosophy | SSH access | Best fit | |---|---|---|---| | Ubuntu/RHEL (general-purpose) | Traditional, familiar, general-purpose package manager | Yes | Teams wanting maximum familiarity and flexibility, willing to own more attack surface | | Flatcar Container Linux (CoreOS's community successor) | Immutable root filesystem, atomic updates, container-focused | Yes (but minimal OS footprint) | A middle ground — immutable/atomic update model, but still traditional SSH-based operations | | Bottlerocket (AWS, but usable outside EKS too) | Minimal, container-optimized, no shell by default | No (a very limited "admin container" instead) | Teams wanting a minimal, AWS-aligned OS even on self-managed or hybrid setups | | Talos Linux | Fully immutable, API-only, zero shell/SSH ever | No, none at all | Maximum attack-surface reduction, teams comfortable with fully API-driven operations | **Why this spectrum is worth presenting as a genuine gradient, not a binary "traditional vs. Talos" choice:** Flatcar and Bottlerocket occupy real, meaningful middle ground — immutable, atomic-update philosophy without going as far as Talos's complete elimination of interactive access. **A strong, senior-level answer names the specific point on this spectrum that fits a given team's actual operational maturity and security requirements**, rather than treating "most locked-down" as automatically "best" — a team with no existing API-driven-operations tooling or runbooks may get more real value from Flatcar's more familiar operational model than from jumping straight to Talos's fully API-only extreme. This exact "match the tool to actual operational maturity, not to the most extreme option" principle recurs throughout this Part — the same reasoning applies to choosing between Ceph and Longhorn, and between kubeadm and a batteries-included distribution like k3s. That principle is the real thread tying this entire Part together — worth restating one final time in the closing summary below. --- ## The Bare-Metal-Specific Gaps Regardless of which provisioning tool is used, running Kubernetes on genuine bare metal (not virtualized cloud infrastructure) surfaces real gaps that managed cloud Kubernetes (Part 5) simply doesn't have to think about, because the cloud provider already solved them. ```mermaid graph TD Gaps["Bare-Metal-Specific
Gaps"] --> G1["LoadBalancer Services
(Part 3): cloud providers
automatically provision a
REAL load balancer — on
bare metal, NOTHING does
this automatically"] Gaps --> G2["Persistent storage (Part 3):
no cloud CSI driver
automatically provisioning
disks — needs its OWN
solution"] Gaps --> G3["Node provisioning itself:
no 'just call an API to get
a new VM' — PHYSICAL
machines, PXE boot, or at
minimum a virtualization
layer"] ``` --- ## Bare-Metal Node Provisioning: PXE Boot and Tinkerbell The third gap named above — actually getting an operating system installed onto a physical machine in the first place — deserves its own treatment, since it's the one gap that has no cloud-API equivalent at all to conceptually borrow from. ```mermaid sequenceDiagram participant Machine as Bare Metal Machine
(freshly racked, blank disk) participant DHCP as DHCP/PXE Server participant TFTP as TFTP/HTTP Boot Server participant Installer as OS Installer Machine->>DHCP: Powers on, network boot
(PXE) requests an IP +
boot instructions DHCP-->>Machine: IP address + location of
the boot loader Machine->>TFTP: Downloads the boot loader
and installer image TFTP-->>Machine: Kernel + installer,
delivered over the network Machine->>Installer: Installer runs, partitions
disk, installs the OS
(Ubuntu, Talos, etc.) Installer->>Machine: Reboots into the freshly-
installed OS - now ready
for kubeadm/k3s/Talos
bootstrapping ``` **Why PXE (Preboot Execution Environment) is worth knowing by name, precisely: it's the standard, decades-old network-boot protocol that makes "zero-touch" bare-metal provisioning possible at all** — a machine with no OS installed yet can still boot, entirely over the network, without anyone physically inserting install media. This is the genuine, low-level mechanism underneath every "automated bare-metal provisioning" tool, including the more modern ones covered next. **Tinkerbell (a CNCF project), worth knowing as the modern, Kubernetes-native evolution of this exact PXE workflow:** it wraps the PXE boot process, OS installation, and even ongoing hardware lifecycle actions (BMC/IPMI power control, firmware updates) into declarative, Kubernetes-style CRDs and workflows — directly extending the CRD/controller pattern from Part 4 to the literal physical-hardware-provisioning layer. A `Workflow` custom resource describes the exact sequence of actions (wipe disk, PXE boot, run installer, reboot) the same declarative way a Deployment describes desired pod state, and Tinkerbell's controllers execute and track it to completion. ```yaml # A genuinely simplified illustration of a Tinkerbell Hardware object - # registering a specific physical machine by its MAC address apiVersion: tinkerbell.org/v1alpha1 kind: Hardware metadata: name: bare-metal-node-1 spec: interfaces: - dhcp: mac: "aa:bb:cc:dd:ee:ff" ip: {address: "192.168.1.50"} netboot: allowPXE: true ``` **Why this matters as a real, worth-knowing capability, connecting directly to Cluster API's own bare-metal infrastructure providers:** Cluster API's `Metal3` provider builds on exactly this kind of bare-metal-provisioning automation (historically via `Ironic`, an OpenStack-originated bare-metal provisioning service, with Tinkerbell as a newer, increasingly popular alternative) — meaning the full stack, from "empty rack of servers" to "CAPI-managed fleet of Kubernetes clusters," genuinely can be declarative and automated end to end, not just from the point a base OS is already installed. This closes the loop back to Cluster API's `ClusterClass` templating covered earlier — the same declarative fleet-management layer can, with Metal3, extend all the way down to the physical hardware itself, not just the Kubernetes objects running on top of already-provisioned machines. BMC/IPMI power control, mentioned above, is worth naming explicitly as the concrete mechanism behind this: it's the standard out-of-band management interface most server-grade hardware exposes, letting Tinkerbell (or any equivalent) power-cycle, reboot, or reimage a machine remotely, without any operating system running on it at all yet. --- ## MetalLB — Solving LoadBalancer Services On-Prem Directly extending the Service types discussion from Part 3 — a genuinely important, concrete gap-filler worth knowing by name. ```mermaid sequenceDiagram participant Svc as LoadBalancer Service participant MetalLB participant Network as Local Network Svc->>MetalLB: New Service of type
LoadBalancer created MetalLB->>MetalLB: Picks an available IP from
a CONFIGURED pool
(e.g. 192.168.1.200-220) MetalLB->>Network: Announces that IP via
ARP (Layer 2 mode) OR
BGP (Layer 3 mode) —
the network topics from
the Linux & Networking
Fundamentals series, Part 2 Network-->>Svc: Traffic to that IP now
reaches the Service ``` **Why this matters, worth stating explicitly: on AWS/Azure/GCP (Part 5), creating a `type: LoadBalancer` Service automatically provisions a REAL cloud load balancer behind the scenes — on bare metal, there's no equivalent cloud API to call, so without MetalLB (or an equivalent), a LoadBalancer Service would simply sit forever in a `Pending` state, with no external IP ever assigned.** MetalLB fills exactly this gap, using genuine networking protocols (ARP or BGP) already covered in the Linux & Networking Fundamentals series, rather than any cloud-specific API. **Worth naming a real, common alternative to MetalLB, even briefly: kube-vip** — solves a genuinely overlapping problem (virtual IP announcement for both `LoadBalancer` Services and, distinctively, the control-plane HA endpoint itself, as an alternative to a dedicated external load balancer for the `--control-plane-endpoint` setup covered earlier in this Part) — worth knowing it exists as a second real option, not just MetalLB by default. --- ## MetalLB Layer 2 vs BGP Mode, In Depth The two modes MetalLB supports solve the "how does the network actually learn to route to this IP" problem in genuinely different ways, each with real, distinct tradeoffs worth knowing precisely. ```mermaid graph TD L2["Layer 2 mode: ONE node
at a time announces the
Service IP via ARP/NDP -
that specific node becomes
the SINGLE entry point for
ALL traffic to that IP"] --> L2Note["Simple, works on ANY
network with no special
router configuration - but
ALL traffic funnels through
ONE node, a real, single-
node bottleneck and failover
delay on that node's failure"] BGP["BGP mode: EVERY node
capable of serving the
Service ANNOUNCES a route
to it via the BGP routing
protocol, to REAL upstream
routers"] --> BGPNote["TRUE load balancing across
MULTIPLE nodes simultaneously
(ECMP - Equal-Cost Multi-
Path routing) - but requires
a BGP-capable upstream
router and real network
team coordination"] ``` ```yaml # Layer 2 mode configuration apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: default-pool spec: addresses: - 192.168.1.200-192.168.1.220 --- apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: l2-advertise spec: ipAddressPools: [default-pool] ``` ```yaml # BGP mode configuration - requires a real upstream BGP peer apiVersion: metallb.io/v1beta1 kind: BGPPeer metadata: name: upstream-router spec: myASN: 64500 peerASN: 64501 peerAddress: 192.168.1.1 --- apiVersion: metallb.io/v1beta1 kind: BGPAdvertisement metadata: name: bgp-advertise spec: ipAddressPools: [default-pool] ``` **Why the choice between them is a genuinely real, concrete tradeoff worth stating precisely, not just "BGP is more advanced so pick it":** Layer 2 mode's single-node-announces-the-IP behavior means a failover (when the announcing node fails and another node takes over the announcement) is genuinely fast but still involves a brief interruption and, critically, **all traffic to that Service funnels through one node's network capacity, even with many healthy backend pods spread across many nodes** — a real, concrete bottleneck at meaningful traffic volume. BGP mode's ECMP-based multi-node announcement genuinely load-balances at the network layer itself, but requires actual coordination with whoever manages the physical network's routers — **a real organizational dependency, not just a Kubernetes-side configuration change**, which is exactly why Layer 2 mode remains the more commonly deployed choice for smaller, simpler on-prem environments where that router-level coordination isn't readily available. --- ## Bare-Metal Storage The same "no automatic cloud provisioning" gap applies to storage — extending the CSI/dynamic-provisioning discussion from Part 3. ```mermaid graph TD Options["Bare-Metal Storage
Options"] --> Local["local-path-provisioner:
simple, uses LOCAL disk
on each node — fine for
dev/test, but NO
redundancy if that
specific node dies"] Options --> Rook["Rook (managing Ceph):
a REAL distributed storage
system running INSIDE
Kubernetes itself, providing
genuine replication and
a proper CSI driver"] ``` **Why Rook/Ceph is worth knowing specifically, and it directly ties back to the replication concepts from the Databases & Storage Reliability series: it brings genuine, distributed, replicated storage — conceptually similar to how a cloud provider's EBS/managed disks work under the hood — to a bare-metal environment, running as Kubernetes-native Operators (Part 4's pattern) rather than requiring a separate, external storage appliance.** **A real, worth-knowing alternative to Rook/Ceph: Longhorn (a CNCF project, originally by Rancher/SUSE)** — genuinely simpler to operate than Ceph, trading some of Ceph's raw feature breadth and massive-scale proven track record for a meaningfully gentler operational learning curve. ```mermaid graph TD Ceph["Rook/Ceph: battle-tested
at MASSIVE scale, extremely
feature-rich (object,
block, AND file storage),
but a genuinely real
operational learning
curve"] --> CephUse["Right for large,
storage-intensive
deployments where the
operational investment
is justified"] Longhorn["Longhorn: simpler,
Kubernetes-native block
storage specifically,
with an approachable web
UI and a MUCH gentler
learning curve"] --> LonghornUse["Right for small-to-
medium clusters wanting
genuine block-storage
redundancy without Ceph's
full operational
complexity"] ``` ```bash # Installing Longhorn is genuinely simpler than standing up Ceph kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.0/deploy/longhorn.yaml ``` **Why this choice is worth presenting as a genuine, real tradeoff, not "always pick the more feature-rich option":** Ceph's operational complexity is real and non-trivial — proper Ceph operation genuinely benefits from dedicated storage expertise, while Longhorn deliberately trades some of that raw capability for approachability, making it a legitimately better fit for smaller teams and clusters where standing up and maintaining a full Ceph deployment would be disproportionate operational overhead relative to the actual storage needs. Both integrate as standard CSI drivers (Part 3) — the choice between them changes operational complexity, not how application manifests actually consume the storage, since PVCs/StorageClasses look identical from the application's point of view regardless of which backend fulfills them. --- ## Air-Gapped Kubernetes A genuinely specialized but real, important scenario worth knowing about — some regulated/defense environments require **zero internet connectivity**, at all, ever. ```mermaid graph TD Problem["Standard Kubernetes
setup assumes internet
access: pulling container
images from public
registries, downloading
Helm charts, etc."] --> AirGap["Air-gapped environments:
NONE of that is possible —
EVERYTHING must be
pre-staged"] AirGap --> Solution["Solution: a PRIVATE,
internal container
registry, mirroring EVERY
required image in advance,
with cluster components
explicitly configured to
pull ONLY from it"] ``` **Why this is worth being aware of, even briefly, tying to the supply chain security discussion from the DevSecOps series (Part 5): an air-gapped cluster's entire software supply chain has to be deliberately, manually curated and mirrored — directly connecting to the SBOM and artifact-signing concepts from that earlier tutorial, since knowing EXACTLY what's running (and verifying it hasn't been tampered with before it's mirrored in) matters even more when there's no ongoing internet-based verification possible after the fact.** Air-gapped requirements and standard bare-metal deployments aren't mutually exclusive — a fully air-gapped environment still needs every other tool covered in this Part (kubeadm/k3s, MetalLB, redundant storage), just with every single image and chart pre-staged through the mirroring process covered next, rather than pulled live. --- ## Air-Gapped Cluster Setup, Step by Step A concrete, worked walkthrough — genuinely useful to be able to narrate end to end, since "air-gapped Kubernetes" is often understood only abstractly. ```mermaid flowchart TD Prep["ON A CONNECTED machine:
identify EVERY required
image (Kubernetes core
components, CNI, CSI,
every application image)"] --> Pull["Pull and TAR-export every
image, plus every Helm
chart and binary needed
(kubeadm, kubelet,
kubectl, crictl)"] Pull --> Transfer["Physically transfer the
bundle across the air gap
(removable media - the
ONLY path across the
boundary)"] Transfer --> Load["INSIDE the air-gapped
network: load images into
a PRIVATE registry running
within the boundary"] Load --> Configure["Configure kubelet/
containerd on every node to
pull ONLY from that private
registry (never the public
internet)"] Configure --> Bootstrap["Bootstrap the cluster
normally (kubeadm/k3s/
Talos) - referencing the
private registry for every
image pull"] ``` > **A concrete, worked example — the actual commands, on the connected side first:** > ```bash > # Identify and pull every required image > images=(registry.k8s.io/kube-apiserver:v1.31.0 registry.k8s.io/etcd:3.5.15-0 docker.io/flannel/flannel:v0.25.5) > for img in "${images[@]}"; do > docker pull "$img" > docker save "$img" -o "$(basename "$img" | tr ':' '_').tar" > done > ``` > > **Then, inside the air-gapped network, after physical transfer:** > ```bash > # Load each image and push to the internal private registry > for tar in *.tar; do > docker load -i "$tar" > done > docker tag registry.k8s.io/kube-apiserver:v1.31.0 internal-registry.local/kube-apiserver:v1.31.0 > docker push internal-registry.local/kube-apiserver:v1.31.0 > > # Bootstrap kubeadm, pointing explicitly at the internal registry > sudo kubeadm init --image-repository internal-registry.local --pod-network-cidr=10.244.0.0/16 > ``` **Why `--image-repository` is worth knowing as the specific, concrete kubeadm flag that makes this possible:** without it, kubeadm defaults to pulling every control-plane component image from the public `registry.k8s.io`, which is completely unreachable inside an air-gapped network by definition — explicitly overriding it to point at the internal mirror is the exact mechanism that makes an air-gapped `kubeadm init` succeed at all. The same pattern applies identically to k3s (`--system-default-registry`) and to any Helm chart's `image.repository` value — every tool covered in this Part that pulls container images needs its own explicit override, following the same underlying principle. Talos, MetalLB, Rook/Ceph, and Cluster API's own controller images all follow this identical pattern too — the specific override flag or values-file key differs per tool, but the underlying requirement is universal across this entire Part's toolset. **A genuinely real, easy-to-overlook detail worth naming explicitly: EVERY image reference, everywhere, needs updating, not just the control plane's.** CNI manifests, CSI driver manifests, Helm chart `values.yaml` image repositories, and any application's own Deployment specs all typically hardcode a public registry path by default — each one needs to be explicitly retargeted at the internal mirror, a genuinely tedious but non-negotiable step, since a single missed reference means that specific component silently fails to pull its image inside the air-gapped environment. --- ## Node Maintenance and Hardware Failure On-Prem A final, genuinely important gap worth naming explicitly: cloud providers automatically replace a failed VM's underlying hardware transparently — on bare metal, a failed physical machine is a real, physical problem someone has to actually go fix. ```mermaid sequenceDiagram participant Node as Failing/Maintenance Node participant Admin as Cluster Admin participant Sched as Scheduler Admin->>Node: kubectl cordon node-5
(mark unschedulable -
Part 1's node conditions) Admin->>Node: kubectl drain node-5
--ignore-daemonsets
(gracefully evict pods,
respecting PodDisruptionBudgets) Sched->>Sched: Reschedules evicted pods
onto OTHER healthy nodes Admin->>Node: Physically service/replace
the hardware Admin->>Node: kubectl uncordon node-5
(once healthy again) ``` **Why `cordon` and `drain` are worth knowing as two genuinely distinct, sequential steps, not one action, precisely because bare-metal maintenance is manual and deliberate in a way cloud node replacement isn't:** `cordon` alone only prevents NEW pods from scheduling onto the node — existing pods keep running, untouched. `drain` (which implicitly cordons first) actively evicts existing pods, respecting PodDisruptionBudgets (Part 2) the same way a voluntary disruption would on any cluster. **This two-step, explicit, human-triggered process is precisely what a cloud provider's automated node replacement (Part 5's cordon-and-drain node-upgrade discussion) does FOR you automatically** — on bare metal, an admin performs the equivalent sequence manually, for a real, physical hardware event rather than a scheduled version upgrade. ```bash # The full, real bare-metal hardware-failure sequence kubectl cordon node-5 kubectl drain node-5 --ignore-daemonsets --delete-emptydir-data --force # ... physically service the hardware, or replace it entirely ... kubectl uncordon node-5 # only once genuinely healthy again ``` **`--delete-emptydir-data` and `--force`, worth knowing precisely why they're sometimes needed:** `drain` refuses to proceed by default if a pod uses `emptyDir` storage (Part 3 — data that would be permanently lost on eviction) or isn't managed by a controller at all — these flags are explicit, deliberate overrides acknowledging that data loss or an unmanaged pod's termination is genuinely expected and acceptable for this specific maintenance operation, not a default to reach for casually. **A genuinely important interaction worth naming: `drain` respects PodDisruptionBudgets and can BLOCK indefinitely if a PDB's minimum-availability requirement can't currently be satisfied.** This is correct, intended behavior, not a bug — if draining a node would violate a critical service's PDB, `drain` waits rather than proceeding, giving an operator the chance to notice and address the underlying capacity issue before forcing through a disruption that would breach the availability guarantee the PDB was specifically created to protect. --- ## A Full Worked Example: Building a Complete Bare-Metal Production Stack Tying every tool covered in this Part together into one coherent, end-to-end architecture — genuinely valuable to be able to narrate as a complete story, not just recite the individual tools. ```mermaid graph TD Hardware["Physical servers, racked"] --> Tinkerbell["Tinkerbell: PXE-boots and
installs the base OS on
every machine"] Tinkerbell --> Talos["Talos Linux: the installed
OS itself - immutable,
API-only, minimal attack
surface"] Talos --> HA["kubeadm-equivalent HA
bootstrap (Talos has its
OWN declarative cluster
bootstrap, conceptually
identical to kubeadm's
--control-plane-endpoint
pattern) - 3 control plane
nodes behind a load
balancer"] HA --> MetalLB["MetalLB (BGP mode, given
genuine production traffic
volume): provides working
LoadBalancer Services"] HA --> Rook["Rook/Ceph: provides
genuinely redundant,
distributed persistent
storage"] HA --> CAPI["Cluster API + ClusterClass
(if managing MULTIPLE such
clusters): templated,
declarative fleet lifecycle"] CAPI --> Fleet["Rancher Fleet: GitOps-based
consistent application/
baseline deployment across
the fleet"] ``` **Why this specific combination is worth presenting as a coherent, real architecture, not just a list of unrelated tools:** every tool here solves one, and only one, genuinely distinct gap identified earlier in this Part — Tinkerbell solves node provisioning, Talos solves node OS security, the HA control plane pattern solves control-plane availability, MetalLB solves the LoadBalancer Service gap, Rook/Ceph solves the storage gap, and Cluster API + Fleet solve fleet-scale lifecycle and consistency once there's more than one such cluster. **A strong, senior-level closing line worth stating explicitly:** "Bare-metal Kubernetes isn't one tool replacing a cloud provider — it's a deliberate assembly of purpose-built tools, each closing exactly one gap a managed cloud provider would otherwise close for you automatically, and understanding WHICH gap each tool closes is what separates knowing the tool names from actually being able to design the system." Not every deployment needs every layer of this stack — a small internal-tools cluster might reasonably skip Tinkerbell (manually image a handful of machines) and Cluster API (manage the one cluster directly), while still genuinely needing MetalLB and some form of redundant storage. Scale the assembly to the actual requirement, not to this diagram's full breadth by default. --- ## A Full Worked Comparison ```mermaid flowchart TD Start{"What's the PRIMARY
goal?"} --> Q1{"Learning/understanding
K8s internals deeply?"} Q1 -->|Yes| KubeAdm["kubeadm — builds real
intuition for Part 1's
components"] Q1 -->|No| Q2{"Edge/IoT/resource-
constrained, or a fast
dev environment?"} Q2 -->|Yes| K3s["k3s / k0s"] Q2 -->|No| Q3{"Managing MANY clusters
across mixed
environments?"} Q3 -->|Yes| RancherCAPI["Rancher (unified UI/RBAC)
and/or Cluster API
(declarative lifecycle)"] Q3 -->|No| Q4{"Security/attack-surface
minimization is the TOP
priority?"} Q4 -->|Yes| Talos["Talos Linux"] ``` | Tool | Solves | Not Designed For | |---|---|---| | **kubeadm** | Bootstrapping a single cluster's control plane, from first principles | Ongoing lifecycle management, multi-cluster, day-2 ops | | **k3s / k0s** | Lightweight, fast, opinionated single-cluster setup | Massive-scale, highly customized control plane configuration | | **Cluster API** | Declarative lifecycle management of MANY clusters, across providers | A quick, one-off single cluster (real overhead to set up the management cluster itself) | | **Rancher** | Unified multi-cluster UI, RBAC, and policy across a heterogeneous fleet | Being a lightweight, minimal-footprint single-cluster tool | | **Talos Linux** | Minimizing node-level attack surface via an immutable, API-only OS | Teams that need traditional shell/SSH-based debugging workflows | | **Tinkerbell** | Zero-touch, declarative bare-metal node provisioning from an empty machine | Clusters already running on pre-provisioned VMs or existing OS installs | | **MetalLB** | LoadBalancer Services on bare metal | Environments already behind a hardware load balancer with its own IP management | | **Rook/Ceph or Longhorn** | Redundant, distributed persistent storage on bare metal | Workloads that only ever need node-local, non-redundant scratch storage | | **Rancher Fleet** | GitOps-based consistent baseline across an entire multi-cluster fleet | A single-cluster deployment with no fleet-wide consistency need | **A closing, worth-remembering note on this table:** every row solves a genuinely distinct, non-overlapping problem — the real skill in bare-metal Kubernetes architecture isn't memorizing this list, it's correctly mapping a specific organizational requirement (which of these gaps actually matters for THIS deployment) onto the smallest set of tools that closes exactly those gaps, without over-provisioning tooling for problems the deployment doesn't actually have. --- ## Part 6 CLI Cheat Sheet ```bash # kubeadm kubeadm init --control-plane-endpoint "k8s-lb.internal:6443" --upload-certs kubeadm join --token --discovery-token-ca-cert-hash sha256: kubeadm token create --print-join-command # generate a new join command (tokens expire) kubeadm upgrade plan kubeadm upgrade apply v1.31.0 # k3s / k0s / MicroK8s curl -sfL https://get.k3s.io | sh - sudo k3s kubectl get nodes microk8s status --wait-ready microk8s enable dns storage ingress # Cluster API clusterctl init --infrastructure aws kubectl get clusters -A kubectl get machinedeployments -A clusterctl describe cluster my-cluster # MetalLB kubectl get ipaddresspool,l2advertisement,bgpadvertisement -n metallb-system kubectl describe svc | grep -A 3 "LoadBalancer Ingress" # Talos talosctl get members talosctl dashboard talosctl logs kubelet ``` --- ## Common Mistakes | Mistake | Why It's Wrong | Fix | |---|---|---| | Assuming kubeadm alone gives you a production-ready cluster | It deliberately only bootstraps the control plane — CNI, storage, monitoring, and day-2 ops are all left entirely to you | Explicitly plan and implement each of these separately, or use a more opinionated/batteries-included tool if that gap is unwanted | | Creating a `type: LoadBalancer` Service on a bare-metal cluster with no MetalLB (or equivalent) installed | It will sit in `Pending` state forever — there's no cloud API to automatically provision a real load balancer | Install MetalLB (or an equivalent) specifically to fill this gap on bare-metal/on-prem clusters | | Using `local-path-provisioner`-style local storage for genuinely critical, must-survive-node-failure data | Provides zero redundancy — losing that one node means losing that data entirely | Use a genuinely distributed storage system (like Rook/Ceph) for data that needs real durability guarantees | | Treating self-managed Kubernetes as strictly cheaper than managed, without accounting for real operational staffing cost | The engineering time spent on control-plane operations, upgrades, and troubleshooting is a real, ongoing cost, not free | Weigh TOTAL cost (infrastructure + genuine engineering time) when comparing self-managed against managed options | | Treating the cost crossover point as a universal, fixed rule ("on-prem is always/never cheaper") | The real crossover genuinely depends on scale, workload predictability, and existing infrastructure investment — it isn't one fixed number | Model the actual workload's specific numbers rather than applying a generic cloud-vs-on-prem heuristic | | Assuming an air-gapped cluster can use standard setup instructions unmodified | Standard tooling assumes internet access for pulling images/charts, which simply isn't available | Deliberately pre-stage a mirrored private registry and explicitly configure every component to use it | | Running a self-managed HA control plane without an external load balancer in front of it | Clients hardcoding one control plane node's IP defeats the purpose of having multiple replicas, and that node's failure breaks cluster access entirely | Always place a load balancer (HAProxy, keepalived, or hardware) in front of a multi-node control plane, referenced via `--control-plane-endpoint` | | Choosing MetalLB's Layer 2 mode for a high-traffic-volume production Service without realizing all traffic funnels through one node | A real, concrete network bottleneck at meaningful traffic volume, even with many healthy backend pods elsewhere | Use BGP mode with genuine upstream router coordination when true multi-node load distribution is required | | Missing an image reference during air-gapped migration (a Helm chart's `values.yaml`, a CNI manifest) | That specific component silently fails to pull its image, since the public registry is unreachable by definition | Audit every manifest, chart, and application spec for hardcoded public registry references, not just the control plane | | Standing up a full Rook/Ceph deployment for a small cluster's modest storage needs | Disproportionate operational overhead relative to the actual requirement — Ceph genuinely benefits from dedicated storage expertise | Consider Longhorn for smaller clusters wanting redundant block storage without Ceph's full operational complexity | | Jumping straight to Talos Linux without existing API-driven-operations tooling or runbooks | A real learning-curve mismatch — no fallback to familiar SSH-based debugging during an actual incident | Consider a middle-ground OS (Flatcar, Bottlerocket) if the team's operational maturity doesn't yet support fully API-only operations | | Physically powering off a bare-metal node for maintenance without cordoning and draining it first | Pods get abruptly killed with no graceful eviction, and PodDisruptionBudgets are bypassed entirely | Always `cordon` then `drain` before any physical hardware intervention, `uncordon` only once genuinely healthy | | Using `--force` on `kubectl drain` as a routine default | Bypasses real safety checks meant to flag unmanaged pods or genuine data-loss risk from `emptyDir` volumes | Reserve `--force`/`--delete-emptydir-data` for maintenance where that specific risk is understood and accepted | --- ## Worked Practice Problems **Problem 1:** A team runs `kubeadm init` successfully, but `kubectl get nodes` shows the control plane node stuck in `NotReady` status. What's the most likely cause, based on what kubeadm deliberately doesn't do? *Answer:* kubeadm deliberately doesn't install a CNI plugin (Part 3) as part of its bootstrapping process — without one, pod networking isn't functional, and the node correctly reports `NotReady` because a required cluster component (networking) isn't in place yet. The fix is applying a CNI plugin manifest (Flannel, Calico, Cilium, or another choice) immediately after `kubeadm init`, exactly as shown in this tutorial's worked example — this is expected, documented behavior, not a bug. **Problem 2:** An organization runs 15 Kubernetes clusters across on-prem data centers, EKS, and AKS, and is struggling with inconsistent RBAC policies and no unified view of cluster health. Would you recommend Cluster API, Rancher, or both, and why? *Answer:* Primarily Rancher — the core problem described (inconsistent RBAC across a heterogeneous fleet, no unified observability) is specifically Rancher's value proposition: one management layer providing consistent policy and visibility across mixed on-prem/cloud clusters. Cluster API would additionally be worth considering if the team ALSO struggles with the actual lifecycle management (creating/scaling/upgrading) of the on-prem clusters specifically in a declarative, automated way — the two tools solve genuinely different, complementary problems (Rancher: ongoing multi-cluster management/policy; CAPI: cluster lifecycle automation), and larger organizations often do use both together. **Problem 3:** A security team is evaluating node operating systems for a new, security-sensitive Kubernetes deployment and is comparing a standard Ubuntu-based node setup against Talos Linux. What's the core security argument for Talos, and what real operational tradeoff should be weighed against it? *Answer:* The core argument: Talos eliminates SSH and shell access entirely, managing nodes exclusively through a declarative API — this closes an entire category of attack surface (no shell for an attacker to abuse even after gaining some level of access) and forces every configuration change through an auditable, declarative path, directly extending the same minimal-attack-surface philosophy already covered for distroless container images in the DevSecOps series, now applied at the host OS level. The real tradeoff to weigh: teams accustomed to traditional SSH-based debugging workflows (checking logs directly on a node, running ad hoc diagnostic commands) need to adjust to Talos's API-only operational model, which can have a real learning curve and may require adapting existing runbooks (Incident Management series) that assume direct node shell access. **Problem 4:** A self-managed cluster's HA control plane loses 2 of its 3 stacked-etcd control plane nodes simultaneously during a rack power event. What happens, and what topology choice, made at setup time, could have changed the outcome? *Answer:* With stacked etcd (each control plane node running its own local etcd member), losing 2 of 3 nodes simultaneously means losing 2 of 3 etcd members at the same time — dropping below the majority quorum (Part 1) required for etcd to accept writes at all, meaning the entire cluster loses the ability to create or update any object until at least one more etcd member becomes available. If the deployment had used an external, dedicated etcd cluster (an alternative topology choice made at setup) instead of stacked etcd, the control-plane-node failures and etcd-member failures would have been genuinely independent events — the external etcd cluster, on its own separate machines, would likely have survived the same rack event untouched, keeping the cluster writable even while 2 of 3 control plane API servers were down. This is precisely the "decoupled blast radius vs. operational simplicity" tradeoff between stacked and external etcd topologies. **Problem 5:** A team provisioning 200 bare-metal servers for a new Kubernetes fleet asks whether they need to manually image each machine before running kubeadm. What's the better, more automated answer? *Answer:* No — this is exactly the gap PXE boot and tools like Tinkerbell exist to close. Rather than manually imaging 200 machines individually, a PXE-based workflow lets each machine network-boot into an automated installer the moment it's racked and powered on, with Tinkerbell (or an equivalent) orchestrating the OS installation declaratively via Kubernetes-style CRDs and Workflows — the same reconciliation-loop pattern from Part 1 and Part 4's Operator pattern, applied to physical hardware provisioning. Combined with Cluster API's Metal3 provider, the entire pipeline — from "200 empty racked servers" to "200 nodes bootstrapped and joined to Kubernetes clusters" — can be genuinely declarative and automated end to end, rather than requiring manual, per-machine imaging work. **Problem 6:** A small platform team (3 engineers) with no prior distributed-storage experience needs redundant persistent storage for a 5-node on-prem cluster. They're debating between Rook/Ceph and Longhorn. What would you recommend, and why? *Answer:* Longhorn, specifically because of the team's stated lack of distributed-storage expertise and small scale — Ceph is genuinely powerful and battle-tested at massive scale, but real, correct Ceph operation (monitoring OSD health, understanding placement groups, tuning replication) benefits substantially from dedicated storage expertise the team doesn't have. Longhorn's deliberately simpler operational model, approachable web UI, and Kubernetes-native design make it a legitimately better fit for a small team's actual needs at this scale — the "more capable" option isn't automatically the right choice when the team's operational capacity to run it correctly is the actual limiting factor, not the storage system's raw feature set. **Problem 7:** A team standardizes on Rancher for multi-cluster management and wants every cluster in their fleet to automatically receive the same baseline monitoring agent and NetworkPolicy defaults, without manually applying them to each new cluster. What Rancher-native mechanism solves this? *Answer:* Rancher Fleet, using a `GitRepo` object with a `clusterSelector` matching the target clusters (e.g., by an environment label). The baseline manifests (monitoring agent DaemonSet, default-deny NetworkPolicy per Part 3) live in a Git repository, and Fleet continuously reconciles every matching cluster in the fleet to match that Git state — new clusters that pick up the matching label automatically receive the baseline the next time Fleet reconciles, with zero manual per-cluster application needed. This is precisely the GitOps-at-fleet-scale pattern Fleet is built for, directly extending the single-cluster GitOps model already covered elsewhere in this course to Rancher's multi-cluster context specifically. **Problem 8:** An on-call engineer, responding to a hardware alert, directly powers off a misbehaving bare-metal node without running `kubectl drain` first, reasoning "it's about to be forcibly removed anyway, why bother." What actually goes wrong, and what should have happened instead? *Answer:* Powering off the node without draining first means every pod running on it is abruptly terminated with no graceful shutdown at all — no SIGTERM, no chance to finish in-flight requests or close connections cleanly (the same graceful-shutdown discipline from Part 1's kubelet discussion), and critically, **PodDisruptionBudgets are bypassed entirely**, since PDBs only govern voluntary disruptions initiated through the API (like `drain`), not an external hardware power-off the cluster has no way to negotiate with. If multiple replicas of a critical service happened to be concentrated on that one node (a real risk without proper anti-affinity, per Part 2), this could take the service's availability below its PDB's minimum with zero warning. The correct sequence: `kubectl cordon` then `kubectl drain` first, letting the scheduler gracefully reschedule pods onto healthy nodes and giving each pod its normal termination grace period — only after drain completes successfully should the physical hardware actually be powered off. --- ## Summary and What's Next - Organizations self-manage Kubernetes for real, concrete reasons — **data residency, cost at scale, latency, existing infrastructure investment, and air-gapped requirements** — a genuine cost/control tradeoff against Part 5's managed-Kubernetes case, not an outdated approach. - **The cost crossover between cloud and self-managed is genuinely workload- and organization-specific** — never a universal rule — and data residency/latency/air-gap requirements are usually the actual deciding factors in practice, with raw cost as a real but secondary consideration. - **The thread tying this entire Part together: match every tool to the team's actual operational maturity and the deployment's genuine requirements** — not to the most extreme or most feature-rich option available, whether that's choosing an OS, a storage backend, or a bootstrapping tool. - **Cordon, then drain, then physically service the hardware, then uncordon** — the manual bare-metal equivalent of a cloud provider's automated node replacement, and worth having as a memorized sequence, not just a concept. - **`drain` respects PodDisruptionBudgets and can legitimately block** — this is correct, intended behavior signaling a real capacity issue, not something to force through by default. - **kubeadm** is the official bootstrapping tool, deliberately scoped to control-plane setup only — it builds genuine intuition for Part 1's components, but leaves CNI, storage, and day-2 operations entirely to you. - **k3s and k0s** are lightweight, opinionated, batteries-included distributions — ideal for edge/IoT and fast development environments, trading some flexibility for dramatic setup simplicity. - **Cluster API** applies Kubernetes's own declarative reconciliation-loop pattern (Part 1) one level up, to manage the lifecycle of entire clusters across many infrastructure providers. - **Rancher** solves a different, complementary problem: unified RBAC, policy, and observability across a heterogeneous, multi-cluster, multi-provider fleet. - **Talos Linux** represents a genuinely modern, security-forward approach — an immutable, API-only OS with no SSH, extending the DevSecOps series' minimal-attack-surface philosophy to the host OS itself. - Bare-metal deployments surface real gaps managed cloud Kubernetes doesn't have — **MetalLB** fills the LoadBalancer Service gap, and **Rook/Ceph** provides genuinely distributed, redundant storage where cloud CSI drivers would otherwise handle it automatically. - **Air-gapped environments** require deliberately pre-staging every dependency (a mirrored private registry), directly connecting to the supply chain security practices from the DevSecOps series. - **kubeadm HA setups need an external load balancer** in front of multiple control plane nodes, and the **stacked vs. external etcd** topology choice is a real decoupling-vs-simplicity tradeoff worth naming precisely. - **`ClusterClass`** turns Cluster API from a single-cluster declarative tool into genuine fleet templating — one edit propagates to every referencing cluster, mirroring a Deployment template's relationship to its Pods. - **PXE boot and Tinkerbell** close the one gap with no cloud-API equivalent at all — automating bare-metal OS provisioning declaratively, feeding directly into Cluster API's Metal3 provider for a fully end-to-end automated bare-metal fleet. - **MetalLB's Layer 2 mode is simple but funnels traffic through one node; BGP mode gives genuine multi-node load distribution** at the cost of real upstream network team coordination. - **Rancher Fleet** applies the GitOps model — already covered for single clusters elsewhere — natively across an entire fleet via cluster label selectors, keeping baselines consistent without manual per-cluster application. - **Node OS choice is a real spectrum**, not a binary — Ubuntu/RHEL, Flatcar, Bottlerocket, and Talos sit at different points between operational familiarity and attack-surface minimization; pick based on the team's actual operational maturity, not just "most locked-down." - **Longhorn is a genuinely simpler alternative to Rook/Ceph** for smaller clusters — the right storage system depends on the team's actual operational capacity, not just raw feature breadth. - **A complete bare-metal production stack composes Tinkerbell, Talos (or another OS choice), an HA control plane, MetalLB, and Rook/Ceph (or Longhorn)** — each tool closing exactly one gap a managed cloud provider would otherwise close automatically. - **`cordon` and `drain` are the manual, human-triggered equivalent of what a cloud provider's automated node replacement does for you** — bare-metal hardware maintenance without them bypasses graceful shutdown and PodDisruptionBudgets entirely, a real, avoidable production risk. **Continue to Part 7** (`07-eks-deep-dive.md`) to go deep on Amazon EKS specifically — Karpenter, EKS Pod Identity, Auto Mode, multi-tenancy, GitOps, and cost optimization, all beyond what Part 5's provider comparison covers. See `questions.md` in this folder for the full interview question bank.