On-Prem & Self-Managed Kubernetes

Table of Contents#

1. Why Anyone Still Runs Kubernetes Themselves
2. kubeadm — The Official Bootstrapping Tool
3. kubeadm HA Control Plane Setup
4. What kubeadm Deliberately Doesn't Do
5. Lightweight Distributions: k3s, k0s, and MicroK8s
6. Cluster API — Using Kubernetes to Manage Kubernetes
7. Cluster API In Depth: ClusterClass and Fleet Templating
8. Rancher — Multi-Cluster Management
9. Rancher Fleet — GitOps for Multi-Cluster
10. Talos Linux — The Immutable, API-Only OS
11. The Bare-Metal-Specific Gaps
12. Bare-Metal Node Provisioning: PXE Boot and Tinkerbell
13. MetalLB — Solving LoadBalancer Services On-Prem
14. MetalLB Layer 2 vs BGP Mode, In Depth
15. Bare-Metal Storage
16. Air-Gapped Kubernetes
17. Air-Gapped Cluster Setup, Step by Step
18. Node Maintenance and Hardware Failure On-Prem
19. A Full Worked Example: Building a Complete Bare-Metal Production Stack
20. A Full Worked Comparison
21. Part 6 CLI Cheat Sheet
22. Common Mistakes
23. Worked Practice Problems
24. Summary and What's Next

Why Anyone Still Runs Kubernetes Themselves#

Given Part 5's case for managed Kubernetes, it's worth directly addressing: why does self-managed, on-premise Kubernetes remain genuinely common, not a legacy relic?

This Part answers that question with every tool needed to actually run it well — bootstrapping, fleet lifecycle, bare-metal-specific gaps, and hardware maintenance.

A strong, senior-level interview line, directly connecting to the RPO/RTO cost-curve discussion from the Disaster Recovery series: "The decision between managed and self-managed Kubernetes is a genuine cost/control tradeoff, exactly like choosing a DR strategy — self-managing trades ongoing operational toil (Part 5's core argument for managed) for control over cost at scale, data residency, and latency that a managed offering genuinely can't provide for certain workloads."

The cost crossover point, worth understanding as a real, concrete shape rather than an abstract claim:

Why this crossover point is worth naming honestly, rather than either extreme ("cloud is always cheaper" or "on-prem is always cheaper"): the crossover depends on genuinely specific factors — existing data center lease/depreciation schedules, staffing already in place, the workload's actual predictability (steady-state vs. genuinely variable), and the real, fully-loaded cost of the engineering time this entire Part's tooling (kubeadm, CAPI, MetalLB, Rook/Ceph, and the ongoing operational burden of running all of it) actually requires. A strong, honest answer names the crossover as workload- and organization-specific, never a blanket rule — this is precisely why the reasons named at the top of this section (data residency, latency, existing investment, air-gapped requirements) are usually the deciding factors in practice, with raw cost-at-scale as a real but secondary consideration that only tips the balance for genuinely large, steady, predictable workloads.

kubeadm — The Official Bootstrapping Tool#

kubeadm is the Kubernetes project's own, official tool for bootstrapping a cluster — genuinely worth knowing hands-on, since it's the closest thing to "vanilla," from-first-principles Kubernetes setup.

# On the FIRST control plane node
sudo kubeadm init --pod-network-cidr=10.244.0.0/16

# Configure kubectl for the current user
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Install a CNI plugin (Part 3) — kubeadm does NOT do this for you
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

# On EVERY worker node — join using the token kubeadm init printed
sudo kubeadm join <control-plane-ip>:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

Why this is such a genuinely valuable exercise even for engineers who'll mostly use managed Kubernetes day to day: running kubeadm init yourself, once, makes every abstract component from Part 1 (the API server, etcd, the scheduler) suddenly concrete and tangible — you SEE them as actual running processes/pods, not just diagram boxes. Many strong Kubernetes engineers deliberately do this at least once specifically to build that intuition.

A genuinely practical follow-up worth trying immediately after: kubectl get pods -n kube-system on a freshly-kubeadm-initialized cluster shows the API server, etcd, scheduler, and controller-manager all running as regular, ordinary Pods — a real, concrete confirmation that Part 1's control-plane components aren't some special, hidden kind of process, just Kubernetes running Kubernetes.

kubeadm HA Control Plane Setup#

The single-node example above is fine for learning, but genuinely production-worthy self-managed clusters need a highly-available control plane — worth understanding the real mechanics, directly extending Part 1's etcd/API-server-HA discussion to the self-managed context.

# First control plane node - note --control-plane-endpoint,
# pointing at the LOAD BALANCER, not this node's own IP
sudo kubeadm init \
  --control-plane-endpoint "k8s-lb.internal:6443" \
  --upload-certs \
  --pod-network-cidr=10.244.0.0/16

# Additional control plane nodes join as CONTROL PLANE members,
# not as workers - note the --control-plane flag
sudo kubeadm join k8s-lb.internal:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <key-from-upload-certs>

Why the external load balancer is genuinely mandatory, not optional, for a real HA setup — worth stating precisely: without it, every kubectl/kubelet client would need to hardcode one specific control plane node's IP, defeating the entire point of having multiple control plane replicas (Part 1's API-server-HA discussion) — the load balancer is what gives clients a single, stable endpoint that transparently routes to whichever control plane nodes are currently healthy, exactly mirroring the managed-provider behavior from Part 5 that you'd otherwise take for granted. kube-vip (introduced later in this Part, alongside MetalLB) is a genuinely common alternative to a dedicated external load balancer specifically for this control-plane-endpoint role.

Stacked vs. external etcd topology, a genuinely important, real architectural decision worth naming precisely:

Why this distinction matters concretely, worth stating as a real tradeoff: with stacked etcd, losing 2 of 3 control plane nodes simultaneously (a real, if uncommon, failure mode — a rack power event, for instance) takes down both API server availability AND etcd quorum (Part 1) at once. External etcd means a control-plane-node failure and an etcd-member failure are genuinely independent events, at the cost of operating more machines — the same "simplicity vs. blast-radius isolation" tradeoff that recurs throughout this course, here applied to the control plane's own internal topology.

Most real production kubeadm deployments use stacked etcd specifically because the operational simplicity is worth it at typical HA scale (3-5 control plane nodes) — external etcd is reserved for genuinely large, mission-critical deployments where the additional blast-radius isolation clearly justifies the extra machines and operational surface.

Worth confirming HA status directly, a real, practical diagnostic worth knowing: kubectl get pods -n kube-system -l component=etcd -o wide shows every etcd member and which node it's running on — a quick, concrete way to verify a cluster's actual etcd topology matches what was intended, rather than assuming.

What kubeadm Deliberately Doesn't Do#

A genuinely important, frequently-tested nuance — kubeadm is explicitly scoped to be a bootstrapping tool ONLY, not a full lifecycle management platform.

Why this matters practically, worth stating explicitly: kubeadm gets you to "a working cluster exists," but everything covered elsewhere in this course — etcd backups (Part 1/4), monitoring (Monitoring Methodologies series), CNI choice and NetworkPolicy enforcement (Part 3, DevSecOps series) — remains entirely your own responsibility to design and implement, with zero built-in guidance or automation from kubeadm itself. This is precisely the gap the higher-level tools covered next (Cluster API, Rancher) exist to fill.

A useful, quick mental model worth stating explicitly: kubeadm is to a Kubernetes cluster roughly what docker run is to a full production container deployment — it gets the core thing genuinely working, correctly, but every operational concern beyond that initial bootstrap (monitoring, backup automation, fleet consistency, day-2 lifecycle) is deliberately left as a separate, composable decision, not bundled in by default.

Lightweight Distributions: k3s, k0s, and MicroK8s#

For resource-constrained environments (edge computing, IoT, development/testing, small on-prem deployments), full upstream Kubernetes can be genuinely heavier than needed — lightweight distributions trim it down.

# Installing k3s is genuinely this simple — one command,
# a real, working cluster in under a minute
curl -sfL https://get.k3s.io | sh -

# Check it's running
sudo k3s kubectl get nodes

# MicroK8s's distinctive add-on model
sudo snap install microk8s --classic
microk8s enable dns storage ingress
microk8s kubectl get nodes

Why the simplicity of that one-line install is worth highlighting explicitly: it's a dramatic contrast to kubeadm's multi-step process, specifically because k3s ships with sane, working defaults for the pieces kubeadm deliberately leaves to you (CNI, storage, ingress) — a genuine, real tradeoff of "opinionated and simple" versus "unopinionated and flexible."

Every one of k3s's bundled defaults (its own lightweight CNI, its local-path-provisioner storage class, its built-in Traefik Ingress Controller) can be individually disabled and swapped for a different choice at install time — it's an opinionated starting point, not a hard architectural lock-in.

Why MicroK8s's add-on model is worth knowing as a genuinely distinct middle ground, precisely between k3s's "batteries-included by default" and kubeadm's "nothing included at all": rather than either extreme, MicroK8s ships a genuinely minimal core and lets you explicitly opt into exactly the components a given use case needs (dns, storage, ingress, metrics-server, and more, each a single named add-on) — a real, practical middle point on the same "opinionated vs. flexible" spectrum every lightweight distribution sits somewhere along.

All three remain genuinely conformant, real Kubernetes underneath their respective packaging choices — the exact same CNCF conformance guarantee already discussed for the major managed providers in Part 5 applies here too, so workload manifests remain fully portable regardless of which lightweight distribution actually runs them.

Cluster API — Using Kubernetes to Manage Kubernetes#

A genuinely elegant, conceptually important tool worth understanding deeply — Cluster API (CAPI) is a CNCF project that manages the lifecycle of Kubernetes clusters themselves using Kubernetes's own declarative, reconciliation-loop model (Part 1) as the management mechanism.

# A simplified CAPI Cluster object — declaring a DESIRED
# cluster, exactly like declaring a Deployment (Part 2)
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: production-cluster
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["10.244.0.0/16"]
  infrastructureRef:
    kind: VSphereCluster    # or AWSCluster, MetalCluster, etc.
    name: production-cluster

Why this is genuinely one of the most conceptually elegant tools worth citing in an interview, and it directly closes the loop with Part 1's reconciliation-loop discussion: "Cluster API applies the EXACT same declarative, reconciliation-loop pattern used for Pods and Deployments — the fundamental idea from Part 1 — one level UP, to entire clusters. You declare 'I want a cluster that looks like this,' and CAPI's controllers continuously work to make that true, exactly the same way a Deployment Controller keeps your pod count correct." CAPI supports a wide range of "infrastructure providers" (AWS, Azure, GCP, vSphere, bare metal, and more), making it a genuinely provider-agnostic way to manage fleets of clusters declaratively.

Worth stating precisely: CAPI can even manage clusters running ON managed cloud providers themselves (an AWSManagedControlPlane for EKS, for instance) — meaning the same declarative, GitOps-friendly lifecycle model can span self-managed on-prem clusters AND managed cloud clusters under one consistent tooling layer, a genuinely real answer to "how do we manage a truly hybrid fleet consistently" beyond just the on-prem case this Part focuses on.

Cluster API In Depth: ClusterClass and Fleet Templating#

Beyond the single Cluster object shown above, CAPI's real production value shows up at fleet scale — worth understanding the templating mechanism that makes managing dozens of similar clusters genuinely tractable.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: dev-team-a
spec:
  topology:
    class: standard-cluster-class   # references a ClusterClass, defined once
    version: v1.31.0
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
        - class: default-worker
          name: md-0
          replicas: 5

Why ClusterClass is worth knowing as a genuinely significant capability, directly extending the "same idea, one level up" framing already used for CAPI itself: this is the exact same relationship a Deployment's pod template has to its individual Pods (Part 2) — define the shape ONCE, in a ClusterClass, and every actual Cluster object becomes a lightweight reference plus a small set of overrides, rather than a fully-repeated, independently-maintained cluster specification. The direct, practical payoff, worth stating explicitly: rolling out a Kubernetes version upgrade or a control-plane configuration change across dozens of clusters becomes a single ClusterClass edit, propagated by CAPI's own reconciliation loops to every referencing cluster — rather than a dozens-of-times-repeated manual change, directly extending the "fleet consistency through templating" theme.

ClusterClass and Rancher Fleet, covered later in this Part, are genuinely complementary, not competing — ClusterClass templates the CLUSTERS themselves, while Fleet templates what gets DEPLOYED onto already-existing clusters; a mature multi-cluster platform commonly uses both together.

A genuinely real production use case worth citing: a platform team offering "Kubernetes as a service" internally to many application teams, where every team's cluster should follow the same organizational baseline (security policies, monitoring agents, networking defaults) but with legitimate per-team sizing differences — ClusterClass is precisely the mechanism that keeps that baseline consistent and centrally upgradable, without each team's cluster drifting independently over time.

Rancher — Multi-Cluster Management#

Rancher (also from SUSE, the same organization behind k3s) addresses a different, related problem: managing MANY clusters — potentially a mix of on-prem, managed cloud, and edge clusters — from one unified control plane and UI.

Why this matters practically, worth stating explicitly: real organizations, especially larger ones, rarely run just ONE Kubernetes cluster, on just ONE platform — Rancher's value proposition is providing a single, consistent RBAC, policy, and observability layer across a genuinely heterogeneous fleet, rather than requiring separate tooling and separate logins per cluster/provider. This is a distinct, complementary concern from CAPI's cluster-lifecycle focus — some organizations use both together.

Rancher itself runs AS a Kubernetes application (typically on its own small "local" cluster), managing every other cluster in the fleet through standard Kubernetes API calls against each — no proprietary agent-based protocol, just the same API surface covered throughout this entire series.

Rancher Fleet, worth knowing as Rancher's own answer to "how do I deploy the SAME application consistently across every cluster in this fleet":

# A Fleet GitRepo object - watches a Git repo, deploys its
# contents to every cluster matching the target selector
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: platform-baseline
  namespace: fleet-default
spec:
  repo: https://github.com/example/platform-baseline
  targets:
    - clusterSelector:
        matchLabels: {env: production}

Why Fleet matters concretely, directly extending the GitOps discussion referenced throughout this course (fully covered in the Automation, CI/CD & GitOps series): it applies the exact same "Git as the single source of truth, continuously reconciled" model already covered for single-cluster GitOps (ArgoCD/Flux) but natively fans it out across an entire Rancher-managed fleet using cluster label selectors — a genuinely practical mechanism for keeping a consistent baseline (monitoring agents, security policies, platform-team-owned components) applied uniformly across every cluster in a heterogeneous fleet, without manually repeating the deployment per cluster.

A team already standardized on ArgoCD for single-cluster GitOps doesn't necessarily need to switch to Fleet — ArgoCD's own ApplicationSet resource solves a genuinely similar multi-cluster fan-out problem; Fleet's distinctive advantage is its native, built-in integration specifically with Rancher's own cluster inventory and RBAC model.

Talos Linux — The Immutable, API-Only OS#

A genuinely modern, distinctive approach worth knowing about — Talos Linux is a purpose-built operating system specifically designed to run Kubernetes, with one radical, deliberate design choice: there is no SSH access at all.

# Talos is configured and managed entirely through its own
# API and CLI — never SSH
talosctl apply-config --nodes 192.168.1.10 --file worker.yaml
talosctl upgrade --nodes 192.168.1.10 --image ghcr.io/siderolabs/installer:v1.6.0

Why this is worth citing as a genuinely forward-looking, distinctive example, directly extending the "minimal attack surface" principle from the DevSecOps series' distroless container discussion to the HOST OS level itself: "Talos applies exactly the same minimal-attack-surface philosophy as distroless container images — but to the node's OPERATING SYSTEM, not just the container. No SSH, no shell, no general-purpose package manager means an attacker who somehow gains access has dramatically less to actually DO, and every legitimate operation happens through an auditable, declarative API instead of ad hoc shell commands."

Talos's own cluster bootstrap process (talosctl bootstrap) is conceptually the same declarative HA pattern already covered for kubeadm — control plane nodes behind a stable endpoint, an explicit etcd topology — just expressed through Talos's own API instead of kubeadm init/join.

A fuller node-OS comparison, worth having as a single reference — Talos isn't the only Kubernetes-optimized OS choice:

Why this spectrum is worth presenting as a genuine gradient, not a binary "traditional vs. Talos" choice: Flatcar and Bottlerocket occupy real, meaningful middle ground — immutable, atomic-update philosophy without going as far as Talos's complete elimination of interactive access. A strong, senior-level answer names the specific point on this spectrum that fits a given team's actual operational maturity and security requirements, rather than treating "most locked-down" as automatically "best" — a team with no existing API-driven-operations tooling or runbooks may get more real value from Flatcar's more familiar operational model than from jumping straight to Talos's fully API-only extreme.

This exact "match the tool to actual operational maturity, not to the most extreme option" principle recurs throughout this Part — the same reasoning applies to choosing between Ceph and Longhorn, and between kubeadm and a batteries-included distribution like k3s.

That principle is the real thread tying this entire Part together — worth restating one final time in the closing summary below.

The Bare-Metal-Specific Gaps#

Regardless of which provisioning tool is used, running Kubernetes on genuine bare metal (not virtualized cloud infrastructure) surfaces real gaps that managed cloud Kubernetes (Part 5) simply doesn't have to think about, because the cloud provider already solved them.

Bare-Metal Node Provisioning: PXE Boot and Tinkerbell#

The third gap named above — actually getting an operating system installed onto a physical machine in the first place — deserves its own treatment, since it's the one gap that has no cloud-API equivalent at all to conceptually borrow from.

Why PXE (Preboot Execution Environment) is worth knowing by name, precisely: it's the standard, decades-old network-boot protocol that makes "zero-touch" bare-metal provisioning possible at all — a machine with no OS installed yet can still boot, entirely over the network, without anyone physically inserting install media. This is the genuine, low-level mechanism underneath every "automated bare-metal provisioning" tool, including the more modern ones covered next.

Tinkerbell (a CNCF project), worth knowing as the modern, Kubernetes-native evolution of this exact PXE workflow: it wraps the PXE boot process, OS installation, and even ongoing hardware lifecycle actions (BMC/IPMI power control, firmware updates) into declarative, Kubernetes-style CRDs and workflows — directly extending the CRD/controller pattern from Part 4 to the literal physical-hardware-provisioning layer. A Workflow custom resource describes the exact sequence of actions (wipe disk, PXE boot, run installer, reboot) the same declarative way a Deployment describes desired pod state, and Tinkerbell's controllers execute and track it to completion.

# A genuinely simplified illustration of a Tinkerbell Hardware object -
# registering a specific physical machine by its MAC address
apiVersion: tinkerbell.org/v1alpha1
kind: Hardware
metadata:
  name: bare-metal-node-1
spec:
  interfaces:
    - dhcp:
        mac: "aa:bb:cc:dd:ee:ff"
        ip: {address: "192.168.1.50"}
      netboot:
        allowPXE: true

Why this matters as a real, worth-knowing capability, connecting directly to Cluster API's own bare-metal infrastructure providers: Cluster API's Metal3 provider builds on exactly this kind of bare-metal-provisioning automation (historically via Ironic, an OpenStack-originated bare-metal provisioning service, with Tinkerbell as a newer, increasingly popular alternative) — meaning the full stack, from "empty rack of servers" to "CAPI-managed fleet of Kubernetes clusters," genuinely can be declarative and automated end to end, not just from the point a base OS is already installed.

This closes the loop back to Cluster API's ClusterClass templating covered earlier — the same declarative fleet-management layer can, with Metal3, extend all the way down to the physical hardware itself, not just the Kubernetes objects running on top of already-provisioned machines.

BMC/IPMI power control, mentioned above, is worth naming explicitly as the concrete mechanism behind this: it's the standard out-of-band management interface most server-grade hardware exposes, letting Tinkerbell (or any equivalent) power-cycle, reboot, or reimage a machine remotely, without any operating system running on it at all yet.

MetalLB — Solving LoadBalancer Services On-Prem#

Directly extending the Service types discussion from Part 3 — a genuinely important, concrete gap-filler worth knowing by name.

Why this matters, worth stating explicitly: on AWS/Azure/GCP (Part 5), creating a type: LoadBalancer Service automatically provisions a REAL cloud load balancer behind the scenes — on bare metal, there's no equivalent cloud API to call, so without MetalLB (or an equivalent), a LoadBalancer Service would simply sit forever in a Pending state, with no external IP ever assigned. MetalLB fills exactly this gap, using genuine networking protocols (ARP or BGP) already covered in the Linux & Networking Fundamentals series, rather than any cloud-specific API.

Worth naming a real, common alternative to MetalLB, even briefly: kube-vip — solves a genuinely overlapping problem (virtual IP announcement for both LoadBalancer Services and, distinctively, the control-plane HA endpoint itself, as an alternative to a dedicated external load balancer for the --control-plane-endpoint setup covered earlier in this Part) — worth knowing it exists as a second real option, not just MetalLB by default.

MetalLB Layer 2 vs BGP Mode, In Depth#

The two modes MetalLB supports solve the "how does the network actually learn to route to this IP" problem in genuinely different ways, each with real, distinct tradeoffs worth knowing precisely.

# Layer 2 mode configuration
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
spec:
  addresses:
    - 192.168.1.200-192.168.1.220
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-advertise
spec:
  ipAddressPools: [default-pool]

# BGP mode configuration - requires a real upstream BGP peer
apiVersion: metallb.io/v1beta1
kind: BGPPeer
metadata:
  name: upstream-router
spec:
  myASN: 64500
  peerASN: 64501
  peerAddress: 192.168.1.1
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: bgp-advertise
spec:
  ipAddressPools: [default-pool]

Why the choice between them is a genuinely real, concrete tradeoff worth stating precisely, not just "BGP is more advanced so pick it": Layer 2 mode's single-node-announces-the-IP behavior means a failover (when the announcing node fails and another node takes over the announcement) is genuinely fast but still involves a brief interruption and, critically, all traffic to that Service funnels through one node's network capacity, even with many healthy backend pods spread across many nodes — a real, concrete bottleneck at meaningful traffic volume. BGP mode's ECMP-based multi-node announcement genuinely load-balances at the network layer itself, but requires actual coordination with whoever manages the physical network's routers — a real organizational dependency, not just a Kubernetes-side configuration change, which is exactly why Layer 2 mode remains the more commonly deployed choice for smaller, simpler on-prem environments where that router-level coordination isn't readily available.

Bare-Metal Storage#

The same "no automatic cloud provisioning" gap applies to storage — extending the CSI/dynamic-provisioning discussion from Part 3.

Why Rook/Ceph is worth knowing specifically, and it directly ties back to the replication concepts from the Databases & Storage Reliability series: it brings genuine, distributed, replicated storage — conceptually similar to how a cloud provider's EBS/managed disks work under the hood — to a bare-metal environment, running as Kubernetes-native Operators (Part 4's pattern) rather than requiring a separate, external storage appliance.

A real, worth-knowing alternative to Rook/Ceph: Longhorn (a CNCF project, originally by Rancher/SUSE) — genuinely simpler to operate than Ceph, trading some of Ceph's raw feature breadth and massive-scale proven track record for a meaningfully gentler operational learning curve.

# Installing Longhorn is genuinely simpler than standing up Ceph
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.0/deploy/longhorn.yaml

Why this choice is worth presenting as a genuine, real tradeoff, not "always pick the more feature-rich option": Ceph's operational complexity is real and non-trivial — proper Ceph operation genuinely benefits from dedicated storage expertise, while Longhorn deliberately trades some of that raw capability for approachability, making it a legitimately better fit for smaller teams and clusters where standing up and maintaining a full Ceph deployment would be disproportionate operational overhead relative to the actual storage needs.

Both integrate as standard CSI drivers (Part 3) — the choice between them changes operational complexity, not how application manifests actually consume the storage, since PVCs/StorageClasses look identical from the application's point of view regardless of which backend fulfills them.

Air-Gapped Kubernetes#

A genuinely specialized but real, important scenario worth knowing about — some regulated/defense environments require zero internet connectivity, at all, ever.

Why this is worth being aware of, even briefly, tying to the supply chain security discussion from the DevSecOps series (Part 5): an air-gapped cluster's entire software supply chain has to be deliberately, manually curated and mirrored — directly connecting to the SBOM and artifact-signing concepts from that earlier tutorial, since knowing EXACTLY what's running (and verifying it hasn't been tampered with before it's mirrored in) matters even more when there's no ongoing internet-based verification possible after the fact.

Air-gapped requirements and standard bare-metal deployments aren't mutually exclusive — a fully air-gapped environment still needs every other tool covered in this Part (kubeadm/k3s, MetalLB, redundant storage), just with every single image and chart pre-staged through the mirroring process covered next, rather than pulled live.

Air-Gapped Cluster Setup, Step by Step#

A concrete, worked walkthrough — genuinely useful to be able to narrate end to end, since "air-gapped Kubernetes" is often understood only abstractly.

A concrete, worked example — the actual commands, on the connected side first:

Copy

# Identify and pull every required image
images=(registry.k8s.io/kube-apiserver:v1.31.0 registry.k8s.io/etcd:3.5.15-0 docker.io/flannel/flannel:v0.25.5)
for img in "${images[@]}"; do
  docker pull "$img"
  docker save "$img" -o "$(basename "$img" | tr ':' '_').tar"
done

Then, inside the air-gapped network, after physical transfer:

Copy

# Load each image and push to the internal private registry
for tar in *.tar; do
  docker load -i "$tar"
done
docker tag registry.k8s.io/kube-apiserver:v1.31.0 internal-registry.local/kube-apiserver:v1.31.0
docker push internal-registry.local/kube-apiserver:v1.31.0

# Bootstrap kubeadm, pointing explicitly at the internal registry
sudo kubeadm init --image-repository internal-registry.local --pod-network-cidr=10.244.0.0/16

Why --image-repository is worth knowing as the specific, concrete kubeadm flag that makes this possible: without it, kubeadm defaults to pulling every control-plane component image from the public registry.k8s.io, which is completely unreachable inside an air-gapped network by definition — explicitly overriding it to point at the internal mirror is the exact mechanism that makes an air-gapped kubeadm init succeed at all.

The same pattern applies identically to k3s (--system-default-registry) and to any Helm chart's image.repository value — every tool covered in this Part that pulls container images needs its own explicit override, following the same underlying principle.

Talos, MetalLB, Rook/Ceph, and Cluster API's own controller images all follow this identical pattern too — the specific override flag or values-file key differs per tool, but the underlying requirement is universal across this entire Part's toolset.

A genuinely real, easy-to-overlook detail worth naming explicitly: EVERY image reference, everywhere, needs updating, not just the control plane's. CNI manifests, CSI driver manifests, Helm chart values.yaml image repositories, and any application's own Deployment specs all typically hardcode a public registry path by default — each one needs to be explicitly retargeted at the internal mirror, a genuinely tedious but non-negotiable step, since a single missed reference means that specific component silently fails to pull its image inside the air-gapped environment.

Node Maintenance and Hardware Failure On-Prem#

A final, genuinely important gap worth naming explicitly: cloud providers automatically replace a failed VM's underlying hardware transparently — on bare metal, a failed physical machine is a real, physical problem someone has to actually go fix.

Why cordon and drain are worth knowing as two genuinely distinct, sequential steps, not one action, precisely because bare-metal maintenance is manual and deliberate in a way cloud node replacement isn't: cordon alone only prevents NEW pods from scheduling onto the node — existing pods keep running, untouched. drain (which implicitly cordons first) actively evicts existing pods, respecting PodDisruptionBudgets (Part 2) the same way a voluntary disruption would on any cluster. This two-step, explicit, human-triggered process is precisely what a cloud provider's automated node replacement (Part 5's cordon-and-drain node-upgrade discussion) does FOR you automatically — on bare metal, an admin performs the equivalent sequence manually, for a real, physical hardware event rather than a scheduled version upgrade.

# The full, real bare-metal hardware-failure sequence
kubectl cordon node-5
kubectl drain node-5 --ignore-daemonsets --delete-emptydir-data --force
# ... physically service the hardware, or replace it entirely ...
kubectl uncordon node-5   # only once genuinely healthy again

--delete-emptydir-data and --force, worth knowing precisely why they're sometimes needed: drain refuses to proceed by default if a pod uses emptyDir storage (Part 3 — data that would be permanently lost on eviction) or isn't managed by a controller at all — these flags are explicit, deliberate overrides acknowledging that data loss or an unmanaged pod's termination is genuinely expected and acceptable for this specific maintenance operation, not a default to reach for casually.

A genuinely important interaction worth naming: drain respects PodDisruptionBudgets and can BLOCK indefinitely if a PDB's minimum-availability requirement can't currently be satisfied. This is correct, intended behavior, not a bug — if draining a node would violate a critical service's PDB, drain waits rather than proceeding, giving an operator the chance to notice and address the underlying capacity issue before forcing through a disruption that would breach the availability guarantee the PDB was specifically created to protect.

A Full Worked Example: Building a Complete Bare-Metal Production Stack#

Tying every tool covered in this Part together into one coherent, end-to-end architecture — genuinely valuable to be able to narrate as a complete story, not just recite the individual tools.

Why this specific combination is worth presenting as a coherent, real architecture, not just a list of unrelated tools: every tool here solves one, and only one, genuinely distinct gap identified earlier in this Part — Tinkerbell solves node provisioning, Talos solves node OS security, the HA control plane pattern solves control-plane availability, MetalLB solves the LoadBalancer Service gap, Rook/Ceph solves the storage gap, and Cluster API + Fleet solve fleet-scale lifecycle and consistency once there's more than one such cluster. A strong, senior-level closing line worth stating explicitly: "Bare-metal Kubernetes isn't one tool replacing a cloud provider — it's a deliberate assembly of purpose-built tools, each closing exactly one gap a managed cloud provider would otherwise close for you automatically, and understanding WHICH gap each tool closes is what separates knowing the tool names from actually being able to design the system."

Not every deployment needs every layer of this stack — a small internal-tools cluster might reasonably skip Tinkerbell (manually image a handful of machines) and Cluster API (manage the one cluster directly), while still genuinely needing MetalLB and some form of redundant storage. Scale the assembly to the actual requirement, not to this diagram's full breadth by default.

A Full Worked Comparison#

A closing, worth-remembering note on this table: every row solves a genuinely distinct, non-overlapping problem — the real skill in bare-metal Kubernetes architecture isn't memorizing this list, it's correctly mapping a specific organizational requirement (which of these gaps actually matters for THIS deployment) onto the smallest set of tools that closes exactly those gaps, without over-provisioning tooling for problems the deployment doesn't actually have.

Part 6 CLI Cheat Sheet#

# kubeadm
kubeadm init --control-plane-endpoint "k8s-lb.internal:6443" --upload-certs
kubeadm join <endpoint> --token <token> --discovery-token-ca-cert-hash sha256:<hash>
kubeadm token create --print-join-command   # generate a new join command (tokens expire)
kubeadm upgrade plan
kubeadm upgrade apply v1.31.0

# k3s / k0s / MicroK8s
curl -sfL https://get.k3s.io | sh -
sudo k3s kubectl get nodes
microk8s status --wait-ready
microk8s enable dns storage ingress

# Cluster API
clusterctl init --infrastructure aws
kubectl get clusters -A
kubectl get machinedeployments -A
clusterctl describe cluster my-cluster

# MetalLB
kubectl get ipaddresspool,l2advertisement,bgpadvertisement -n metallb-system
kubectl describe svc <loadbalancer-svc> | grep -A 3 "LoadBalancer Ingress"

# Talos
talosctl get members
talosctl dashboard
talosctl logs kubelet

Common Mistakes#

Worked Practice Problems#

Problem 1: A team runs kubeadm init successfully, but kubectl get nodes shows the control plane node stuck in NotReady status. What's the most likely cause, based on what kubeadm deliberately doesn't do?

Answer: kubeadm deliberately doesn't install a CNI plugin (Part 3) as part of its bootstrapping process — without one, pod networking isn't functional, and the node correctly reports NotReady because a required cluster component (networking) isn't in place yet. The fix is applying a CNI plugin manifest (Flannel, Calico, Cilium, or another choice) immediately after kubeadm init, exactly as shown in this tutorial's worked example — this is expected, documented behavior, not a bug.

Problem 2: An organization runs 15 Kubernetes clusters across on-prem data centers, EKS, and AKS, and is struggling with inconsistent RBAC policies and no unified view of cluster health. Would you recommend Cluster API, Rancher, or both, and why?

Answer: Primarily Rancher — the core problem described (inconsistent RBAC across a heterogeneous fleet, no unified observability) is specifically Rancher's value proposition: one management layer providing consistent policy and visibility across mixed on-prem/cloud clusters. Cluster API would additionally be worth considering if the team ALSO struggles with the actual lifecycle management (creating/scaling/upgrading) of the on-prem clusters specifically in a declarative, automated way — the two tools solve genuinely different, complementary problems (Rancher: ongoing multi-cluster management/policy; CAPI: cluster lifecycle automation), and larger organizations often do use both together.

Problem 3: A security team is evaluating node operating systems for a new, security-sensitive Kubernetes deployment and is comparing a standard Ubuntu-based node setup against Talos Linux. What's the core security argument for Talos, and what real operational tradeoff should be weighed against it?

Answer: The core argument: Talos eliminates SSH and shell access entirely, managing nodes exclusively through a declarative API — this closes an entire category of attack surface (no shell for an attacker to abuse even after gaining some level of access) and forces every configuration change through an auditable, declarative path, directly extending the same minimal-attack-surface philosophy already covered for distroless container images in the DevSecOps series, now applied at the host OS level. The real tradeoff to weigh: teams accustomed to traditional SSH-based debugging workflows (checking logs directly on a node, running ad hoc diagnostic commands) need to adjust to Talos's API-only operational model, which can have a real learning curve and may require adapting existing runbooks (Incident Management series) that assume direct node shell access.

Problem 4: A self-managed cluster's HA control plane loses 2 of its 3 stacked-etcd control plane nodes simultaneously during a rack power event. What happens, and what topology choice, made at setup time, could have changed the outcome?

Answer: With stacked etcd (each control plane node running its own local etcd member), losing 2 of 3 nodes simultaneously means losing 2 of 3 etcd members at the same time — dropping below the majority quorum (Part 1) required for etcd to accept writes at all, meaning the entire cluster loses the ability to create or update any object until at least one more etcd member becomes available. If the deployment had used an external, dedicated etcd cluster (an alternative topology choice made at setup) instead of stacked etcd, the control-plane-node failures and etcd-member failures would have been genuinely independent events — the external etcd cluster, on its own separate machines, would likely have survived the same rack event untouched, keeping the cluster writable even while 2 of 3 control plane API servers were down. This is precisely the "decoupled blast radius vs. operational simplicity" tradeoff between stacked and external etcd topologies.

Problem 5: A team provisioning 200 bare-metal servers for a new Kubernetes fleet asks whether they need to manually image each machine before running kubeadm. What's the better, more automated answer?

Answer: No — this is exactly the gap PXE boot and tools like Tinkerbell exist to close. Rather than manually imaging 200 machines individually, a PXE-based workflow lets each machine network-boot into an automated installer the moment it's racked and powered on, with Tinkerbell (or an equivalent) orchestrating the OS installation declaratively via Kubernetes-style CRDs and Workflows — the same reconciliation-loop pattern from Part 1 and Part 4's Operator pattern, applied to physical hardware provisioning. Combined with Cluster API's Metal3 provider, the entire pipeline — from "200 empty racked servers" to "200 nodes bootstrapped and joined to Kubernetes clusters" — can be genuinely declarative and automated end to end, rather than requiring manual, per-machine imaging work.

Problem 6: A small platform team (3 engineers) with no prior distributed-storage experience needs redundant persistent storage for a 5-node on-prem cluster. They're debating between Rook/Ceph and Longhorn. What would you recommend, and why?

Answer: Longhorn, specifically because of the team's stated lack of distributed-storage expertise and small scale — Ceph is genuinely powerful and battle-tested at massive scale, but real, correct Ceph operation (monitoring OSD health, understanding placement groups, tuning replication) benefits substantially from dedicated storage expertise the team doesn't have. Longhorn's deliberately simpler operational model, approachable web UI, and Kubernetes-native design make it a legitimately better fit for a small team's actual needs at this scale — the "more capable" option isn't automatically the right choice when the team's operational capacity to run it correctly is the actual limiting factor, not the storage system's raw feature set.

Problem 7: A team standardizes on Rancher for multi-cluster management and wants every cluster in their fleet to automatically receive the same baseline monitoring agent and NetworkPolicy defaults, without manually applying them to each new cluster. What Rancher-native mechanism solves this?

Answer: Rancher Fleet, using a GitRepo object with a clusterSelector matching the target clusters (e.g., by an environment label). The baseline manifests (monitoring agent DaemonSet, default-deny NetworkPolicy per Part 3) live in a Git repository, and Fleet continuously reconciles every matching cluster in the fleet to match that Git state — new clusters that pick up the matching label automatically receive the baseline the next time Fleet reconciles, with zero manual per-cluster application needed. This is precisely the GitOps-at-fleet-scale pattern Fleet is built for, directly extending the single-cluster GitOps model already covered elsewhere in this course to Rancher's multi-cluster context specifically.

Problem 8: An on-call engineer, responding to a hardware alert, directly powers off a misbehaving bare-metal node without running kubectl drain first, reasoning "it's about to be forcibly removed anyway, why bother." What actually goes wrong, and what should have happened instead?

Answer: Powering off the node without draining first means every pod running on it is abruptly terminated with no graceful shutdown at all — no SIGTERM, no chance to finish in-flight requests or close connections cleanly (the same graceful-shutdown discipline from Part 1's kubelet discussion), and critically, PodDisruptionBudgets are bypassed entirely, since PDBs only govern voluntary disruptions initiated through the API (like drain), not an external hardware power-off the cluster has no way to negotiate with. If multiple replicas of a critical service happened to be concentrated on that one node (a real risk without proper anti-affinity, per Part 2), this could take the service's availability below its PDB's minimum with zero warning. The correct sequence: kubectl cordon then kubectl drain first, letting the scheduler gracefully reschedule pods onto healthy nodes and giving each pod its normal termination grace period — only after drain completes successfully should the physical hardware actually be powered off.

Summary and What's Next#

• Organizations self-manage Kubernetes for real, concrete reasons — data residency, cost at scale, latency, existing infrastructure investment, and air-gapped requirements — a genuine cost/control tradeoff against Part 5's managed-Kubernetes case, not an outdated approach.
• The cost crossover between cloud and self-managed is genuinely workload- and organization-specific — never a universal rule — and data residency/latency/air-gap requirements are usually the actual deciding factors in practice, with raw cost as a real but secondary consideration.
• The thread tying this entire Part together: match every tool to the team's actual operational maturity and the deployment's genuine requirements — not to the most extreme or most feature-rich option available, whether that's choosing an OS, a storage backend, or a bootstrapping tool.
• Cordon, then drain, then physically service the hardware, then uncordon — the manual bare-metal equivalent of a cloud provider's automated node replacement, and worth having as a memorized sequence, not just a concept.
• drain respects PodDisruptionBudgets and can legitimately block — this is correct, intended behavior signaling a real capacity issue, not something to force through by default.
• kubeadm is the official bootstrapping tool, deliberately scoped to control-plane setup only — it builds genuine intuition for Part 1's components, but leaves CNI, storage, and day-2 operations entirely to you.
• k3s and k0s are lightweight, opinionated, batteries-included distributions — ideal for edge/IoT and fast development environments, trading some flexibility for dramatic setup simplicity.
• Cluster API applies Kubernetes's own declarative reconciliation-loop pattern (Part 1) one level up, to manage the lifecycle of entire clusters across many infrastructure providers.
• Rancher solves a different, complementary problem: unified RBAC, policy, and observability across a heterogeneous, multi-cluster, multi-provider fleet.
• Talos Linux represents a genuinely modern, security-forward approach — an immutable, API-only OS with no SSH, extending the DevSecOps series' minimal-attack-surface philosophy to the host OS itself.
• Bare-metal deployments surface real gaps managed cloud Kubernetes doesn't have — MetalLB fills the LoadBalancer Service gap, and Rook/Ceph provides genuinely distributed, redundant storage where cloud CSI drivers would otherwise handle it automatically.
• Air-gapped environments require deliberately pre-staging every dependency (a mirrored private registry), directly connecting to the supply chain security practices from the DevSecOps series.
• kubeadm HA setups need an external load balancer in front of multiple control plane nodes, and the stacked vs. external etcd topology choice is a real decoupling-vs-simplicity tradeoff worth naming precisely.
• ClusterClass turns Cluster API from a single-cluster declarative tool into genuine fleet templating — one edit propagates to every referencing cluster, mirroring a Deployment template's relationship to its Pods.
• PXE boot and Tinkerbell close the one gap with no cloud-API equivalent at all — automating bare-metal OS provisioning declaratively, feeding directly into Cluster API's Metal3 provider for a fully end-to-end automated bare-metal fleet.
• MetalLB's Layer 2 mode is simple but funnels traffic through one node; BGP mode gives genuine multi-node load distribution at the cost of real upstream network team coordination.
• Rancher Fleet applies the GitOps model — already covered for single clusters elsewhere — natively across an entire fleet via cluster label selectors, keeping baselines consistent without manual per-cluster application.
• Node OS choice is a real spectrum, not a binary — Ubuntu/RHEL, Flatcar, Bottlerocket, and Talos sit at different points between operational familiarity and attack-surface minimization; pick based on the team's actual operational maturity, not just "most locked-down."
• Longhorn is a genuinely simpler alternative to Rook/Ceph for smaller clusters — the right storage system depends on the team's actual operational capacity, not just raw feature breadth.
• A complete bare-metal production stack composes Tinkerbell, Talos (or another OS choice), an HA control plane, MetalLB, and Rook/Ceph (or Longhorn) — each tool closing exactly one gap a managed cloud provider would otherwise close automatically.
• cordon and drain are the manual, human-triggered equivalent of what a cloud provider's automated node replacement does for you — bare-metal hardware maintenance without them bypasses graceful shutdown and PodDisruptionBudgets entirely, a real, avoidable production risk.

Continue to Part 7 (07-eks-deep-dive.md) to go deep on Amazon EKS specifically — Karpenter, EKS Pod Identity, Auto Mode, multi-tenancy, GitOps, and cost optimization, all beyond what Part 5's provider comparison covers. See questions.md in this folder for the full interview question bank.