On-Prem & Self-Managed Kubernetes
Table of Contents#
- Why Anyone Still Runs Kubernetes Themselves
- kubeadm — The Official Bootstrapping Tool
- What kubeadm Deliberately Doesn't Do
- Lightweight Distributions: k3s and k0s
- Cluster API — Using Kubernetes to Manage Kubernetes
- Rancher — Multi-Cluster Management
- Talos Linux — The Immutable, API-Only OS
- The Bare-Metal-Specific Gaps
- MetalLB — Solving LoadBalancer Services On-Prem
- Bare-Metal Storage
- Air-Gapped Kubernetes
- A Full Worked Comparison
- Common Mistakes
- Worked Practice Problems
- Summary — The Complete Kubernetes Deep Dive Series
Why Anyone Still Runs Kubernetes Themselves#
Given Part 5's case for managed Kubernetes, it's worth directly addressing: why does self-managed, on-premise Kubernetes remain genuinely common, not a legacy relic?
Diagram
A strong, senior-level interview line, directly connecting to the RPO/RTO cost-curve discussion from the Disaster Recovery series: "The decision between managed and self-managed Kubernetes is a genuine cost/control tradeoff, exactly like choosing a DR strategy — self-managing trades ongoing operational toil (Part 5's core argument for managed) for control over cost at scale, data residency, and latency that a managed offering genuinely can't provide for certain workloads."
kubeadm — The Official Bootstrapping Tool#
kubeadm is the Kubernetes project's own, official tool for bootstrapping a cluster — genuinely worth knowing hands-on, since it's the closest thing to "vanilla," from-first-principles Kubernetes setup.
# On the FIRST control plane node sudo kubeadm init --pod-network-cidr=10.244.0.0/16 # Configure kubectl for the current user mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config # Install a CNI plugin (Part 3) — kubeadm does NOT do this for you kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml # On EVERY worker node — join using the token kubeadm init printed sudo kubeadm join <control-plane-ip>:6443 --token <token> \ --discovery-token-ca-cert-hash sha256:<hash>
Diagram
Why this is such a genuinely valuable exercise even for engineers who'll mostly use managed Kubernetes day to day: running kubeadm init yourself, once, makes every abstract component from Part 1 (the API server, etcd, the scheduler) suddenly concrete and tangible — you SEE them as actual running processes/pods, not just diagram boxes. Many strong Kubernetes engineers deliberately do this at least once specifically to build that intuition.
What kubeadm Deliberately Doesn't Do#
A genuinely important, frequently-tested nuance — kubeadm is explicitly scoped to be a bootstrapping tool ONLY, not a full lifecycle management platform.
Diagram
Why this matters practically, worth stating explicitly: kubeadm gets you to "a working cluster exists," but everything covered elsewhere in this course — etcd backups (Part 1/4), monitoring (Monitoring Methodologies series), CNI choice and NetworkPolicy enforcement (Part 3, DevSecOps series) — remains entirely your own responsibility to design and implement, with zero built-in guidance or automation from kubeadm itself. This is precisely the gap the higher-level tools covered next (Cluster API, Rancher) exist to fill.
Lightweight Distributions: k3s and k0s#
For resource-constrained environments (edge computing, IoT, development/testing, small on-prem deployments), full upstream Kubernetes can be genuinely heavier than needed — lightweight distributions trim it down.
Diagram
# Installing k3s is genuinely this simple — one command, # a real, working cluster in under a minute curl -sfL https://get.k3s.io | sh - # Check it's running sudo k3s kubectl get nodes
Why the simplicity of that one-line install is worth highlighting explicitly: it's a dramatic contrast to kubeadm's multi-step process, specifically because k3s ships with sane, working defaults for the pieces kubeadm deliberately leaves to you (CNI, storage, ingress) — a genuine, real tradeoff of "opinionated and simple" versus "unopinionated and flexible."
Cluster API — Using Kubernetes to Manage Kubernetes#
A genuinely elegant, conceptually important tool worth understanding deeply — Cluster API (CAPI) is a CNCF project that manages the lifecycle of Kubernetes clusters themselves using Kubernetes's own declarative, reconciliation-loop model (Part 1) as the management mechanism.
Diagram
# A simplified CAPI Cluster object — declaring a DESIRED # cluster, exactly like declaring a Deployment (Part 2) apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: production-cluster spec: clusterNetwork: pods: cidrBlocks: ["10.244.0.0/16"] infrastructureRef: kind: VSphereCluster # or AWSCluster, MetalCluster, etc. name: production-cluster
Why this is genuinely one of the most conceptually elegant tools worth citing in an interview, and it directly closes the loop with Part 1's reconciliation-loop discussion: "Cluster API applies the EXACT same declarative, reconciliation-loop pattern used for Pods and Deployments — the fundamental idea from Part 1 — one level UP, to entire clusters. You declare 'I want a cluster that looks like this,' and CAPI's controllers continuously work to make that true, exactly the same way a Deployment Controller keeps your pod count correct." CAPI supports a wide range of "infrastructure providers" (AWS, Azure, GCP, vSphere, bare metal, and more), making it a genuinely provider-agnostic way to manage fleets of clusters declaratively.
Rancher — Multi-Cluster Management#
Rancher (also from SUSE, the same organization behind k3s) addresses a different, related problem: managing MANY clusters — potentially a mix of on-prem, managed cloud, and edge clusters — from one unified control plane and UI.
Diagram
Why this matters practically, worth stating explicitly: real organizations, especially larger ones, rarely run just ONE Kubernetes cluster, on just ONE platform — Rancher's value proposition is providing a single, consistent RBAC, policy, and observability layer across a genuinely heterogeneous fleet, rather than requiring separate tooling and separate logins per cluster/provider. This is a distinct, complementary concern from CAPI's cluster-lifecycle focus — some organizations use both together.
Talos Linux — The Immutable, API-Only OS#
A genuinely modern, distinctive approach worth knowing about — Talos Linux is a purpose-built operating system specifically designed to run Kubernetes, with one radical, deliberate design choice: there is no SSH access at all.
Diagram
# Talos is configured and managed entirely through its own # API and CLI — never SSH talosctl apply-config --nodes 192.168.1.10 --file worker.yaml talosctl upgrade --nodes 192.168.1.10 --image ghcr.io/siderolabs/installer:v1.6.0
Why this is worth citing as a genuinely forward-looking, distinctive example, directly extending the "minimal attack surface" principle from the DevSecOps series' distroless container discussion to the HOST OS level itself: "Talos applies exactly the same minimal-attack-surface philosophy as distroless container images — but to the node's OPERATING SYSTEM, not just the container. No SSH, no shell, no general-purpose package manager means an attacker who somehow gains access has dramatically less to actually DO, and every legitimate operation happens through an auditable, declarative API instead of ad hoc shell commands."
The Bare-Metal-Specific Gaps#
Regardless of which provisioning tool is used, running Kubernetes on genuine bare metal (not virtualized cloud infrastructure) surfaces real gaps that managed cloud Kubernetes (Part 5) simply doesn't have to think about, because the cloud provider already solved them.
Diagram
MetalLB — Solving LoadBalancer Services On-Prem#
Directly extending the Service types discussion from Part 3 — a genuinely important, concrete gap-filler worth knowing by name.
Diagram
Why this matters, worth stating explicitly: on AWS/Azure/GCP (Part 5), creating a type: LoadBalancer Service automatically provisions a REAL cloud load balancer behind the scenes — on bare metal, there's no equivalent cloud API to call, so without MetalLB (or an equivalent), a LoadBalancer Service would simply sit forever in a Pending state, with no external IP ever assigned. MetalLB fills exactly this gap, using genuine networking protocols (ARP or BGP) already covered in the Linux & Networking Fundamentals series, rather than any cloud-specific API.
Bare-Metal Storage#
The same "no automatic cloud provisioning" gap applies to storage — extending the CSI/dynamic-provisioning discussion from Part 3.
Diagram
Why Rook/Ceph is worth knowing specifically, and it directly ties back to the replication concepts from the Databases & Storage Reliability series: it brings genuine, distributed, replicated storage — conceptually similar to how a cloud provider's EBS/managed disks work under the hood — to a bare-metal environment, running as Kubernetes-native Operators (Part 4's pattern) rather than requiring a separate, external storage appliance.
Air-Gapped Kubernetes#
A genuinely specialized but real, important scenario worth knowing about — some regulated/defense environments require zero internet connectivity, at all, ever.
Diagram
Why this is worth being aware of, even briefly, tying to the supply chain security discussion from the DevSecOps series (Part 5): an air-gapped cluster's entire software supply chain has to be deliberately, manually curated and mirrored — directly connecting to the SBOM and artifact-signing concepts from that earlier tutorial, since knowing EXACTLY what's running (and verifying it hasn't been tampered with before it's mirrored in) matters even more when there's no ongoing internet-based verification possible after the fact.
A Full Worked Comparison#
Diagram
| Tool | Solves | Not Designed For |
|---|---|---|
| kubeadm | Bootstrapping a single cluster's control plane, from first principles | Ongoing lifecycle management, multi-cluster, day-2 ops |
| k3s / k0s | Lightweight, fast, opinionated single-cluster setup | Massive-scale, highly customized control plane configuration |
| Cluster API | Declarative lifecycle management of MANY clusters, across providers | A quick, one-off single cluster (real overhead to set up the management cluster itself) |
| Rancher | Unified multi-cluster UI, RBAC, and policy across a heterogeneous fleet | Being a lightweight, minimal-footprint single-cluster tool |
| Talos Linux | Minimizing node-level attack surface via an immutable, API-only OS | Teams that need traditional shell/SSH-based debugging workflows |
Common Mistakes#
| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Assuming kubeadm alone gives you a production-ready cluster | It deliberately only bootstraps the control plane — CNI, storage, monitoring, and day-2 ops are all left entirely to you | Explicitly plan and implement each of these separately, or use a more opinionated/batteries-included tool if that gap is unwanted |
Creating a type: LoadBalancer Service on a bare-metal cluster with no MetalLB (or equivalent) installed | It will sit in Pending state forever — there's no cloud API to automatically provision a real load balancer | Install MetalLB (or an equivalent) specifically to fill this gap on bare-metal/on-prem clusters |
Using local-path-provisioner-style local storage for genuinely critical, must-survive-node-failure data | Provides zero redundancy — losing that one node means losing that data entirely | Use a genuinely distributed storage system (like Rook/Ceph) for data that needs real durability guarantees |
| Treating self-managed Kubernetes as strictly cheaper than managed, without accounting for real operational staffing cost | The engineering time spent on control-plane operations, upgrades, and troubleshooting is a real, ongoing cost, not free | Weigh TOTAL cost (infrastructure + genuine engineering time) when comparing self-managed against managed options |
| Assuming an air-gapped cluster can use standard setup instructions unmodified | Standard tooling assumes internet access for pulling images/charts, which simply isn't available | Deliberately pre-stage a mirrored private registry and explicitly configure every component to use it |
Worked Practice Problems#
Problem 1: A team runs kubeadm init successfully, but kubectl get nodes shows the control plane node stuck in NotReady status. What's the most likely cause, based on what kubeadm deliberately doesn't do?
Answer: kubeadm deliberately doesn't install a CNI plugin (Part 3) as part of its bootstrapping process — without one, pod networking isn't functional, and the node correctly reports NotReady because a required cluster component (networking) isn't in place yet. The fix is applying a CNI plugin manifest (Flannel, Calico, Cilium, or another choice) immediately after kubeadm init, exactly as shown in this tutorial's worked example — this is expected, documented behavior, not a bug.
Problem 2: An organization runs 15 Kubernetes clusters across on-prem data centers, EKS, and AKS, and is struggling with inconsistent RBAC policies and no unified view of cluster health. Would you recommend Cluster API, Rancher, or both, and why?
Answer: Primarily Rancher — the core problem described (inconsistent RBAC across a heterogeneous fleet, no unified observability) is specifically Rancher's value proposition: one management layer providing consistent policy and visibility across mixed on-prem/cloud clusters. Cluster API would additionally be worth considering if the team ALSO struggles with the actual lifecycle management (creating/scaling/upgrading) of the on-prem clusters specifically in a declarative, automated way — the two tools solve genuinely different, complementary problems (Rancher: ongoing multi-cluster management/policy; CAPI: cluster lifecycle automation), and larger organizations often do use both together.
Problem 3: A security team is evaluating node operating systems for a new, security-sensitive Kubernetes deployment and is comparing a standard Ubuntu-based node setup against Talos Linux. What's the core security argument for Talos, and what real operational tradeoff should be weighed against it?
Answer: The core argument: Talos eliminates SSH and shell access entirely, managing nodes exclusively through a declarative API — this closes an entire category of attack surface (no shell for an attacker to abuse even after gaining some level of access) and forces every configuration change through an auditable, declarative path, directly extending the same minimal-attack-surface philosophy already covered for distroless container images in the DevSecOps series, now applied at the host OS level. The real tradeoff to weigh: teams accustomed to traditional SSH-based debugging workflows (checking logs directly on a node, running ad hoc diagnostic commands) need to adjust to Talos's API-only operational model, which can have a real learning curve and may require adapting existing runbooks (Incident Management series) that assume direct node shell access.
Summary — The Complete Kubernetes Deep Dive Series#
- Organizations self-manage Kubernetes for real, concrete reasons — data residency, cost at scale, latency, existing infrastructure investment, and air-gapped requirements — a genuine cost/control tradeoff against Part 5's managed-Kubernetes case, not an outdated approach.
- kubeadm is the official bootstrapping tool, deliberately scoped to control-plane setup only — it builds genuine intuition for Part 1's components, but leaves CNI, storage, and day-2 operations entirely to you.
- k3s and k0s are lightweight, opinionated, batteries-included distributions — ideal for edge/IoT and fast development environments, trading some flexibility for dramatic setup simplicity.
- Cluster API applies Kubernetes's own declarative reconciliation-loop pattern (Part 1) one level up, to manage the lifecycle of entire clusters across many infrastructure providers.
- Rancher solves a different, complementary problem: unified RBAC, policy, and observability across a heterogeneous, multi-cluster, multi-provider fleet.
- Talos Linux represents a genuinely modern, security-forward approach — an immutable, API-only OS with no SSH, extending the DevSecOps series' minimal-attack-surface philosophy to the host OS itself.
- Bare-metal deployments surface real gaps managed cloud Kubernetes doesn't have — MetalLB fills the LoadBalancer Service gap, and Rook/Ceph provides genuinely distributed, redundant storage where cloud CSI drivers would otherwise handle it automatically.
- Air-gapped environments require deliberately pre-staging every dependency (a mirrored private registry), directly connecting to the supply chain security practices from the DevSecOps series.
This completes the Kubernetes Deep Dive series (architecture, scheduling/workloads, networking/storage, service mesh/etcd/operators, managed Kubernetes, and on-prem/self-managed provisioning). See questions.md in this folder for the full interview question bank covering all six parts.