Cluster Upgrades & Lifecycle Management

Assumes you're comfortable with kubeadm bootstrapping from Part 6, managed cluster patterns from Parts 5 and 7, PodDisruptionBudgets from Part 2, and etcd operations from Part 4 — this chapter is about safely moving a running cluster forward in time, not re-introducing any of those foundations.

Table of Contents#

1. Why This Part Exists
2. The Kubernetes Release Cadence and Support Window
3. Version Skew Policy, Precisely
4. The kubeadm Upgrade Workflow
5. A Full Worked kubeadm Upgrade, Node by Node
6. Managed Cluster Upgrades — EKS, GKE, AKS Patterns
7. In-Place vs. Blue-Green Node Upgrades
8. A Worked Blue-Green EKS Node Upgrade
9. Finding and Fixing Deprecated APIs Before Upgrading
10. Upgrading Add-ons: CNI, CSI Drivers, and Operators
11. PodDisruptionBudgets and Drain Safety During Upgrades
12. Rollback Strategy — What Can and Can't Be Undone
13. etcd's Own Version Lifecycle
14. etcd Backup Before Any Upgrade
15. Node OS and Kernel Patching — a Separate Lifecycle
16. Testing an Upgrade Before Production
17. Fleet-Wide Upgrade Orchestration
18. Automating the Upgrade Pipeline
19. A Full Worked Scenario: Upgrading the checkout Cluster End-to-End
20. Post-Upgrade Verification Checklist
21. Part 15 CLI Cheat Sheet
22. A Worked Numeric Example: Why the Extra Rigor Pays For Itself
23. Quick Reference: Every Upgrade Decision in This Chapter
24. A Note on Upgrade Communication
25. Upgrade Cadence by Environment Tier
26. Common Mistakes and Interview Traps
27. Worked Practice Problems
28. Summary and What's Next

Why This Part Exists#

Every earlier part in this series described a cluster at a single point in time — this closing chapter is about the one thing every cluster owner eventually has to do repeatedly for the entire life of the cluster: move it forward to a new Kubernetes version without turning that routine maintenance into an outage. A cluster that's never upgraded doesn't stay safely frozen — it ages out of its vendor's support window, accumulates unpatched CVEs, and eventually faces a much riskier multi-version jump instead of a series of small, well-practiced ones. Treating upgrades as routine, low-drama maintenance rather than a rare, high-stakes event is the mindset shift this entire chapter is built around.

The throughline system closes out here too: checkout, catalog, inventory, and recommendations, running across however many nodes this series' worked examples have accumulated, all moving through one real upgrade cycle together in this chapter's final worked scenario. Every system this series introduced — the GPU-backed recommendation-model from Part 14, the multi-tenant guardrails from Part 13, the security posture from Part 11 — has to survive that same upgrade cycle intact, which is exactly what makes this a fitting closing chapter rather than an afterthought: an upgrade is the one operation that touches every other part of this series at once.

The Kubernetes Release Cadence and Support Window#

Kubernetes ships new minor versions on a regular cadence, and each one is supported for a limited window — understanding this rhythm is what turns "upgrade eventually" into an actual, plannable maintenance schedule rather than a reactive scramble once a cluster falls out of support.

Version Skew Policy, Precisely#

The version skew policy defines exactly how far out of sync different cluster components are allowed to be during an upgrade — this is what makes a safe, phased upgrade possible at all, rather than requiring every component to jump versions in one atomic, all-or-nothing operation.

This skew tolerance is precisely what makes the "control plane first, then workers" upgrade order (next section) both correct and safe — the API server can run one minor version ahead of the kubelets and still be within policy, giving an operator a real, supported window to roll worker nodes forward gradually rather than needing to upgrade every node in the cluster in one synchronized instant. Without this tolerance, every cluster upgrade would require a full, simultaneous stop-the-world cutover across every node at once — exactly the kind of high-risk, all-or-nothing operation the phased approach in this chapter exists to avoid.

The kubeadm Upgrade Workflow#

Part 6 covered kubeadm bootstrapping a cluster from scratch — upgrading an existing kubeadm cluster follows a distinct, ordered workflow that respects the version skew policy above at every step.

A Full Worked kubeadm Upgrade, Node by Node#

# On the primary control plane node
kubeadm upgrade plan
apt-get update && apt-get install -y kubeadm=1.31.0-1.1
kubeadm upgrade apply v1.31.0
apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
systemctl daemon-reload && systemctl restart kubelet

# On each ADDITIONAL control plane node
apt-get install -y kubeadm=1.31.0-1.1
kubeadm upgrade node
apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
systemctl daemon-reload && systemctl restart kubelet

# On each worker node, ONE AT A TIME
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
apt-get install -y kubeadm=1.31.0-1.1
kubeadm upgrade node
apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon <node-name>

--ignore-daemonsets --delete-emptydir-data on the drain step is worth explaining precisely, since both flags silently change what actually happens: DaemonSet-managed pods (Part 2) are excluded from drain because they're expected to run on every node by design and will simply restart on the node once it's back — draining would otherwise fail waiting for something that isn't supposed to leave. --delete-emptydir-data acknowledges that any pod using an emptyDir volume (Part 3) will lose that data on eviction, since emptyDir is explicitly node-local, ephemeral storage — omitting this flag causes drain to refuse to proceed at all rather than silently destroying data.

Managed Cluster Upgrades — EKS, GKE, AKS Patterns#

Parts 5 and 7 covered managed Kubernetes generally — the upgrade-specific pattern across all three major providers splits into two genuinely separate operations: control plane upgrade (fully provider-managed) and node group/pool upgrade (where the operator retains real, consequential choices).

The version skew policy from earlier in this chapter applies identically here even though the control plane upgrade itself is a "black box" — the moment the control plane finishes its provider-managed upgrade, every existing node's kubelet is now potentially several minor versions behind it, and the same "is this within the allowed 3-minor-version skew" check applies just as strictly as it would on a self-managed kubeadm cluster.

In-Place vs. Blue-Green Node Upgrades#

Every managed provider's node upgrade ultimately chooses between two fundamentally different strategies — understanding the tradeoff, not just the button to click, is what separates an operator who can debug a failed upgrade from one who's just following a wizard.

A Worked Blue-Green EKS Node Upgrade#

# 1. Create a new, target-version managed node group alongside the existing one
eksctl create nodegroup --cluster checkout-prod --name workers-v1-31 \
  --node-type m5.xlarge --nodes 5 --node-version 1.31

# 2. Cordon the OLD node group so no new pods schedule there
kubectl cordon -l eks.amazonaws.com/nodegroup=workers-v1-30

# 3. Drain old nodes gradually, letting the Scheduler place evicted
# pods onto the new, already-Ready node group
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data

# 4. Once every old node is drained and empty, confirm application
# health, THEN remove the old node group
eksctl delete nodegroup --cluster checkout-prod --name workers-v1-30

The rollback path here is genuinely simple precisely because both node groups coexist during the transition: if step 3's drained pods show any problem on the new node group, kubectl uncordon on the old node group immediately makes it schedulable again, and new pods (or a rolled-back Deployment) can land back on the known-good version without any node ever having been destroyed — this is the direct, concrete payoff of the "old node group is still fully intact" row from the comparison table above. Step 4's old-node-group deletion should be treated as a deliberate, separately-approved action, not an automatic tail end of the same script — it's the one point in this sequence that actually forecloses the rollback path this whole strategy exists to preserve.

Finding and Fixing Deprecated APIs Before Upgrading#

Every Kubernetes upgrade can remove APIs that were deprecated long enough ago — an apply that worked perfectly on the old version can fail outright on the new one if it targets a apiVersion that no longer exists, which is why checking for this before upgrading, not after, is non-negotiable.

pluto detect-helm -o wide
pluto detect-files -d ./k8s-manifests/
kubectl-convert -f old-manifest.yaml --output-version apps/v1

A concrete before/after makes kubectl convert's value obvious rather than abstract. A manifest authored years ago against a since-removed apiVersion fails outright on a newer cluster:

$ kubectl apply -f old-ingress.yaml
error: unable to recognize "old-ingress.yaml": no matches for kind "Ingress" in version "extensions/v1beta1"

$ kubectl convert -f old-ingress.yaml --output-version networking.k8s.io/v1 > new-ingress.yaml
$ kubectl apply -f new-ingress.yaml
ingress.networking.k8s.io/checkout created

This is a genuinely low-effort fix once the deprecated API is actually identified — the friction is almost always in finding every affected manifest across a large repository or Helm chart set (Pluto/kubent's actual job) rather than in performing the conversion itself, which is precisely why the scanning step matters more than the conversion step in practice. A GitOps-managed cluster (the automation-cicd-gitops tutorial series) has a real advantage here: every manifest lives in one versioned source of truth, making a repository-wide Pluto scan comprehensive by construction, rather than needing to separately track down manifests that might exist only as ad-hoc kubectl apply history with no durable record anywhere.

Alpha APIs can be removed with zero deprecation notice at all, beta APIs are guaranteed at least 9 months or 3 minor releases of deprecation notice before removal, per the Kubernetes API deprecation policy — this is precisely why depending on an alpha feature in production (Part 14's DRA caveat is a direct example) carries meaningfully more upgrade risk than depending on a stable, GA API.

Upgrading Add-ons: CNI, CSI Drivers, and Operators#

Everything in this chapter so far upgrades core Kubernetes itself — the CNI plugin (Part 3), CSI storage drivers (Part 3), the service mesh control plane (Part 4), and every Operator-managed dependency (Part 4) all have their own, entirely separate version lifecycles that a cluster-version upgrade does not touch automatically.

PodDisruptionBudgets and Drain Safety During Upgrades#

Part 2 introduced PodDisruptionBudgets as a general concept — an upgrade is the single moment they matter most, since kubectl drain (used throughout this chapter) explicitly respects them, refusing to evict a pod if doing so would violate the budget.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-service-pdb
  namespace: checkout
spec:
  minAvailable: 2       # drain will never evict below 2 healthy replicas
  selector:
    matchLabels: { app: checkout-service }

Rollback Strategy — What Can and Can't Be Undone#

Not every part of a cluster upgrade is equally reversible — knowing which parts genuinely have a clean rollback path and which don't shapes how cautiously each step should be approached.

etcd's Own Version Lifecycle#

etcd (Part 1, Part 4) has its own independent release cadence and its own version compatibility rules — a Kubernetes minor version upgrade does not automatically upgrade the etcd binary underneath it, and kubeadm only bundles a specific, tested etcd version per Kubernetes release rather than always shipping etcd's own latest release.

kubectl -n kube-system get pods -l component=etcd -o jsonpath='{.items[0].spec.containers[0].image}'
ETCDCTL_API=3 etcdctl version

etcd Backup Before Any Upgrade#

Part 4 covered etcd's disaster-recovery drill in general — a cluster upgrade is one of the specific, scheduled moments that backup exists for, and skipping it "just this once because the upgrade is routine" is exactly the shortcut that turns a routine upgrade into an unrecoverable incident.

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-pre-upgrade-$(date +%F).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-pre-upgrade-$(date +%F).db --write-out=table

This single command, run and verified immediately before kubeadm upgrade apply, is the difference between "a failed upgrade is a stressful but bounded recovery exercise" and "a failed upgrade is a potential total cluster-state loss" — given the previous section's finding that kubeadm has no supported downgrade path at all, this snapshot is not a nice-to-have companion to the upgrade, it's the only real safety net a self-managed control plane upgrade has.

Node OS and Kernel Patching — a Separate Lifecycle#

A node's underlying operating system and kernel have their own patch cadence, entirely independent of the Kubernetes version running on top of them — a cluster can be perfectly current on Kubernetes while running nodes with months of unpatched OS-level CVEs, or vice versa.

For managed node images (EKS-optimized AMIs, GKE's Container-Optimized OS, AKS node images), a Kubernetes-version node upgrade and an OS-patch node upgrade are frequently the same operation — the managed image bundles a specific, tested combination of OS, kernel, container runtime, and kubelet together, so replacing a node onto a newer managed image (the blue-green or in-place patterns from earlier in this chapter) updates all three layers at once. Self-managed nodes (Part 6) don't get this bundling for free — OS patching there is a genuinely separate maintenance task from a Kubernetes version upgrade, and needs its own explicit schedule rather than being assumed to happen "whenever Kubernetes gets upgraded."

Testing an Upgrade Before Production#

Every technique in this chapter reduces risk during the upgrade itself — testing against a realistic staging cluster first reduces the chance of discovering a problem during production upgrade at all, shifting the moment of discovery to a much cheaper, lower-stakes environment.

Fleet-Wide Upgrade Orchestration#

Part 13 introduced fleet-management tooling (Cluster API, Rancher Fleet) as the escape hatch for multi-tenancy at the separate-clusters tier — upgrade orchestration is exactly where that investment pays for itself directly, sequencing a version rollout across many clusters deliberately rather than leaving each cluster's upgrade to chance or manual tracking.

This is the direct multi-cluster analogue of the blue-green node-pool pattern from earlier in this chapter, one level up the stack — instead of validating a new node group before cutting production traffic to it, an organization running many clusters validates a new Kubernetes version on one canary cluster before rolling it to the rest of the fleet, catching a version-specific regression against one cluster's blast radius instead of discovering it simultaneously across every cluster the organization runs.

Automating the Upgrade Pipeline#

Every step this chapter has walked through manually — the deprecated-API scan, the etcd snapshot, the sequenced node upgrades, the post-upgrade verification — is mechanical enough to encode as a pipeline rather than a runbook a human executes by hand each time, and doing so is what makes the fleet-wide canary rollout from the previous section practically achievable at all.

The deprecated-API scan as an automated, run-blocking pipeline gate is the single highest-leverage automation in this list — turning "someone remembered to run Pluto before upgrading" into "the pipeline physically cannot proceed past a deprecated-API finding" removes the exact human-memory failure mode behind a large fraction of real upgrade incidents, the same way a required CI test gate removes "someone forgot to run the tests" as a failure mode for application deploys (the automation-cicd-gitops tutorial series covers this general CI/CD gating pattern in depth, applied here specifically to cluster lifecycle rather than application deployment).

A Full Worked Scenario: Upgrading the checkout Cluster End-to-End#

Every step in this final worked scenario draws directly from an earlier chapter in this series: the deprecated-API scan uses the Gateway API migration urgency from Parts 8-9, the etcd backup uses Part 4's disaster-recovery mechanics, the drain sequence respects the PodDisruptionBudgets from Part 2, and the whole sequence follows the version skew policy this chapter opened with — a fitting closing illustration that a safe production upgrade isn't a separate skill from everything else in this series, it's the disciplined application of all of it at once, under the added constraint of zero acceptable downtime. Any one of these seven steps skipped under time pressure reintroduces exactly the risk the rest of this chapter was written to eliminate.

Post-Upgrade Verification Checklist#

A cluster reporting every node Ready isn't sufficient proof an upgrade succeeded — this checklist closes the gap between "the upgrade command completed" and "the cluster is genuinely healthy."

• kubectl get nodes — every node Ready, all reporting the expected new version under kubelet version
• kubectl get --raw='/readyz?verbose' (Part 10) — every control-plane health check passing, not just the top-line node status
• Every namespace's critical workloads confirmed healthy — Ready count matches desired replicas, no unexpected CrashLoopBackOff/Pending (Part 10's diagnostic trees, run deliberately rather than waited-for)
• Add-ons (CNI, CSI, service mesh) confirmed on a version compatible with the new cluster version, per the earlier section
• pluto/kubent re-run against the now-upgraded cluster's live objects, confirming zero remaining deprecated-API usage
• A synthetic end-to-end request through the throughline system's own critical path (a real request to checkout-service, not just infrastructure-level health checks) confirmed successful
• The pre-upgrade etcd snapshot retained per the organization's backup retention policy, not deleted immediately after a successful upgrade — a regression can surface days later, well after "the upgrade went fine" was declared

Part 15 CLI Cheat Sheet#

A Worked Numeric Example: Why the Extra Rigor Pays For Itself#

The staging, PDB, and blue-green practices in this chapter all cost real engineering time — working through a concrete cost comparison makes clear why that investment is justified rather than taking it on faith.

Assume checkout-service processes $50,000 of transactions per hour at typical traffic, and a botched in-place node upgrade (no PDB, no staging validation) causes a 45-minute partial outage before the team identifies and rolls back the problem:

Compare that to the ongoing cost of doing this chapter's recommended practices properly: a staging cluster kept in version lockstep (roughly one additional cluster's infrastructure cost, likely a few hundred dollars a month for a modest environment) plus perhaps 2-3 hours of engineering time per upgrade cycle to run the pre-upgrade deprecated-API scan and PDB review. A single avoided incident of the size above pays for years of that ongoing discipline — this is the same "prevention is cheaper than the incident it prevents" logic that justifies Part 11's staged security rollouts and Part 10's troubleshooting runbook investment, applied here specifically to upgrade risk.

Quick Reference: Every Upgrade Decision in This Chapter#

A Note on Upgrade Communication#

Every technique in this chapter reduces technical risk — the remaining risk in most real upgrades is organizational: stakeholders surprised by a maintenance window, an on-call engineer unaware an upgrade is in progress when an unrelated alert fires, or a dependent team's own deploy landing mid-upgrade. A technically perfect upgrade can still produce a confusing, wasted incident-response effort if nobody outside the upgrade team knew it was happening.

None of this is unique to Kubernetes — it's the same change-management discipline that applies to any production maintenance window — but it's worth stating explicitly here, since a technically flawless upgrade that blindsides an on-call engineer still counts as a bad outcome for the team involved, regardless of how cleanly every command in this chapter actually executed.

Upgrade Cadence by Environment Tier#

Not every cluster in a fleet should upgrade on the same schedule — a deliberate lag between environment tiers is itself a risk-reduction mechanism, turning staging into a live early-warning system for production rather than just a pre-deployment checklist.

A cluster genuinely constrained to a single environment (no separate dev/staging) should still simulate this staggering — a scheduled, deliberate delay before touching production, rather than upgrading the moment a new version is available, preserves most of the benefit even without dedicated lower environments.

This staggered cadence is also what turns the fleet-wide canary rollout from earlier in this chapter into a two-layer safety net rather than a single check: staging catches a regression before it ever reaches a canary production cluster, and the canary cluster catches whatever staging's traffic pattern didn't happen to exercise.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A team upgrades their kubeadm cluster's control plane from 1.29 to 1.31 in one step, skipping 1.30 entirely, reasoning that "it's just two minor versions, it should be fine." The upgrade partially fails with confusing API-compatibility errors. What was the actual mistake?

Answer: Kubernetes and kubeadm's version skew policy explicitly does not support skipping a minor version during an upgrade — every upgrade path assumes sequential, one-minor-version-at-a-time steps, and jumping from 1.29 directly to 1.31 bypasses assumptions the upgrade tooling and the API server's own internal compatibility handling depend on. The correct path is upgrading 1.29 → 1.30 → 1.31 as two separate, sequential upgrade operations, verifying cluster health after each one before proceeding to the next.

Problem 2: After a successful control-plane upgrade via kubeadm upgrade apply, an operator notices worker node kubelets are still on the previous minor version and asks whether this is a sign the upgrade didn't fully complete. Is it?

Answer: No — this is expected and explicitly supported by the version skew policy, which allows kubelets to lag the API server by up to 3 minor versions. The control-plane-first, workers-gradually upgrade sequence in this chapter depends entirely on this tolerance; seeing older kubelet versions immediately after a control-plane upgrade is the correct intermediate state, not a failure, as long as the gap stays within the supported 3-minor-version window while workers are upgraded on their own rolling schedule.

Problem 3: A cluster owner wants the safest possible strategy for a major EKS node group upgrade involving a significantly changed node AMI, but is also under pressure to minimize infrastructure cost during the change. Which strategy should they choose, and what's the actual tradeoff they're accepting?

Answer: Blue-green node group upgrade is the safer choice for a significant AMI change specifically because it allows full validation of the new node group before any real traffic shifts onto it, and offers instant, clean rollback (the old node group remains fully intact) if something goes wrong — properties in-place rolling upgrade doesn't offer. The tradeoff being accepted is the transient cost of running two full node groups simultaneously during the validation and cutover window; the team is explicitly trading a bounded, temporary cost increase for meaningfully lower risk on a change significant enough (a new AMI, not just a routine version bump) to justify it.

Problem 4: A platform team runs 12 production clusters across different regions for different customer segments. They upgrade all 12 simultaneously over one weekend to "get it over with in one push." A version-specific regression in the new Kubernetes release affects every cluster identically at the same time. What upgrade-orchestration practice from this chapter would have limited the blast radius, and why didn't a per-cluster staging test (previous section) alone catch it?

Answer: A canary-then-wave fleet upgrade rollout (upgrade one cluster first, bake, then roll to the rest in waves) would have limited the blast radius to one cluster instead of all 12 simultaneously — a regression discovered on a single canary cluster is a contained incident; the same regression hitting all 12 clusters at once is a full outage across every customer segment simultaneously. A staging environment alone doesn't necessarily catch every production-scale regression, since staging's traffic patterns, data volume, or specific workload mix can differ enough from any single production cluster's real conditions — fleet-wide canary rollout adds a second, independent safety net by using a real production cluster as the first, contained exposure to the new version, rather than relying on staging fidelity alone.

Summary and What's Next#

Cluster upgrades are where every other discipline in this series converges at once — the version skew policy makes a phased rollout possible, PodDisruptionBudgets keep a drain safe, etcd backups are the real rollback mechanism for a self-managed control plane, and the deprecated-API discipline this chapter covered is exactly what Parts 8-9's Gateway API migration made concretely urgent. Treating an upgrade as routine, well-rehearsed maintenance — not a rare, high-stakes event — is the single idea worth carrying forward from this chapter above the rest.

Part 16 shifts from keeping a cluster current to sizing it correctly in the first place: capacity planning. Autoscaling (Part 12) reacts to load that has already arrived; Part 16 covers the slower-moving decisions — node pool shape, instance types, failure-domain headroom, growth forecasting — that determine the ceiling autoscaling reacts within. The two disciplines meet at exactly the moment this chapter's worked upgrade scenario needed extra node capacity to absorb a rolling drain — Part 16 is what makes sure that capacity was actually planned for, not discovered as a surprise mid-upgrade.