# Kubernetes Deep Dive — Part 16: Capacity Planning for Kubernetes Clusters > **Series:** Kubernetes Deep Dive (16 of 19) > **Part 1:** `01-architecture-and-control-plane.md` — Architecture & Control Plane > **Part 2:** `02-scheduling-and-workloads.md` — Scheduling & Workload Objects > **Part 3:** `03-networking-and-storage.md` — Networking (CNI) & Storage (CSI) > **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators > **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE > **Part 6:** `06-onprem-and-cluster-provisioning.md` — On-Prem & Self-Managed Kubernetes > **Part 7:** `07-eks-deep-dive.md` — Amazon EKS in Production Depth > **Part 8:** `08-gateway-api-and-envoy-gateway.md` — Gateway API & Envoy Gateway > **Part 9:** `09-gateway-api-across-providers.md` — Gateway API Across GKE, EKS, AKS & On-Prem > **Part 10:** `10-troubleshooting-kubernetes.md` — Troubleshooting Kubernetes, Systematically > **Part 11:** `11-kubernetes-security-deep-dive.md` — Kubernetes Security Deep Dive (CKS-Aligned) > **Part 12:** `12-autoscaling-hpa-vpa-keda.md` — Autoscaling: HPA, VPA, KEDA & Cluster Autoscaling > **Part 13:** `13-multi-tenancy-and-cluster-sharing.md` — Multi-Tenancy & Cluster Sharing at Scale > **Part 14:** `14-ai-ml-workloads-on-kubernetes.md` — Running AI/ML Workloads on Kubernetes > **Part 15:** `15-cluster-upgrades-and-lifecycle.md` — Cluster Upgrades & Lifecycle Management > **Part 16:** This file — Capacity Planning for Kubernetes Clusters > **Part 17:** `17-cilium-ebpf-networking.md` — Cilium & eBPF Networking Deep Dive > **Part 18:** `18-backup-and-disaster-recovery.md` — Backup & Disaster Recovery for Workloads > **Part 19:** `19-building-custom-controllers-and-operators.md` — Building Custom Controllers and Operators > **Questions:** `questions.md` Assumes you're comfortable with VPA's recommender/updater/admission-controller mechanics and HPA/Cluster Autoscaler behavior from Part 12, the QoS classes and resource requests/limits model from Part 2, GPU scheduling from Part 14, and the tenant-quota sizing math from Part 13 — this chapter is about turning all of that into a repeatable, forward-looking planning process rather than reintroducing any of it. ## Table of Contents 1. [Why Capacity Planning Deserves Its Own Chapter](#why-capacity-planning-deserves-its-own-chapter) 2. [Capacity Planning vs. Autoscaling — Where One Ends and the Other Begins](#capacity-planning-vs-autoscaling--where-one-ends-and-the-other-begins) 3. [The Overprovisioning Problem, in Real Numbers](#the-overprovisioning-problem-in-real-numbers) 4. [Right-Sizing Requests From Real Usage Data](#right-sizing-requests-from-real-usage-data) 5. [Turning VPA's Recommender Into a Planning Signal](#turning-vpas-recommender-into-a-planning-signal) 6. [Cluster-Wide Headroom — How Much Slack a Healthy Cluster Actually Needs](#cluster-wide-headroom--how-much-slack-a-healthy-cluster-actually-needs) 7. [Surviving a Failure Domain: the Node and AZ Loss Calculation](#surviving-a-failure-domain-the-node-and-az-loss-calculation) 8. [Node Pool Capacity Math — Sizing Instance Types for a Workload Mix](#node-pool-capacity-math--sizing-instance-types-for-a-workload-mix) 9. [Bin Packing and Instance-Type Fragmentation](#bin-packing-and-instance-type-fragmentation) 10. [Blending On-Demand, Reserved, and Spot Capacity](#blending-on-demand-reserved-and-spot-capacity) 11. [Forecasting Growth — Trend-Based Planning vs. Reactive-Only Autoscaling](#forecasting-growth--trend-based-planning-vs-reactive-only-autoscaling) 12. [Seasonal and Event-Driven Capacity Planning](#seasonal-and-event-driven-capacity-planning) 13. [Cost Visibility With Kubecost and OpenCost](#cost-visibility-with-kubecost-and-opencost) 14. [Capacity Planning for GPU Pools](#capacity-planning-for-gpu-pools) 15. [Capacity Planning in a Multi-Tenant Cluster](#capacity-planning-in-a-multi-tenant-cluster) 16. [Batch and CronJob Capacity — A Different Shape of Demand](#batch-and-cronjob-capacity--a-different-shape-of-demand) 17. [Storage Capacity Planning — PVs, Snapshots, and Growth](#storage-capacity-planning--pvs-snapshots-and-growth) 18. [Control-Plane Capacity Planning — the Part Everyone Forgets](#control-plane-capacity-planning--the-part-everyone-forgets) 19. [Building a Recurring Capacity Review Process](#building-a-recurring-capacity-review-process) 20. [A Full Worked Scenario: Sizing `checkout`'s Node Pool for a Product Launch](#a-full-worked-scenario-sizing-checkouts-node-pool-for-a-product-launch) 21. [A Full Worked Scenario: A Reactive-Only Team Gets Caught Out](#a-full-worked-scenario-a-reactive-only-team-gets-caught-out) 22. [A Full Worked Scenario: Consolidating an Overprovisioned Fleet](#a-full-worked-scenario-consolidating-an-overprovisioned-fleet) 23. [Part 16 CLI Cheat Sheet](#part-16-cli-cheat-sheet) 24. [A Capacity Review Checklist](#a-capacity-review-checklist) 25. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps) 26. [Worked Practice Problems](#worked-practice-problems) 27. [Summary and What's Next](#summary-and-whats-next) --- ## Why Capacity Planning Deserves Its Own Chapter **Autoscaling (Part 12) answers "how do I react to load that's already here"; capacity planning answers a different, harder question — "how much capacity should exist at all, before load arrives, so that reacting to it is even possible."** These are not the same skill, and treating them as one is the single most common gap in an otherwise well-autoscaled cluster: an HPA can only scale a Deployment up to the replica count a node pool actually has room for, and a Cluster Autoscaler can only add a node as fast as the cloud provider's API and the node's own boot time allow — neither one manufactures capacity instantly, and neither one decides, on its own, whether the *ceiling* they're scaling toward is the right one. This chapter treats capacity planning as its own discipline: reading real usage data instead of guessing at requests and limits, sizing node pools deliberately instead of accreting them accidentally, planning for the loss of an entire failure domain, and building the recurring review habit that keeps a cluster's shape matched to its actual workload over time rather than drifting silently for a year until a bill or an outage forces the conversation. The throughline system carries over unchanged: `checkout-service` (namespace `checkout`), `catalog-service` (namespace `catalog`), `inventory-service` (namespace `inventory`), and the `recommendations` team's GPU-backed `recommendation-model` (namespace `recommendations`, introduced in Part 14) — this chapter treats all four as one platform team's actual capacity-planning workload. > [!NOTE] > This chapter is deliberately platform/SRE-facing, not application-facing. Part 12 already covers how an > individual workload scales itself; this chapter covers how the *cluster underneath* every workload is > sized, funded, and kept honest over time — the conversation a platform team has with finance and with > itself, not the one an application team has about its own Deployment. ## Capacity Planning vs. Autoscaling — Where One Ends and the Other Begins **The clearest way to separate the two is by time horizon: autoscaling operates on a horizon of seconds to minutes and reacts to a metric that has already moved; capacity planning operates on a horizon of weeks to quarters and decides what the autoscaler is allowed to scale into.** ```mermaid flowchart LR Plan["Capacity planning
(weeks-to-quarters horizon):
node pool shape, instance types,
quota ceilings, budget"] --> Bound["Sets the ceiling
autoscalers operate inside"] Bound --> HPA["HPA: reacts to load,
seconds-to-minutes horizon"] Bound --> CA["Cluster Autoscaler/Karpenter:
reacts to unschedulable pods,
minutes horizon"] Bound --> VPA["VPA: reacts to observed
usage drift, hours-to-days horizon"] classDef plan fill:#f0e9fb,stroke:#6d43c0,color:#10161c classDef react fill:#e5f0fa,stroke:#1d6fb8,color:#10161c class Plan,Bound plan class HPA,CA,VPA react ``` **Neither layer can substitute for the other.** An autoscaling-only strategy with no capacity planning behind it eventually hits a real ceiling — a node pool's max size, a cloud account's instance-family quota, a budget nobody sized against actual growth — and fails exactly when load is highest and margin for error is lowest. A capacity-planning-only strategy with no autoscaling wastes money on permanently-provisioned headroom that autoscaling would have reclaimed automatically during quiet hours. The two are complementary layers of the same problem, one setting the boundary and the other operating inside it. | Question | Layer that answers it | |---|---| | "Should this Deployment have 3 replicas or 12 right now?" | HPA (Part 12) | | "Is this pod's memory request still accurate given the last month of usage?" | VPA recommender (Part 12), fed by this chapter's review cadence | | "Does the `checkout` node pool have room to scale to 12 replicas at all?" | Capacity planning (this chapter) | | "What instance type and how many nodes do we provision for Q3's projected traffic?" | Capacity planning (this chapter) | | "Can this cluster survive losing one availability zone without dropping requests?" | Capacity planning (this chapter) | ## The Overprovisioning Problem, in Real Numbers **The dominant real-world capacity-planning failure isn't running out of room — it's the opposite, and it's gotten measurably worse, not better, as Kubernetes adoption has matured.** Recent fleet-wide measurements found CPU overprovisioning across production clusters climbing from roughly 40% to 69% year over year, with memory overprovisioning reaching close to 79% — real utilization sitting at single-digit percentages even as Kubernetes adoption itself reached roughly 80% of production workloads. GPU pools are worse still: fleet-wide GPU utilization averaging around 5%, even though a well-tuned cluster can sustain 40-50%+ (Part 14 covers why GPU scheduling in particular fragments this badly). ```mermaid xychart-beta title "Typical fleet-wide CPU: requested vs. actually used" x-axis ["checkout", "catalog", "inventory", "recommendations"] y-axis "CPU cores" 0 --> 40 bar "Requested (reserves capacity)" [24, 18, 12, 30] bar "p95 actual usage" [9, 6, 4, 11] ``` **This gap matters for two distinct reasons, not one.** First, the obvious one: every core requested but unused is a core the organization is paying for and getting nothing from. Second, the less obvious one this chapter cares about more: a cluster whose requests don't reflect real usage makes every capacity-planning calculation downstream of it — node pool sizing, failure-domain headroom, forecasted growth — wrong in the same direction, because all of those calculations start from requested capacity, not actual capacity. Fixing the request-vs-usage gap isn't just a cost optimization; it's a precondition for every other calculation in this chapter being trustworthy. > [!IMPORTANT] > **Best Practice**: Treat overprovisioned requests as a data-quality problem before treating capacity > planning as a math problem. Sizing a node pool, a failure-domain buffer, or a growth forecast against > requests that are 3-8x real usage produces a plan that's wrong by the same 3-8x factor — no amount of care > in the forecasting math downstream fixes an input that was never accurate to begin with. ## Right-Sizing Requests From Real Usage Data **The correct starting point for any capacity plan is the same starting point Part 12's VPA recommender uses internally: observed usage, not a guess copy-pasted from a template chart.** A team writing a Deployment's resource requests for the first time genuinely has no usage data to work from — a reasonable default (100m CPU / 128Mi memory request for a typical Go/Node service, roughly 250m CPU / 512Mi memory for a JVM service) is a legitimate starting point, but it is a placeholder, not a target, and it must be revisited against real data within the first few weeks of production traffic. ```bash # One week of real usage is the minimum useful observation window — # shorter windows miss weekly traffic cycles (weekday vs. weekend load) kubectl top pod -n checkout --containers # For a real distribution rather than a single point-in-time snapshot, # query the metrics pipeline (Prometheus) directly for percentiles: ``` ```promql quantile_over_time(0.95, container_memory_working_set_bytes{namespace="checkout", pod=~"checkout-service-.*"}[7d]) ``` | Percentile | What it tells you | Where to use it | |---|---|---| | p50 (median) | Typical steady-state usage | Rarely the right number to set a request from — half of all observations exceed it | | p95 | A safe request baseline that tolerates normal variance without triggering CPU throttling or memory pressure | The standard starting point for a `request` value | | p99 | Captures rarer spikes | A reasonable floor for a `limit`, or the request itself for a latency-sensitive service with no tolerance for throttling | | max observed | The single worst moment in the window | Useful context, but sizing a permanent request off one outlier reintroduces the overprovisioning problem this section exists to fix | **Explaining the throttling trap two levels deep, since it's the single most common outcome of guessing low on CPU**: the symptom is `checkout-service` showing elevated p99 latency under load despite `kubectl top` reporting CPU usage comfortably under its limit. The immediate cause is CPU throttling — the kernel's CFS bandwidth controller enforces the `limit` over a rolling 100ms quota period, not as a smooth ongoing cap, so a bursty container can exhaust its entire period's quota in a few milliseconds and sit idle, throttled, for the rest of the window, an effect invisible to any metric that only samples average usage. The underlying condition is that the request/limit pair was set from a single load test's average CPU reading rather than from a percentile distribution capturing real burst behavior — averaging hides exactly the bursts that trigger throttling, which is why `container_cpu_cfs_throttled_periods_total` (not raw CPU usage) is the metric that actually confirms this failure mode. > [!WARNING] > A container can show low *average* CPU usage and still be heavily throttled — throttling is a function of > burst behavior within each 100ms accounting period, not average consumption over minutes. Always check > `container_cpu_cfs_throttled_periods_total` alongside `container_cpu_usage_seconds_total` before concluding > a CPU limit is "plenty," especially for latency-sensitive request-handling services with naturally spiky > per-request CPU profiles. ## Turning VPA's Recommender Into a Planning Signal **Part 12 covered VPA's `updateMode` for actually applying recommendations to running pods — this chapter uses the same recommender component for a different purpose: as a standing, always-current data source for capacity planning, even on workloads where automatic updates (`Auto`/`Recreate`) are deliberately not enabled.** ```yaml apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: checkout-service-capacity-signal namespace: checkout spec: targetRef: apiVersion: apps/v1 kind: Deployment name: checkout-service updatePolicy: updateMode: "Off" # never touches the running pods — pure recommendation signal ``` ```bash kubectl describe vpa checkout-service-capacity-signal -n checkout | grep -A8 "Recommendation:" ``` **Running VPA in `Off` mode across every Deployment in a cluster, purely for its recommendation output, is a deliberately low-risk way to get the section above's percentile-based sizing without writing a single PromQL query by hand** — the recommender already does that percentile calculation internally, using a decaying histogram of real observed usage. A platform team can then aggregate every workload's VPA recommendation once a quarter into a single spreadsheet or dashboard, comparing current requests against recommended requests cluster-wide, which becomes the primary input to the node-pool sizing math later in this chapter. | Node pool sizing input | Where it comes from | |---|---| | Per-workload right-sized requests | VPA recommender output (`Off` mode), aggregated across every Deployment | | Expected replica count under normal load | HPA `minReplicas`/`maxReplicas` and its own historical scaling pattern (Part 12) | | Expected growth over the planning horizon | Trend-based forecasting (later in this chapter) | | Required failure-domain headroom | The AZ/node-loss calculation (later in this chapter) | > [!TIP] > **Best Practice**: Run VPA in `Off` mode on every production Deployment as a baseline, even ones you have > no intention of letting VPA automatically update yet. The recommendation signal is valuable on its own, > completely independent of whether you ever flip `updateMode` to something that acts on it — treat "install > the recommender" and "let it touch running pods" as two separate decisions, not one. ## Cluster-Wide Headroom — How Much Slack a Healthy Cluster Actually Needs **A cluster running at 100% of its allocatable capacity has zero room to schedule a replacement pod when a node fails, zero room for the Cluster Autoscaler to add nodes before new pods are already failing to schedule, and zero room to absorb a legitimate short-term traffic spike without immediately going `Pending`.** Cluster-wide headroom is capacity deliberately held in reserve, above what current workloads request, for exactly these moments. ```mermaid flowchart TD Total["Total cluster
allocatable capacity"] --> Workload["Committed to running
workloads' requests"] Total --> Headroom["Deliberate headroom"] Headroom --> Failure["Failure-domain buffer:
survives losing 1 node
(or 1 AZ) without
evicting anything"] Headroom --> Burst["Burst buffer: absorbs a
real traffic spike while
the Cluster Autoscaler
provisions new nodes"] Headroom --> Upgrade["Upgrade buffer: room for
Part 15's rolling node
replacement during upgrades"] classDef committed fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef headroom fill:#e5f5ea,stroke:#1f8a4c,color:#10161c class Workload committed class Headroom,Failure,Burst,Upgrade headroom ``` **Each headroom category exists to answer a different failure mode, and sizing them together (rather than picking one arbitrary "headroom percentage" and calling it done) is what makes the number defensible in a capacity review.** The failure-domain buffer alone is covered in depth in the next section since it has an exact, computable answer; the burst and upgrade buffers are typically expressed as a percentage of steady -state committed capacity, informed by how bursty a workload mix actually is and how long node provisioning takes in that specific cloud/region. | Buffer type | Typical sizing approach | |---|---| | Failure-domain buffer | Computed exactly from N-1 node or 1-AZ loss — see the next section | | Burst buffer | Percentage of committed capacity (commonly 15-30%), sized from how bursty real traffic actually is, cross-referenced against the Cluster Autoscaler's actual node-provisioning latency for that instance type/region | | Upgrade buffer | At minimum, enough to run one extra node's worth of pods during a Part 15 rolling node replacement — often satisfied by the failure-domain buffer already covering this case | > [!NOTE] > Headroom is not idle waste the way an overprovisioned individual pod's request is — the overprovisioning > problem earlier in this chapter is about requests that don't reflect real *workload* usage; headroom is a > deliberate, sized, cluster-level reserve that exists specifically so an unplanned event doesn't turn into > an outage. Don't let a cost-optimization initiative eliminate headroom by mistaking it for the same kind of > waste right-sizing individual requests fixes. ## Surviving a Failure Domain: the Node and AZ Loss Calculation **This is the one headroom category with an exact, computable answer rather than a judgment call — decide which failure domain the cluster must survive losing (one node, or one entire availability zone), then size committed capacity so the *remaining* domains alone can still hold every workload's requests.** ``` Example: checkout, catalog, and inventory namespaces spread across 3 AZs, each currently requesting a combined 60 CPU cores at steady state. N-1 AZ loss requirement: the remaining 2 AZs must together hold all 60 cores. => Each AZ must be sized to hold up to 30 cores alone in the worst case (if load isn't perfectly even across AZs, size for the busiest AZ's share, not a naive 1/3 split). If today's 3-AZ cluster is sized at 20 cores of capacity per AZ (60 total, evenly split with zero headroom), losing one AZ leaves 40 cores of capacity for 60 cores of requested workload — a guaranteed scheduling failure for roughly a third of pods the moment that AZ goes down. Correctly sized for AZ-loss survival: each AZ provisioned with 30 cores of capacity (90 cores total, 50% more than steady-state requests) — losing any one AZ still leaves 60 cores across the remaining two, exactly enough to reschedule everything. ``` ```mermaid flowchart LR subgraph AZa["AZ-a: 30 cores"] Pa["20 cores
of pods"] end subgraph AZb["AZ-b: 30 cores"] Pb["20 cores
of pods"] end subgraph AZc["AZ-c: 30 cores — LOST"] Pc["20 cores of pods
need rescheduling"] end Pc -.->|"reschedules into
remaining headroom"| AZa Pc -.->|"reschedules into
remaining headroom"| AZb classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c class AZa,AZb ok class AZc crit ``` **The formula generalizes cleanly: for surviving the loss of 1 of N failure domains, each domain needs capacity for `total-requested-capacity / (N - 1)`, not `total-requested-capacity / N`.** For 3 AZs that's a 50% headroom requirement over a naive even split; for 4 AZs it drops to roughly 33%; the fewer failure domains a cluster spans, the more headroom each one needs to carry to survive losing any single one — which is itself a real argument for spreading across more AZs when the workload and latency budget tolerate it. > [!CAUTION] > `PodDisruptionBudget`s and topology spread constraints (Part 2) only control *how evenly* pods are placed > and *how many can be voluntarily disrupted at once* — they do nothing to guarantee the *capacity* exists to > reschedule pods after an involuntary AZ loss. A perfectly configured topology spread constraint across 3 > AZs with zero failure-domain headroom still produces a scheduling failure the moment one AZ actually goes > down; the constraint controls placement, this section's math controls whether placement is even possible > afterward. ## Node Pool Capacity Math — Sizing Instance Types for a Workload Mix **Once total required capacity (including headroom) is known, the remaining question is which instance type, and how many of them, actually deliver it — and different instance types don't just cost differently, they bin-pack differently against a given workload's request shape.** ``` Workload mix for the catalog node pool: 24 pods, each requesting 2 CPU / 4Gi memory (a 1:2 CPU-to-memory ratio). Option A — general-purpose instance, 8 vCPU / 32Gi (a 1:4 ratio): CPU-bound: 8 vCPU / 2 CPU per pod = 4 pods/node by CPU Memory-bound: 32Gi / 4Gi per pod = 8 pods/node by memory Binding constraint: CPU (4 pods/node) — memory is left stranded, since only 16Gi of the 32Gi is ever used at the CPU-bound pod count. Nodes needed: 24 / 4 = 6 nodes. Option B — compute-optimized instance, 8 vCPU / 16Gi (a 1:2 ratio, matching the workload's own ratio): CPU-bound: 4 pods/node. Memory-bound: 16Gi / 4Gi = 4 pods/node. Both constraints agree — no resource is stranded. Nodes needed: 24 / 4 = 6 nodes — same node count, but Option B's instance type is typically priced lower per-core for a memory-to-CPU ratio the workload doesn't actually need, since Option A is paying for 16Gi of memory per node that never gets used. ``` **The general lesson**: match the node pool's own CPU:memory ratio to the *actual* ratio of the workloads that will run on it, not to whichever instance family happens to be the organization's historical default. A workload mix skewed heavily toward one resource (CPU-heavy batch processing, memory-heavy caching tiers) stranding capacity on a mismatched instance type is a quiet, recurring cost identical in effect to the overprovisioned-request problem earlier in this chapter, except it happens at the node level instead of the pod level. | Signal | Suggests | |---|---| | Nodes consistently hit their memory-bound pod ceiling with CPU still idle | Switch to a higher-memory-ratio instance type, or a workload mix needs review | | Nodes consistently hit their CPU-bound pod ceiling with memory still idle | Switch to a higher-CPU-ratio (compute-optimized) instance type | | Both ceilings land close together across most nodes | The instance type is well-matched to the workload mix — the ideal state | ## Bin Packing and Instance-Type Fragmentation **Even a well-matched instance type can fragment badly if a node pool mixes very differently-sized pods, because the scheduler's bin-packing is NP-hard in the general case and settles for a good-enough placement, not a provably optimal one.** ```mermaid flowchart TD Mixed["Node pool mixing large
(4 CPU) and small
(0.25 CPU) pod requests"] --> Frag["Large pods leave small,
oddly-shaped gaps behind
on partially-filled nodes"] Frag --> Waste["Gaps too small for another
large pod, but the node
isn't scaled down because
small pods still occupy it"] Waste --> Fix1["Separate node pools by
request size, or"] Waste --> Fix2["A bin-packing-aware
autoscaler (Karpenter's
consolidation) that actively
repacks and drains"] classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c class Mixed,Frag,Waste warn class Fix1,Fix2 ok ``` **This exact fragmentation pattern is worse, not just similarly bad, for GPU pools** — a pod requesting 2 whole GPUs cannot be scheduled even when the cluster has 3 GPUs free in total, if those 3 free GPUs are spread as single units across 3 different nodes rather than concentrated on one. The Cluster Autoscaler often doesn't even recognize this as a capacity shortage worth scaling for, because in aggregate the cluster genuinely isn't "out of GPUs" — it's out of GPUs *in the shape this specific pod needs them*. This chapter's [GPU section](#capacity-planning-for-gpu-pools) below covers the direct mitigation. > [!TIP] > **Best Practice**: Use consistent request sizes within a single node pool wherever the workload allows it, > and split genuinely different-sized workloads (a fleet of small stateless API pods vs. a handful of large > batch-processing pods) into separate node pools rather than one mixed pool. Consolidating autoscalers like > Karpenter actively repack and drain underfilled nodes over time, but starting from a workload mix that > doesn't fragment in the first place needs far less active repacking to stay efficient. ## Blending On-Demand, Reserved, and Spot Capacity **Everything so far has assumed a single, uniform kind of node — real capacity planning also decides *how* that capacity is purchased, since on-demand, reserved/committed-use, and spot instances differ enormously in both price and the failure characteristics a capacity plan has to design around.** ```mermaid quadrantChart title Price vs. interruption risk by purchase model x-axis Cheap --> Expensive y-axis Frequent interruption --> Never interrupted quadrant-1 Premium, stable quadrant-2 Rarely the right default quadrant-3 Best value for interruptible work quadrant-4 Sweet spot for steady-state baseline "Spot/preemptible": [0.15, 0.15] "On-demand": [0.75, 0.85] "Reserved/committed-use": [0.45, 0.85] ``` | Purchase model | Discount vs. on-demand | Interruption behavior | Best fit | |---|---|---|---| | On-demand | None (baseline price) | Never interrupted by the provider | Burst headroom, workloads with zero tolerance for eviction | | Reserved / committed-use | Commonly 30-60%, in exchange for a 1-3 year usage commitment | Never interrupted — identical availability to on-demand | The steady-state floor of a capacity plan — the portion of capacity that's genuinely always running | | Spot / preemptible | Commonly 60-90% | Can be reclaimed by the provider with as little as 30-120 seconds' notice | Stateless, horizontally-replicated, interruption-tolerant workloads; batch/training jobs that checkpoint | **The capacity-planning discipline here is deciding what fraction of the plan built in earlier sections goes on each tier, not treating the choice as purely a procurement afterthought.** A common, defensible split: size reserved/committed-use capacity to cover the steady-state baseline from the forecasting sections above (since that portion is, by definition, always running and benefits most from a multi-year discount), size on-demand to cover the failure-domain and burst headroom from earlier in this chapter (since headroom must be guaranteed available, not reclaimable), and route genuinely interruption-tolerant workloads — batch processing, CI runners, a subset of stateless replicas behind an HPA with enough minimum on-demand replicas to stay available during a mass spot reclamation — onto spot capacity for the largest cost reduction. ```yaml # A Karpenter NodePool expressing this split for one workload tier — # spot preferred, falling back to on-demand only if spot is unavailable apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: catalog-batch-interruptible spec: template: spec: requirements: - key: karpenter.sh/capacity-type operator: In values: ["spot", "on-demand"] # tries spot first, falls back automatically ``` > [!WARNING] > Never place the failure-domain headroom this chapter's N-1 calculation depends on entirely on spot > capacity — a spot reclamation event and an actual AZ failure can coincide (cloud providers frequently > reclaim spot capacity across a whole region during genuine demand spikes, which correlates with exactly > the kind of event that also stresses on-demand capacity), and headroom that can itself disappear on short > notice isn't headroom at all. Keep the specific capacity earmarked for failure-domain survival on > reserved/on-demand tiers, and treat spot exclusively as cost optimization on top of an already-solid > on-demand/reserved floor. > [!TIP] > **Best Practice**: For interruption-tolerant workloads on spot, always run enough `minReplicas` on > guaranteed (reserved/on-demand) capacity to stay available through a full spot reclamation event, and use > `PodDisruptionBudget`s (Part 2/13) so a mass spot interruption doesn't evict every replica of a workload > simultaneously — the same disruption-budget discipline that protects against a voluntary node drain applies > equally to an involuntary, provider-initiated spot reclamation. ## Forecasting Growth — Trend-Based Planning vs. Reactive-Only Autoscaling **A cluster that only ever reacts to the Cluster Autoscaler noticing unschedulable pods is, by construction, always at least one provisioning cycle behind real demand — trend-based forecasting exists to get ahead of that lag for demand that's predictable in advance, reserving pure reactive autoscaling for the genuinely unpredictable remainder.** ```mermaid xychart-beta title "checkout-service requested CPU: 6-month trend vs. reactive-only capacity" x-axis ["M1", "M2", "M3", "M4", "M5", "M6"] y-axis "CPU cores" 0 --> 60 line "Actual demand trend" [18, 21, 24, 30, 38, 48] line "Reactive-only capacity (always one step behind)" [18, 18, 21, 24, 30, 38] ``` **The gap between the two lines is the real cost of reactive-only planning**: at any given month, capacity is sized to what demand *was* one cycle ago, not what it *is* now — a gap that's tolerable when growth is slow and autoscaling latency is low, and genuinely dangerous when growth accelerates faster than node provisioning can keep pace, which is exactly the moment a business cares most about not having an outage (a product launch, a marketing campaign, a viral traffic event). | Approach | Strength | Weakness | |---|---|---| | Reactive-only (Cluster Autoscaler/Karpenter/HPA alone) | Zero manual forecasting effort; automatically right-sizes to whatever demand actually shows up | Structurally always behind real demand by at least one provisioning cycle; fails hardest exactly when growth is fastest | | Trend-based forecasting (linear/seasonal regression on historical usage) | Gets capacity in place *before* demand arrives, for the predictable component of growth | Requires real historical data and a recurring review process; doesn't help with genuinely novel, unpredictable spikes | | Both together (this chapter's recommendation) | Forecasting handles the predictable trend; reactive autoscaling handles the residual, unpredictable variance around that trend | Requires actually running the review process below on a cadence, not a one-time exercise | **A simple, honest forecasting method beats no forecasting method at all**: plot 6-12 months of requested (right-sized, per the earlier sections) capacity per namespace/team, fit a linear or simple seasonal trend line, and provision node pool ceilings for the *projected* number at the end of the next planning horizon — not the current number. This doesn't need to be sophisticated machine learning to be far better than pure reactive autoscaling; the improvement over doing nothing is what matters, not the sophistication of the model. ## Seasonal and Event-Driven Capacity Planning **Growth-trend forecasting captures gradual change; seasonal and event-driven planning captures the sharp, recurring or one-off spikes a smooth trend line misses entirely — a marketing campaign, a known peak shopping period, a product launch date.** ```mermaid gantt title Capacity ramp for a known Q4 peak shopping event dateFormat YYYY-MM-DD section Planning Forecast reviewed, node pool ceilings raised :done, plan1, 2026-10-01, 14d Load test against raised ceilings :done, plan2, 2026-10-15, 7d section Ramp Pre-scale node pools ahead of event :active, ramp1, 2026-11-20, 5d Peak event window :crit, peak1, 2026-11-25, 4d section Wind-down Scale back to steady-state ceilings :wind1, 2026-11-29, 7d ``` **Pre-scaling ahead of a known event deliberately trades a few days of paying for unused headroom against the far more expensive alternative of the Cluster Autoscaler racing to catch up in real time during the highest-stakes traffic window of the year.** This is one of the few places in this chapter where the "never overprovision" instinct from earlier sections is deliberately overridden — the cost of temporary, known-duration overprovisioning during a defined peak window is cheap insurance compared to the cost of an outage during that same window. > [!IMPORTANT] > **Best Practice**: Load-test against the *raised* ceilings before the actual event, not just raise the > ceilings and hope. A node pool sized correctly on paper for projected peak load can still fail in practice > if a downstream dependency (a database, a third-party API, an internal service that wasn't part of the > capacity review) can't actually sustain the same multiplier — capacity planning for one service in > isolation is incomplete if its dependencies weren't planned for the same event. ## Cost Visibility With Kubecost and OpenCost **Every calculation in this chapter — right-sizing, headroom, node pool shape, forecasting — ultimately resolves to a dollar figure, and Kubernetes has no native mechanism for attributing a shared cluster's bill back to the namespaces, teams, or workloads actually consuming it.** OpenCost (the CNCF-hosted open-source engine) and Kubecost (the commercial product built on top of it) exist specifically to close this gap, tracing real cost — not just requested resources — down to the namespace, label, or workload level. ```bash # OpenCost's allocation API — real cost, not just requested capacity, # attributed per namespace over a trailing window curl -s "http://opencost.opencost.svc.cluster.local:9003/allocation/compute?window=7d&aggregate=namespace" | \ jq '.data[] | to_entries[] | {namespace: .key, totalCost: .value.totalCost}' ``` | Tool | Best fit | |---|---| | OpenCost | Single-cluster teams needing real per-namespace/per-workload cost attribution without a commercial license | | Kubecost | Multi-cluster teams needing centralized governance, longer metric retention, and cross-cluster reporting on top of the same OpenCost engine | | Per-namespace `ResourceQuota` usage (Part 13) | A free, rougher proxy — attributes *requested*, not actual, cost, and doesn't account for shared control-plane or node-level overhead | **The distinction between requested-based and actual-usage-based cost attribution matters directly for this chapter's overprovisioning problem**: a namespace whose actual cost (from Kubecost/OpenCost) is far below its Part 13-style ResourceQuota-based cost estimate is exactly the namespace this chapter's right-sizing sections should target first — cost tooling turns "we suspect we're overprovisioned somewhere" into a ranked, namespace-by-namespace list of exactly where the waste actually is. > [!TIP] > **Best Practice**: Feed Kubecost/OpenCost's per-namespace cost data directly into the recurring capacity > review process (later in this chapter) as a standing input, not a one-off audit. A namespace that looked > correctly sized during onboarding can drift into significant waste six months later as traffic patterns or > code changes shift its real usage — cost visibility only earns its keep as a continuously reviewed signal. ## Capacity Planning for GPU Pools **GPU capacity planning inherits every general principle above and adds one the general case doesn't have: GPUs are indivisible by default (Part 14's device-plugin model), so fragmentation and utilization waste are both structurally worse than for CPU/memory.** ``` recommendations team's recommendation-model: currently 4 replicas, each requesting 1 full GPU, on a node pool of A100 nodes with 8 GPUs each (2 nodes, 16 GPUs total, 12 GPUs idle at steady state). Fleet-wide GPU utilization observed at 8% — consistent with the industry-wide ~5% average this chapter's research found, and far below what MIG/time-slicing (Part 14) could sustain on the same hardware. ``` | GPU capacity lever | What it does | Cross-reference | |---|---|---| | MIG (Multi-Instance GPU) partitioning | Splits one physical GPU into several right-sized virtual GPUs for workloads that don't need a full one | Part 14's GPU sharing strategies | | Consistent per-pool GPU request sizes | Prevents the single-GPU-vs-multi-GPU fragmentation failure mode from the bin-packing section above | This chapter, bin-packing section | | Separate node pools for single-GPU and multi-GPU workloads | Structurally prevents fragmentation rather than relying on the scheduler to avoid it | This chapter, bin-packing section | | Treating GPU saturation, fragmentation, and queue depth as first-class autoscaling signals | Moves GPU autoscaling beyond a pod-count heuristic toward a workload-level guarantee model (gang scheduling, Kueue-style queueing — Part 14) | Part 14's gang scheduling section | **Forecasting GPU capacity specifically needs a longer lead time than CPU/memory forecasting**, because GPU node provisioning is frequently gated by cloud-provider capacity reservations or quota increases that take weeks, not the minutes a standard compute node takes to provision — a GPU capacity plan discovered to be short during the forecasting review in the earlier section needs to trigger a reservation request immediately, not after the shortfall is already visible in production. > [!WARNING] > A pod requesting 2 GPUs can fail to schedule even when the cluster reports 4 or more GPUs "free" in total, > if those free GPUs are fragmented as single units across separate nodes — and because the cluster genuinely > isn't out of GPUs in aggregate, the Cluster Autoscaler often does not treat this as a scaling trigger at > all. This produces a confusing symptom: a `Pending` GPU pod, plenty of "free" GPU capacity in > `kubectl describe node`, and no new node provisioned — the fix is the same as the general bin-packing > section above (consistent request sizes, separate pools per GPU-count tier), applied specifically to the > resource that fragments worst. ## Capacity Planning in a Multi-Tenant Cluster **Part 13 covered sizing a single tenant's `ResourceQuota` from its own HPA ceiling — cluster-wide capacity planning is the aggregate version of that same exercise across every tenant sharing the cluster, plus the headroom this chapter adds on top.** ``` Cluster total capacity plan = sum of every tenant's right-sized, HPA-ceiling-aware ResourceQuota (Part 13's worked example, repeated per tenant) + failure-domain headroom (this chapter) + burst headroom (this chapter) - any expected overlap from tenants whose peak load windows genuinely never coincide (a real, defensible reduction, not a guess — confirm via actual historical peak-timing data per tenant before assuming peaks don't overlap). ``` **The "peaks never overlap" reduction deserves real scrutiny before being trusted** — it's a legitimate way to avoid needlessly summing every tenant's absolute peak as if they all happen simultaneously, but it's also an easy place to under-provision by assuming independence that doesn't actually hold. A retail `checkout` namespace and a `recommendations` namespace serving the same storefront very plausibly peak at exactly the same moment (a flash sale drives both checkout volume and recommendation-serving volume together), which means treating their peaks as independent would understate combined peak demand precisely when it matters most. | Sizing input | Per-tenant (Part 13) | Cluster-wide (this chapter) | |---|---|---| | Steady-state requests | Right-sized per workload | Summed across every tenant | | Growth headroom | Tenant's own HPA ceiling | Cluster-wide forecast trend, this chapter | | Failure-domain survival | Not addressed at the tenant level | Addressed once, cluster-wide, since a node/AZ loss affects every tenant simultaneously | | Peak-timing correlation | Not relevant at single-tenant scope | Must be checked explicitly — see above | ## Batch and CronJob Capacity — A Different Shape of Demand **Everything so far has implicitly assumed a long-running Deployment's steady-plus-burst demand shape — Jobs and CronJobs (Part 2, Part 10's troubleshooting angle) demand capacity in short, scheduled bursts instead, and sizing a node pool for them the same way as a long-running service either wastes money holding permanent capacity idle between runs, or causes the exact `Pending`-pod scheduling failures Part 10 already covers if the burst is undersized.** ```mermaid gantt title inventory's nightly reconciliation Job vs. a long-running Deployment's demand shape dateFormat HH:mm axisFormat %H:%M section Long-running Deployment Steady baseline demand, all day :active, dep1, 00:00, 24h section Batch Job Idle :done, idle1, 00:00, 2h Nightly reconcile — sharp burst :crit, job1, 02:00, 1h Idle :done, idle2, 03:00, 21h ``` **The capacity-planning answer for this shape is almost always "scale the node pool to zero (or near-zero) between runs and let the Cluster Autoscaler/Karpenter provision burst capacity on demand," rather than holding permanent headroom for a workload that's only actually running a small fraction of each day.** This is one of the few places in this chapter where the right answer inverts the earlier failure-domain-headroom guidance: a batch workload with a generous completion deadline (`activeDeadlineSeconds`, Part 10) can tolerate the provisioning latency of scaling up from zero in a way a customer-facing service's failure-domain buffer cannot. | Batch capacity question | Answer differs from long-running services because... | |---|---| | Should this pool hold permanent headroom? | Usually no — provisioning latency is tolerable if the Job's deadline has slack, so scale-to-zero between runs is the cost-correct default | | What instance type fits best? | Batch/compute-heavy Jobs often skew far more CPU-heavy than the general workload mix — apply this chapter's CPU:memory-ratio matching to the batch pool specifically, not the cluster's general-purpose pool | | Is spot capacity appropriate? | Frequently yes, more so than for request-serving workloads — a batch Job that checkpoints or can simply be retried from `backoffLimit` (Part 10) tolerates a spot reclamation far better than an in-flight user request does | | How does concurrent-run overlap affect sizing? | `concurrencyPolicy: Forbid`/`Replace` (Part 10) caps concurrent demand at exactly one run's worth; `Allow` (the default) can transiently double demand if two runs overlap, which must be sized for explicitly if left at the default | > [!TIP] > **Best Practice**: Give batch/CronJob node pools their own dedicated taint/toleration pairing (Part 2) > separate from request-serving workloads, sized and purchased (spot-heavy, scale-to-zero-capable) completely > differently from the steady-state pools this chapter's earlier sections cover — mixing batch and > long-running workloads in the same pool reintroduces the bin-packing fragmentation problem from earlier in > this chapter, since the two have fundamentally different demand shapes over time even when their > instantaneous resource requests look similar. ## Storage Capacity Planning — PVs, Snapshots, and Growth **Every section so far has been compute-shaped — storage capacity planning follows a genuinely different growth curve, because a `PersistentVolumeClaim` (Part 3) almost never shrinks on its own even as the workload behind it scales up and down, and unbounded log/data growth inside a volume is a slow-burning capacity failure a purely compute-focused review will never catch.** ```mermaid flowchart TD Provision["PVC provisioned
at initial estimated size"] --> Grow["Data grows over time
(logs, uploads, DB tables)"] Grow --> Check{"Storage usage
vs. PVC capacity?"} Check -->|"Approaching full"| Expand["CSI volume expansion
(Part 3) — online resize
if the StorageClass allows it"] Check -->|"Comfortable headroom"| Continue["Continue monitoring —
re-check on the same
cadence as compute review"] Expand --> Snapshot["Snapshot growth (Part 3's
CSI snapshot support) adds
its own separate storage
consumption over time"] classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c class Check,Expand,Snapshot warn class Continue ok ``` **Two growth curves need separate tracking, not one**: the primary volume's own data growth (which `kubectl get pvc` combined with the CSI driver's usage metrics can track directly), and the accumulated storage consumed by snapshots and backups (Part 18 covers backup/DR strategy in depth) — a snapshot retention policy that keeps 90 days of daily snapshots for a rapidly-changing dataset can silently accumulate more total storage cost than the primary volume it's protecting, especially for a copy-on-write-based CSI snapshot implementation where changed-block volume matters more than raw dataset size. ```bash kubectl get pvc -n inventory -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage,STORAGECLASS:.spec.storageClassName # CSI-driver-specific usage metrics (varies by provider) are what actually # shows consumed-vs-provisioned, since a PVC's own status only reports # the provisioned size, not how full it currently is ``` | Signal | What it tells you | Action | |---|---|---| | PVC usage consistently above ~80% of provisioned capacity | Approaching a hard failure (writes start failing, not just degrading) | Trigger CSI volume expansion (Part 3) proactively, not reactively during an incident | | PVC provisioned far above actual usage, unchanged for months | The same overprovisioning pattern this chapter applies to compute, applied to storage | Right-size on the next planned maintenance window — shrinking a PVC in place isn't supported by most CSI drivers, so this usually means provisioning a smaller replacement and migrating data | | Snapshot storage growing faster than primary volume growth | Retention policy or snapshot frequency may be miscalibrated for this dataset's actual change rate | Review retention windows against Part 18's actual recovery-point-objective requirements, not a default "keep everything" policy | > [!IMPORTANT] > Most CSI drivers support only *expanding* a PVC online, never shrinking one — an overprovisioned volume > discovered during a capacity review typically can't be right-sized in place at all. Plan storage requests > more conservatively on the "don't go too small" side than compute requests, since the cost of a mistake is > a planned data-migration project rather than a one-line YAML edit. ## Control-Plane Capacity Planning — the Part Everyone Forgets **Every section so far has been about worker-node/workload capacity — the control plane (Part 1) has its own capacity limits that a growing cluster can genuinely outgrow, and managed offerings (EKS, GKE, AKS) hide this concern just well enough that teams frequently never plan for it at all.** | Control-plane resource | Growth driver | Symptom when undersized | |---|---|---| | etcd storage/IOPS | Total object count across every namespace — CRDs, Secrets, ConfigMaps, and every tenant's own objects (Part 13) all count | API Server latency climbs cluster-wide; etcd compaction/defrag backlogs (Part 4) | | API Server request throughput | Number of controllers/operators watching/listing objects, and how efficiently they do it | Client-side rate limiting, slow `kubectl` responses, controller reconcile lag | | Scheduler throughput | Total pod churn rate (creates/deletes/reschedules per second) across the whole cluster | Pods sit `Pending` longer than expected even with node capacity available | | Node count ceiling (self-managed) | A single control plane's practical scaling ceiling before requiring architectural changes | Documented per-distribution; approaching it is a signal to plan a cluster split or a different topology, not just add more etcd hardware | **This is the single most common blind spot in this chapter's whole subject**, precisely because managed Kubernetes offerings scale their control planes mostly transparently — a team that's never operated self-managed clusters (Part 6) can go years without ever having to think about etcd capacity at all, right up until a genuinely large multi-tenant cluster (many CRDs, many tenants, high object churn) starts surfacing API Server latency that has nothing to do with any individual workload's own resource requests. > [!NOTE] > Even on a fully managed control plane, watch for provider-published control-plane request-rate quotas and > object-count soft limits (both EKS and GKE publish these) — a capacity review that only tracks worker-node > headroom while a cluster's object count or API request rate quietly approaches a provider ceiling still > misses a real capacity risk, just one the provider is technically managing on your behalf. ## Building a Recurring Capacity Review Process **A capacity plan is only as good as its last review — every input this chapter covers (usage percentiles, growth trend, cost attribution, GPU utilization) changes continuously, and a plan built once and never revisited degrades into exactly the overprovisioning-or-undersizing problem this chapter opened with.** ```mermaid flowchart TD Start(["Quarterly capacity review"]) --> Usage["Pull VPA recommender +
Kubecost/OpenCost data
for every namespace"] Usage --> Compare["Compare requested vs.
right-sized vs. actual cost"] Compare --> Trend["Update growth-trend
forecast with latest data"] Trend --> Events{"Known upcoming
events/launches?"} Events -->|"Yes"| Ramp["Plan pre-scaling ramp
+ dependency load test"] Events -->|"No"| Adjust["Adjust node pool ceilings
and quotas for next quarter"] Ramp --> Adjust Adjust --> Verify["Re-run failure-domain
survival math against
new totals"] Verify --> Done(["Publish updated plan,
schedule next review"]) classDef step fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef decision fill:#fbeee0,stroke:#b8650f,color:#10161c class Usage,Compare,Trend,Ramp,Adjust,Verify step class Events decision ``` **A quarterly cadence is a reasonable default for most platforms**, tightened to monthly for a cluster experiencing genuinely rapid growth or serving a business with frequent large events, and always triggered ad hoc immediately after any change that materially shifts the workload mix — a large new tenant onboarding (Part 13), a major feature launch, or a significant architecture change (a new GPU-backed service, a database migration changing per-pod resource shape). > [!TIP] > **Best Practice**: Assign explicit, named ownership of the capacity review process to a specific team or > role — "capacity planning" without an owner reliably becomes nobody's job the moment the original engineer > who cared about it moves to a different project, which is exactly how clusters silently drift back into the > overprovisioning or under-headroom states this chapter exists to prevent. ## A Full Worked Scenario: Sizing `checkout`'s Node Pool for a Product Launch **The `checkout` team has a confirmed launch date for a new feature expected to roughly triple traffic for two weeks — walking the full capacity-planning process this chapter covers, start to finish.** 1. **Right-size current requests** (this chapter, section 4): pull 90 days of VPA recommender data for `checkout-service`. Current requests: 500m CPU / 512Mi memory per pod. Recommender's p95-based suggestion: 350m CPU / 420Mi memory — the team had been running 30%+ overprovisioned without realizing it. 2. **Establish the steady-state baseline** from the corrected numbers: 8 replicas × 350m CPU = 2.8 CPU cores, 8 × 420Mi = 3.4Gi memory at normal load. 3. **Apply the 3x launch multiplier** to the *corrected* baseline, not the original overprovisioned one: 8.4 CPU cores, 10.2Gi memory at projected launch peak, roughly 24 replicas via HPA (Part 12). 4. **Apply failure-domain headroom** (this chapter, section 7) across the 3 AZs `checkout` spans: for N-1 AZ survival, each AZ needs capacity for the full projected peak divided by 2, not divided by 3 — roughly 4.2 CPU cores / 5.1Gi memory of *committed* capacity per AZ during the launch window. 5. **Check node pool instance-type fit** (this chapter, section 8): the current general-purpose instance type's CPU:memory ratio matches `checkout-service`'s own ratio closely enough that no instance-type change is needed — just more nodes of the same type. 6. **Pre-scale and load-test** (this chapter, section 11) the raised node pool ceiling and HPA `maxReplicas` a week before launch, confirming `catalog-service` and `inventory-service` (both real dependencies of `checkout-service` in this throughline) can also sustain 3x load — the launch plan is incomplete if only `checkout-service` itself was capacity-planned. 7. **Wind down** the temporarily raised ceilings roughly a week after the launch window closes, once traffic data confirms the spike has genuinely passed rather than settled at a new, permanently higher baseline — if it has settled higher, that becomes an input to the *next* quarterly trend-forecast review rather than an immediate wind-down. **The single most important step in this sequence is step 1** — every later calculation in this scenario would have inherited a 30% error if the team had launched capacity planning from the original, unverified request values instead of correcting them first. This is the direct, worked consequence of this chapter's earlier point that right-sizing is a precondition for every downstream calculation, not an optional first step. ## A Full Worked Scenario: A Reactive-Only Team Gets Caught Out **A contrasting scenario: the `catalog` team never adopted trend-based forecasting, relying entirely on Cluster Autoscaler and HPA to react to whatever load arrives.** `catalog-service` traffic has been growing roughly 15% month-over-month for two quarters — a real, visible trend in retrospect, but nobody was plotting it because the reactive autoscaling stack had, so far, always kept up. In month seven, a competitor's outage drives an unplanned 40% traffic surge on top of the existing organic growth, in a single day. **Two levels deep**: the symptom is `catalog-service` returning elevated 503 rates for roughly 20 minutes during the surge, despite HPA correctly detecting the load and requesting more replicas immediately. The immediate cause is the node pool's configured maximum size — set months earlier, when 15%-per-month growth still fit comfortably under it — being reached before HPA's requested replica count was satisfied, so new pods sat `Pending` (Part 10's diagnostic tree) waiting on Cluster Autoscaler, which itself was waiting on the node pool's hard ceiling. The underlying condition is that nobody had revisited the node pool's maximum size against the accumulating growth trend in over two quarters, because the team's only capacity signal was "has anything broken yet" — and nothing had, right up until an external, unplanned event pushed cumulative organic growth plus a surge past a ceiling that was already close to being outgrown on its own. > [!CAUTION] > A reactive-only capacity strategy can look completely healthy for months while quietly consuming all of > its own margin — the absence of a visible problem is not the same evidence as "capacity is correctly > sized," and by the time a reactive-only gap actually manifests as an outage, it's frequently coincident > with exactly the kind of external event (a competitor's outage driving traffic over, a viral moment) that > makes the outage maximally visible and costly. **The fix applied afterward**: the team adopted this chapter's quarterly review process, plotted the same 15%-per-month trend that had been sitting in their metrics the whole time, and raised the node pool ceiling proactively for projected month-nine demand — closing exactly the gap trend-based forecasting exists to close, one quarter later than it should have been closed, but before the next surge. ## A Full Worked Scenario: Consolidating an Overprovisioned Fleet **A third scenario, this time a pure cost-recovery exercise: a platform team runs Kubecost across the whole cluster for the first time and finds `inventory-service` costing roughly 3x what its actual traffic and usage pattern would justify.** ```bash curl -s ".../allocation/compute?window=30d&aggregate=namespace" | \ jq '.data[] | to_entries[] | select(.key=="inventory") | .value' # efficiency: 0.11 <- only 11% of requested cost reflects actual usage ``` Investigation (this chapter's right-sizing method, section 4) finds `inventory-service`'s requests were set during an initial capacity-constrained launch two years earlier, when the team deliberately over-requested "to be safe" against a since-resolved database bottleneck that made every request unusually slow at the time. The underlying database issue was fixed within a month of launch; the resource requests, set once and never revisited, stayed at the original conservative sizing for two full years. **The fix**: run VPA in `Off` mode (this chapter, section 5) for two weeks to confirm a stable, current recommendation, then apply the corrected requests via a standard rolling deployment. Post-fix, Kubecost's efficiency metric for `inventory-service` moves from 11% to roughly 70% (never expected to reach 100%, since some headroom above p95 usage is the deliberate, correct choice from section 4, not remaining waste), and the node pool it shares with `catalog-service` consolidates from 9 nodes down to 6 as the Cluster Autoscaler's bin-packing (this chapter, section 9) reclaims the newly-freed slack. **The generalizable lesson**: a resource request set correctly at one point in time silently becomes wrong the moment the underlying condition that justified it changes — this scenario's two-year gap is an extreme but realistic case of exactly the "nobody owns a periodic review" failure this chapter's recurring-review section exists to close permanently, not just fix once. ## Part 16 CLI Cheat Sheet | Command | Purpose | |---|---| | `kubectl top pod -n --containers` | Quick, current-moment resource usage snapshot per container | | `kubectl describe vpa -n ` | Read a VPA's recommendation without it ever touching running pods (`updateMode: "Off"`) | | `kubectl get nodes -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.cpu,ALLOCATABLE:.status.allocatable.cpu` | Compare raw node capacity against what's actually schedulable | | `kubectl describe node \| grep -A5 "Allocated resources"` | Per-node view of how much of allocatable capacity is currently committed | | `kubectl get resourcequota -A -o json \| jq ...` | Aggregate requested capacity per namespace (Part 13's chargeback query, reused here) | | `curl .../allocation/compute?window=7d&aggregate=namespace` | OpenCost/Kubecost real-cost attribution per namespace | | `kubectl get pods -A --field-selector status.phase=Pending` | Cluster-wide view of anything currently unable to schedule — a live capacity-shortage signal | ## A Capacity Review Checklist - [ ] Pulled VPA recommender (`Off` mode) data for every production Deployment in the review window - [ ] Compared current requests against p95-based recommendations; flagged any namespace over roughly 2x its recommended requests as a right-sizing priority - [ ] Pulled Kubecost/OpenCost per-namespace efficiency figures and cross-checked against the flagged list above - [ ] Updated the growth-trend forecast with the latest quarter's usage data - [ ] Confirmed node pool maximum sizes still exceed the updated forecast's projected peak, with headroom - [ ] Re-ran the N-1 failure-domain survival calculation against updated total capacity - [ ] Checked for known upcoming events/launches requiring a pre-scaling ramp and dependency load test - [ ] Reviewed GPU pool utilization and fragmentation signals separately, per this chapter's GPU section - [ ] Confirmed control-plane capacity (etcd size, API request rate, object count) against provider-published ceilings, even on a managed control plane - [ ] Published the updated plan and confirmed a named owner for the next review cycle ## Common Mistakes and Interview Traps | Mistake | Why it's wrong | Correct approach | |---|---|---| | Treating autoscaling as a substitute for capacity planning | Autoscalers can only scale within a ceiling someone else set — they don't decide what that ceiling should be | Use capacity planning to set the ceiling autoscalers operate inside, not as an alternative to it | | Sizing a node pool from requested capacity without checking if requests reflect real usage | Requests can be 3-8x real usage (this chapter's overprovisioning data) — every downstream calculation inherits that same error | Right-size requests from VPA recommender/percentile data first, before any node pool sizing math | | Dividing total capacity evenly by the number of failure domains for headroom | Understates the true headroom needed — surviving the loss of 1 of N domains requires each domain to individually hold `total / (N-1)`, not `total / N` | Use the N-1 formula, not a naive even split | | Assuming multiple tenants' or services' peak loads never overlap without checking | Two services fed by the same underlying business event (a flash sale) very plausibly peak together | Verify peak-timing correlation from real historical data before assuming independence | | Planning capacity for one service's launch traffic without checking its real dependencies | A perfectly-sized `checkout-service` still fails if `catalog-service` or `inventory-service` weren't planned for the same multiplier | Capacity-plan the whole dependency chain for a known event, not just the customer-facing service | | Ignoring control-plane capacity because the control plane is "managed" | Managed control planes still publish object-count and request-rate ceilings that a growing multi-tenant cluster can approach | Track control-plane-level metrics/quotas in the same recurring review as worker-node capacity | | Treating a capacity plan as a one-time exercise | Usage, cost, and growth trends all change continuously — a plan built once accretes drift exactly like unreviewed resource requests do | Run the recurring review process on a fixed cadence with a named owner | ## Worked Practice Problems **Problem 1**: A cluster spans 3 availability zones and currently requests 90 CPU cores of workload, evenly split (30 cores per AZ) with each AZ provisioned at exactly 30 cores of capacity and zero headroom. How much additional capacity per AZ is needed to survive losing any one AZ, and why? *Answer*: Surviving the loss of 1 of 3 AZs requires the remaining 2 AZs to together hold all 90 cores, which means each AZ must individually be capable of holding up to 45 cores in the worst case (`90 / (3-1)`), not the naive `90 / 3 = 30`. Each AZ needs 15 additional cores of provisioned capacity beyond its current 30, a 50% increase over the naive even split — losing any one AZ then leaves the remaining two AZs (90 cores of combined capacity) exactly able to absorb the lost AZ's 30 cores of rescheduled workload. **Problem 2**: A team's Kubecost report shows a namespace at 12% cost efficiency (actual usage vs. requested cost). What should this trigger, and what's the correct first step before changing any resource requests? *Answer*: 12% efficiency signals the namespace's requests are roughly 8x its real usage — a strong right-sizing candidate, not a minor tuning opportunity. The correct first step is not to immediately edit the manifest's requests, but to run (or check an existing) VPA recommender in `Off` mode against real recent usage data first, confirming the recommended values reflect a stable, representative usage pattern (not a brief anomaly) before applying corrected requests — acting on a single cost snapshot without confirming the underlying usage data is stable risks under-sizing just as badly as the original overprovisioning. **Problem 3**: A GPU pod requesting 2 GPUs stays `Pending` even though `kubectl describe nodes` shows 5 GPUs "free" across the cluster in total, and the Cluster Autoscaler has not added any new nodes. What's actually happening, and what two structural fixes address it? *Answer*: The 5 free GPUs are fragmented as single units across separate nodes — none of them has 2 free GPUs together, so the pod's 2-GPU request can't be satisfied by any single node, even though the cluster isn't out of GPU capacity in aggregate. Because the cluster genuinely has free GPUs somewhere, the Cluster Autoscaler often doesn't recognize this as a scaling trigger at all. The two structural fixes are: enforcing consistent GPU request sizes within a node pool so fragmentation of this shape can't occur, and/or splitting single-GPU and multi-GPU workloads into separate, dedicated node pools so a multi-GPU pod is never competing for space against a scattered set of single-GPU allocations. **Problem 4**: A platform team raises a node pool's ceiling and HPA `maxReplicas` ahead of a planned product launch expected to triple `checkout-service` traffic, but the launch still produces a partial outage. Postmortem finds `checkout-service` itself scaled correctly and had plenty of headroom. What capacity-planning step was most likely skipped? *Answer*: The team almost certainly capacity-planned `checkout-service` in isolation without extending the same review to its real dependencies (`catalog-service`, `inventory-service`, or an external database/API) — a customer-facing service scaling correctly under 3x load still produces user-visible failures if a dependency it calls on every request wasn't sized or load-tested for the same multiplier. The fix is treating a known-event capacity plan as covering the whole dependency chain, confirmed via an actual load test against the raised ceilings before the event, not just the one service that happens to be customer-facing. ## Summary and What's Next Capacity planning is the deliberate, forward-looking layer that decides what autoscaling is allowed to scale into — right-sizing requests from real usage data first, since every later calculation inherits that number's accuracy; sizing cluster-wide headroom explicitly for failure-domain loss, burst absorption, and upgrades rather than picking one arbitrary percentage; matching node pool instance types to a workload's real CPU:memory ratio and request-size consistency to avoid bin-packing fragmentation; forecasting growth trends to get ahead of demand instead of always reacting one cycle behind it; and treating GPU pools, multi-tenant aggregation, and control-plane capacity as their own specific cases within the same overall discipline. None of it holds without a recurring, owned review process — the single habit that keeps a cluster's shape honest against its actual, continuously-changing workload instead of drifting silently until a bill or an outage forces the conversation. Part 17 moves from planning how much cluster to run to a deep look at what actually moves packets through it: Cilium and eBPF-based networking. It builds directly on Part 3's base networking model and the brief Cilium Gateway mention in Part 8, going deep on the eBPF data plane itself, kube-proxy replacement, identity -based network policy, Hubble observability, and Cilium's sidecar-less service mesh — the layer every capacity number in this chapter ultimately has to move traffic across.