Capacity Planning for Kubernetes Clusters

Assumes you're comfortable with VPA's recommender/updater/admission-controller mechanics and HPA/Cluster Autoscaler behavior from Part 12, the QoS classes and resource requests/limits model from Part 2, GPU scheduling from Part 14, and the tenant-quota sizing math from Part 13 — this chapter is about turning all of that into a repeatable, forward-looking planning process rather than reintroducing any of it.

Table of Contents#

1. Why Capacity Planning Deserves Its Own Chapter
2. Capacity Planning vs. Autoscaling — Where One Ends and the Other Begins
3. The Overprovisioning Problem, in Real Numbers
4. Right-Sizing Requests From Real Usage Data
5. Turning VPA's Recommender Into a Planning Signal
6. Cluster-Wide Headroom — How Much Slack a Healthy Cluster Actually Needs
7. Surviving a Failure Domain: the Node and AZ Loss Calculation
8. Node Pool Capacity Math — Sizing Instance Types for a Workload Mix
9. Bin Packing and Instance-Type Fragmentation
10. Blending On-Demand, Reserved, and Spot Capacity
11. Forecasting Growth — Trend-Based Planning vs. Reactive-Only Autoscaling
12. Seasonal and Event-Driven Capacity Planning
13. Cost Visibility With Kubecost and OpenCost
14. Capacity Planning for GPU Pools
15. Capacity Planning in a Multi-Tenant Cluster
16. Batch and CronJob Capacity — A Different Shape of Demand
17. Storage Capacity Planning — PVs, Snapshots, and Growth
18. Control-Plane Capacity Planning — the Part Everyone Forgets
19. Building a Recurring Capacity Review Process
20. A Full Worked Scenario: Sizing checkout's Node Pool for a Product Launch
21. A Full Worked Scenario: A Reactive-Only Team Gets Caught Out
22. A Full Worked Scenario: Consolidating an Overprovisioned Fleet
23. Part 16 CLI Cheat Sheet
24. A Capacity Review Checklist
25. Common Mistakes and Interview Traps
26. Worked Practice Problems
27. Summary and What's Next

Why Capacity Planning Deserves Its Own Chapter#

Autoscaling (Part 12) answers "how do I react to load that's already here"; capacity planning answers a different, harder question — "how much capacity should exist at all, before load arrives, so that reacting to it is even possible." These are not the same skill, and treating them as one is the single most common gap in an otherwise well-autoscaled cluster: an HPA can only scale a Deployment up to the replica count a node pool actually has room for, and a Cluster Autoscaler can only add a node as fast as the cloud provider's API and the node's own boot time allow — neither one manufactures capacity instantly, and neither one decides, on its own, whether the ceiling they're scaling toward is the right one.

This chapter treats capacity planning as its own discipline: reading real usage data instead of guessing at requests and limits, sizing node pools deliberately instead of accreting them accidentally, planning for the loss of an entire failure domain, and building the recurring review habit that keeps a cluster's shape matched to its actual workload over time rather than drifting silently for a year until a bill or an outage forces the conversation.

The throughline system carries over unchanged: checkout-service (namespace checkout), catalog-service (namespace catalog), inventory-service (namespace inventory), and the recommendations team's GPU-backed recommendation-model (namespace recommendations, introduced in Part 14) — this chapter treats all four as one platform team's actual capacity-planning workload.

Capacity Planning vs. Autoscaling — Where One Ends and the Other Begins#

The clearest way to separate the two is by time horizon: autoscaling operates on a horizon of seconds to minutes and reacts to a metric that has already moved; capacity planning operates on a horizon of weeks to quarters and decides what the autoscaler is allowed to scale into.

Neither layer can substitute for the other. An autoscaling-only strategy with no capacity planning behind it eventually hits a real ceiling — a node pool's max size, a cloud account's instance-family quota, a budget nobody sized against actual growth — and fails exactly when load is highest and margin for error is lowest. A capacity-planning-only strategy with no autoscaling wastes money on permanently-provisioned headroom that autoscaling would have reclaimed automatically during quiet hours. The two are complementary layers of the same problem, one setting the boundary and the other operating inside it.

The Overprovisioning Problem, in Real Numbers#

The dominant real-world capacity-planning failure isn't running out of room — it's the opposite, and it's gotten measurably worse, not better, as Kubernetes adoption has matured. Recent fleet-wide measurements found CPU overprovisioning across production clusters climbing from roughly 40% to 69% year over year, with memory overprovisioning reaching close to 79% — real utilization sitting at single-digit percentages even as Kubernetes adoption itself reached roughly 80% of production workloads. GPU pools are worse still: fleet-wide GPU utilization averaging around 5%, even though a well-tuned cluster can sustain 40-50%+ (Part 14 covers why GPU scheduling in particular fragments this badly).

This gap matters for two distinct reasons, not one. First, the obvious one: every core requested but unused is a core the organization is paying for and getting nothing from. Second, the less obvious one this chapter cares about more: a cluster whose requests don't reflect real usage makes every capacity-planning calculation downstream of it — node pool sizing, failure-domain headroom, forecasted growth — wrong in the same direction, because all of those calculations start from requested capacity, not actual capacity. Fixing the request-vs-usage gap isn't just a cost optimization; it's a precondition for every other calculation in this chapter being trustworthy.

Right-Sizing Requests From Real Usage Data#

The correct starting point for any capacity plan is the same starting point Part 12's VPA recommender uses internally: observed usage, not a guess copy-pasted from a template chart. A team writing a Deployment's resource requests for the first time genuinely has no usage data to work from — a reasonable default (100m CPU / 128Mi memory request for a typical Go/Node service, roughly 250m CPU / 512Mi memory for a JVM service) is a legitimate starting point, but it is a placeholder, not a target, and it must be revisited against real data within the first few weeks of production traffic.

# One week of real usage is the minimum useful observation window —
# shorter windows miss weekly traffic cycles (weekday vs. weekend load)
kubectl top pod -n checkout --containers
# For a real distribution rather than a single point-in-time snapshot,
# query the metrics pipeline (Prometheus) directly for percentiles:

quantile_over_time(0.95, container_memory_working_set_bytes{namespace="checkout", pod=~"checkout-service-.*"}[7d])

Explaining the throttling trap two levels deep, since it's the single most common outcome of guessing low on CPU: the symptom is checkout-service showing elevated p99 latency under load despite kubectl top reporting CPU usage comfortably under its limit. The immediate cause is CPU throttling — the kernel's CFS bandwidth controller enforces the limit over a rolling 100ms quota period, not as a smooth ongoing cap, so a bursty container can exhaust its entire period's quota in a few milliseconds and sit idle, throttled, for the rest of the window, an effect invisible to any metric that only samples average usage. The underlying condition is that the request/limit pair was set from a single load test's average CPU reading rather than from a percentile distribution capturing real burst behavior — averaging hides exactly the bursts that trigger throttling, which is why container_cpu_cfs_throttled_periods_total (not raw CPU usage) is the metric that actually confirms this failure mode.

Turning VPA's Recommender Into a Planning Signal#

Part 12 covered VPA's updateMode for actually applying recommendations to running pods — this chapter uses the same recommender component for a different purpose: as a standing, always-current data source for capacity planning, even on workloads where automatic updates (Auto/Recreate) are deliberately not enabled.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-service-capacity-signal
  namespace: checkout
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-service
  updatePolicy:
    updateMode: "Off"   # never touches the running pods — pure recommendation signal

kubectl describe vpa checkout-service-capacity-signal -n checkout | grep -A8 "Recommendation:"

Running VPA in Off mode across every Deployment in a cluster, purely for its recommendation output, is a deliberately low-risk way to get the section above's percentile-based sizing without writing a single PromQL query by hand — the recommender already does that percentile calculation internally, using a decaying histogram of real observed usage. A platform team can then aggregate every workload's VPA recommendation once a quarter into a single spreadsheet or dashboard, comparing current requests against recommended requests cluster-wide, which becomes the primary input to the node-pool sizing math later in this chapter.

Cluster-Wide Headroom — How Much Slack a Healthy Cluster Actually Needs#

A cluster running at 100% of its allocatable capacity has zero room to schedule a replacement pod when a node fails, zero room for the Cluster Autoscaler to add nodes before new pods are already failing to schedule, and zero room to absorb a legitimate short-term traffic spike without immediately going Pending. Cluster-wide headroom is capacity deliberately held in reserve, above what current workloads request, for exactly these moments.

Each headroom category exists to answer a different failure mode, and sizing them together (rather than picking one arbitrary "headroom percentage" and calling it done) is what makes the number defensible in a capacity review. The failure-domain buffer alone is covered in depth in the next section since it has an exact, computable answer; the burst and upgrade buffers are typically expressed as a percentage of steady -state committed capacity, informed by how bursty a workload mix actually is and how long node provisioning takes in that specific cloud/region.

Surviving a Failure Domain: the Node and AZ Loss Calculation#

This is the one headroom category with an exact, computable answer rather than a judgment call — decide which failure domain the cluster must survive losing (one node, or one entire availability zone), then size committed capacity so the remaining domains alone can still hold every workload's requests.

The formula generalizes cleanly: for surviving the loss of 1 of N failure domains, each domain needs capacity for total-requested-capacity / (N - 1), not total-requested-capacity / N. For 3 AZs that's a 50% headroom requirement over a naive even split; for 4 AZs it drops to roughly 33%; the fewer failure domains a cluster spans, the more headroom each one needs to carry to survive losing any single one — which is itself a real argument for spreading across more AZs when the workload and latency budget tolerate it.

Node Pool Capacity Math — Sizing Instance Types for a Workload Mix#

Once total required capacity (including headroom) is known, the remaining question is which instance type, and how many of them, actually deliver it — and different instance types don't just cost differently, they bin-pack differently against a given workload's request shape.

The general lesson: match the node pool's own CPU:memory ratio to the actual ratio of the workloads that will run on it, not to whichever instance family happens to be the organization's historical default. A workload mix skewed heavily toward one resource (CPU-heavy batch processing, memory-heavy caching tiers) stranding capacity on a mismatched instance type is a quiet, recurring cost identical in effect to the overprovisioned-request problem earlier in this chapter, except it happens at the node level instead of the pod level.

Bin Packing and Instance-Type Fragmentation#

Even a well-matched instance type can fragment badly if a node pool mixes very differently-sized pods, because the scheduler's bin-packing is NP-hard in the general case and settles for a good-enough placement, not a provably optimal one.

This exact fragmentation pattern is worse, not just similarly bad, for GPU pools — a pod requesting 2 whole GPUs cannot be scheduled even when the cluster has 3 GPUs free in total, if those 3 free GPUs are spread as single units across 3 different nodes rather than concentrated on one. The Cluster Autoscaler often doesn't even recognize this as a capacity shortage worth scaling for, because in aggregate the cluster genuinely isn't "out of GPUs" — it's out of GPUs in the shape this specific pod needs them. This chapter's GPU section below covers the direct mitigation.

Blending On-Demand, Reserved, and Spot Capacity#

Everything so far has assumed a single, uniform kind of node — real capacity planning also decides how that capacity is purchased, since on-demand, reserved/committed-use, and spot instances differ enormously in both price and the failure characteristics a capacity plan has to design around.

The capacity-planning discipline here is deciding what fraction of the plan built in earlier sections goes on each tier, not treating the choice as purely a procurement afterthought. A common, defensible split: size reserved/committed-use capacity to cover the steady-state baseline from the forecasting sections above (since that portion is, by definition, always running and benefits most from a multi-year discount), size on-demand to cover the failure-domain and burst headroom from earlier in this chapter (since headroom must be guaranteed available, not reclaimable), and route genuinely interruption-tolerant workloads — batch processing, CI runners, a subset of stateless replicas behind an HPA with enough minimum on-demand replicas to stay available during a mass spot reclamation — onto spot capacity for the largest cost reduction.

# A Karpenter NodePool expressing this split for one workload tier —
# spot preferred, falling back to on-demand only if spot is unavailable
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: catalog-batch-interruptible
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]   # tries spot first, falls back automatically

Forecasting Growth — Trend-Based Planning vs. Reactive-Only Autoscaling#

A cluster that only ever reacts to the Cluster Autoscaler noticing unschedulable pods is, by construction, always at least one provisioning cycle behind real demand — trend-based forecasting exists to get ahead of that lag for demand that's predictable in advance, reserving pure reactive autoscaling for the genuinely unpredictable remainder.

The gap between the two lines is the real cost of reactive-only planning: at any given month, capacity is sized to what demand was one cycle ago, not what it is now — a gap that's tolerable when growth is slow and autoscaling latency is low, and genuinely dangerous when growth accelerates faster than node provisioning can keep pace, which is exactly the moment a business cares most about not having an outage (a product launch, a marketing campaign, a viral traffic event).

A simple, honest forecasting method beats no forecasting method at all: plot 6-12 months of requested (right-sized, per the earlier sections) capacity per namespace/team, fit a linear or simple seasonal trend line, and provision node pool ceilings for the projected number at the end of the next planning horizon — not the current number. This doesn't need to be sophisticated machine learning to be far better than pure reactive autoscaling; the improvement over doing nothing is what matters, not the sophistication of the model.

Seasonal and Event-Driven Capacity Planning#

Growth-trend forecasting captures gradual change; seasonal and event-driven planning captures the sharp, recurring or one-off spikes a smooth trend line misses entirely — a marketing campaign, a known peak shopping period, a product launch date.

Pre-scaling ahead of a known event deliberately trades a few days of paying for unused headroom against the far more expensive alternative of the Cluster Autoscaler racing to catch up in real time during the highest-stakes traffic window of the year. This is one of the few places in this chapter where the "never overprovision" instinct from earlier sections is deliberately overridden — the cost of temporary, known-duration overprovisioning during a defined peak window is cheap insurance compared to the cost of an outage during that same window.

Cost Visibility With Kubecost and OpenCost#

Every calculation in this chapter — right-sizing, headroom, node pool shape, forecasting — ultimately resolves to a dollar figure, and Kubernetes has no native mechanism for attributing a shared cluster's bill back to the namespaces, teams, or workloads actually consuming it. OpenCost (the CNCF-hosted open-source engine) and Kubecost (the commercial product built on top of it) exist specifically to close this gap, tracing real cost — not just requested resources — down to the namespace, label, or workload level.

# OpenCost's allocation API — real cost, not just requested capacity,
# attributed per namespace over a trailing window
curl -s "http://opencost.opencost.svc.cluster.local:9003/allocation/compute?window=7d&aggregate=namespace" | \
  jq '.data[] | to_entries[] | {namespace: .key, totalCost: .value.totalCost}'

The distinction between requested-based and actual-usage-based cost attribution matters directly for this chapter's overprovisioning problem: a namespace whose actual cost (from Kubecost/OpenCost) is far below its Part 13-style ResourceQuota-based cost estimate is exactly the namespace this chapter's right-sizing sections should target first — cost tooling turns "we suspect we're overprovisioned somewhere" into a ranked, namespace-by-namespace list of exactly where the waste actually is.

Capacity Planning for GPU Pools#

GPU capacity planning inherits every general principle above and adds one the general case doesn't have: GPUs are indivisible by default (Part 14's device-plugin model), so fragmentation and utilization waste are both structurally worse than for CPU/memory.

Forecasting GPU capacity specifically needs a longer lead time than CPU/memory forecasting, because GPU node provisioning is frequently gated by cloud-provider capacity reservations or quota increases that take weeks, not the minutes a standard compute node takes to provision — a GPU capacity plan discovered to be short during the forecasting review in the earlier section needs to trigger a reservation request immediately, not after the shortfall is already visible in production.

Capacity Planning in a Multi-Tenant Cluster#

Part 13 covered sizing a single tenant's ResourceQuota from its own HPA ceiling — cluster-wide capacity planning is the aggregate version of that same exercise across every tenant sharing the cluster, plus the headroom this chapter adds on top.

The "peaks never overlap" reduction deserves real scrutiny before being trusted — it's a legitimate way to avoid needlessly summing every tenant's absolute peak as if they all happen simultaneously, but it's also an easy place to under-provision by assuming independence that doesn't actually hold. A retail checkout namespace and a recommendations namespace serving the same storefront very plausibly peak at exactly the same moment (a flash sale drives both checkout volume and recommendation-serving volume together), which means treating their peaks as independent would understate combined peak demand precisely when it matters most.

Batch and CronJob Capacity — A Different Shape of Demand#

Everything so far has implicitly assumed a long-running Deployment's steady-plus-burst demand shape — Jobs and CronJobs (Part 2, Part 10's troubleshooting angle) demand capacity in short, scheduled bursts instead, and sizing a node pool for them the same way as a long-running service either wastes money holding permanent capacity idle between runs, or causes the exact Pending-pod scheduling failures Part 10 already covers if the burst is undersized.

The capacity-planning answer for this shape is almost always "scale the node pool to zero (or near-zero) between runs and let the Cluster Autoscaler/Karpenter provision burst capacity on demand," rather than holding permanent headroom for a workload that's only actually running a small fraction of each day. This is one of the few places in this chapter where the right answer inverts the earlier failure-domain-headroom guidance: a batch workload with a generous completion deadline (activeDeadlineSeconds, Part 10) can tolerate the provisioning latency of scaling up from zero in a way a customer-facing service's failure-domain buffer cannot.

Storage Capacity Planning — PVs, Snapshots, and Growth#

Every section so far has been compute-shaped — storage capacity planning follows a genuinely different growth curve, because a PersistentVolumeClaim (Part 3) almost never shrinks on its own even as the workload behind it scales up and down, and unbounded log/data growth inside a volume is a slow-burning capacity failure a purely compute-focused review will never catch.

Two growth curves need separate tracking, not one: the primary volume's own data growth (which kubectl get pvc combined with the CSI driver's usage metrics can track directly), and the accumulated storage consumed by snapshots and backups (Part 18 covers backup/DR strategy in depth) — a snapshot retention policy that keeps 90 days of daily snapshots for a rapidly-changing dataset can silently accumulate more total storage cost than the primary volume it's protecting, especially for a copy-on-write-based CSI snapshot implementation where changed-block volume matters more than raw dataset size.

kubectl get pvc -n inventory -o custom-columns=NAME:.metadata.name,CAPACITY:.status.capacity.storage,STORAGECLASS:.spec.storageClassName
# CSI-driver-specific usage metrics (varies by provider) are what actually
# shows consumed-vs-provisioned, since a PVC's own status only reports
# the provisioned size, not how full it currently is

Control-Plane Capacity Planning — the Part Everyone Forgets#

Every section so far has been about worker-node/workload capacity — the control plane (Part 1) has its own capacity limits that a growing cluster can genuinely outgrow, and managed offerings (EKS, GKE, AKS) hide this concern just well enough that teams frequently never plan for it at all.

This is the single most common blind spot in this chapter's whole subject, precisely because managed Kubernetes offerings scale their control planes mostly transparently — a team that's never operated self-managed clusters (Part 6) can go years without ever having to think about etcd capacity at all, right up until a genuinely large multi-tenant cluster (many CRDs, many tenants, high object churn) starts surfacing API Server latency that has nothing to do with any individual workload's own resource requests.

Building a Recurring Capacity Review Process#

A capacity plan is only as good as its last review — every input this chapter covers (usage percentiles, growth trend, cost attribution, GPU utilization) changes continuously, and a plan built once and never revisited degrades into exactly the overprovisioning-or-undersizing problem this chapter opened with.

A quarterly cadence is a reasonable default for most platforms, tightened to monthly for a cluster experiencing genuinely rapid growth or serving a business with frequent large events, and always triggered ad hoc immediately after any change that materially shifts the workload mix — a large new tenant onboarding (Part 13), a major feature launch, or a significant architecture change (a new GPU-backed service, a database migration changing per-pod resource shape).

A Full Worked Scenario: Sizing checkout's Node Pool for a Product Launch#

The checkout team has a confirmed launch date for a new feature expected to roughly triple traffic for two weeks — walking the full capacity-planning process this chapter covers, start to finish.

1. Right-size current requests (this chapter, section 4): pull 90 days of VPA recommender data for checkout-service. Current requests: 500m CPU / 512Mi memory per pod. Recommender's p95-based suggestion: 350m CPU / 420Mi memory — the team had been running 30%+ overprovisioned without realizing it.
2. Establish the steady-state baseline from the corrected numbers: 8 replicas × 350m CPU = 2.8 CPU cores, 8 × 420Mi = 3.4Gi memory at normal load.
3. Apply the 3x launch multiplier to the corrected baseline, not the original overprovisioned one: 8.4 CPU cores, 10.2Gi memory at projected launch peak, roughly 24 replicas via HPA (Part 12).
4. Apply failure-domain headroom (this chapter, section 7) across the 3 AZs checkout spans: for N-1 AZ survival, each AZ needs capacity for the full projected peak divided by 2, not divided by 3 — roughly 4.2 CPU cores / 5.1Gi memory of committed capacity per AZ during the launch window.
5. Check node pool instance-type fit (this chapter, section 8): the current general-purpose instance type's CPU:memory ratio matches checkout-service's own ratio closely enough that no instance-type change is needed — just more nodes of the same type.
6. Pre-scale and load-test (this chapter, section 11) the raised node pool ceiling and HPA maxReplicas a week before launch, confirming catalog-service and inventory-service (both real dependencies of checkout-service in this throughline) can also sustain 3x load — the launch plan is incomplete if only checkout-service itself was capacity-planned.
7. Wind down the temporarily raised ceilings roughly a week after the launch window closes, once traffic data confirms the spike has genuinely passed rather than settled at a new, permanently higher baseline — if it has settled higher, that becomes an input to the next quarterly trend-forecast review rather than an immediate wind-down.

The single most important step in this sequence is step 1 — every later calculation in this scenario would have inherited a 30% error if the team had launched capacity planning from the original, unverified request values instead of correcting them first. This is the direct, worked consequence of this chapter's earlier point that right-sizing is a precondition for every downstream calculation, not an optional first step.

A Full Worked Scenario: A Reactive-Only Team Gets Caught Out#

A contrasting scenario: the catalog team never adopted trend-based forecasting, relying entirely on Cluster Autoscaler and HPA to react to whatever load arrives.

catalog-service traffic has been growing roughly 15% month-over-month for two quarters — a real, visible trend in retrospect, but nobody was plotting it because the reactive autoscaling stack had, so far, always kept up. In month seven, a competitor's outage drives an unplanned 40% traffic surge on top of the existing organic growth, in a single day.

Two levels deep: the symptom is catalog-service returning elevated 503 rates for roughly 20 minutes during the surge, despite HPA correctly detecting the load and requesting more replicas immediately. The immediate cause is the node pool's configured maximum size — set months earlier, when 15%-per-month growth still fit comfortably under it — being reached before HPA's requested replica count was satisfied, so new pods sat Pending (Part 10's diagnostic tree) waiting on Cluster Autoscaler, which itself was waiting on the node pool's hard ceiling. The underlying condition is that nobody had revisited the node pool's maximum size against the accumulating growth trend in over two quarters, because the team's only capacity signal was "has anything broken yet" — and nothing had, right up until an external, unplanned event pushed cumulative organic growth plus a surge past a ceiling that was already close to being outgrown on its own.

The fix applied afterward: the team adopted this chapter's quarterly review process, plotted the same 15%-per-month trend that had been sitting in their metrics the whole time, and raised the node pool ceiling proactively for projected month-nine demand — closing exactly the gap trend-based forecasting exists to close, one quarter later than it should have been closed, but before the next surge.

A Full Worked Scenario: Consolidating an Overprovisioned Fleet#

A third scenario, this time a pure cost-recovery exercise: a platform team runs Kubecost across the whole cluster for the first time and finds inventory-service costing roughly 3x what its actual traffic and usage pattern would justify.

curl -s ".../allocation/compute?window=30d&aggregate=namespace" | \
  jq '.data[] | to_entries[] | select(.key=="inventory") | .value'
# efficiency: 0.11   <- only 11% of requested cost reflects actual usage

Investigation (this chapter's right-sizing method, section 4) finds inventory-service's requests were set during an initial capacity-constrained launch two years earlier, when the team deliberately over-requested "to be safe" against a since-resolved database bottleneck that made every request unusually slow at the time. The underlying database issue was fixed within a month of launch; the resource requests, set once and never revisited, stayed at the original conservative sizing for two full years.

The fix: run VPA in Off mode (this chapter, section 5) for two weeks to confirm a stable, current recommendation, then apply the corrected requests via a standard rolling deployment. Post-fix, Kubecost's efficiency metric for inventory-service moves from 11% to roughly 70% (never expected to reach 100%, since some headroom above p95 usage is the deliberate, correct choice from section 4, not remaining waste), and the node pool it shares with catalog-service consolidates from 9 nodes down to 6 as the Cluster Autoscaler's bin-packing (this chapter, section 9) reclaims the newly-freed slack.

The generalizable lesson: a resource request set correctly at one point in time silently becomes wrong the moment the underlying condition that justified it changes — this scenario's two-year gap is an extreme but realistic case of exactly the "nobody owns a periodic review" failure this chapter's recurring-review section exists to close permanently, not just fix once.

Part 16 CLI Cheat Sheet#

A Capacity Review Checklist#

• Pulled VPA recommender (Off mode) data for every production Deployment in the review window
• Compared current requests against p95-based recommendations; flagged any namespace over roughly 2x its recommended requests as a right-sizing priority
• Pulled Kubecost/OpenCost per-namespace efficiency figures and cross-checked against the flagged list above
• Updated the growth-trend forecast with the latest quarter's usage data
• Confirmed node pool maximum sizes still exceed the updated forecast's projected peak, with headroom
• Re-ran the N-1 failure-domain survival calculation against updated total capacity
• Checked for known upcoming events/launches requiring a pre-scaling ramp and dependency load test
• Reviewed GPU pool utilization and fragmentation signals separately, per this chapter's GPU section
• Confirmed control-plane capacity (etcd size, API request rate, object count) against provider-published ceilings, even on a managed control plane
• Published the updated plan and confirmed a named owner for the next review cycle

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A cluster spans 3 availability zones and currently requests 90 CPU cores of workload, evenly split (30 cores per AZ) with each AZ provisioned at exactly 30 cores of capacity and zero headroom. How much additional capacity per AZ is needed to survive losing any one AZ, and why?

Answer: Surviving the loss of 1 of 3 AZs requires the remaining 2 AZs to together hold all 90 cores, which means each AZ must individually be capable of holding up to 45 cores in the worst case (90 / (3-1)), not the naive 90 / 3 = 30. Each AZ needs 15 additional cores of provisioned capacity beyond its current 30, a 50% increase over the naive even split — losing any one AZ then leaves the remaining two AZs (90 cores of combined capacity) exactly able to absorb the lost AZ's 30 cores of rescheduled workload.

Problem 2: A team's Kubecost report shows a namespace at 12% cost efficiency (actual usage vs. requested cost). What should this trigger, and what's the correct first step before changing any resource requests?

Answer: 12% efficiency signals the namespace's requests are roughly 8x its real usage — a strong right-sizing candidate, not a minor tuning opportunity. The correct first step is not to immediately edit the manifest's requests, but to run (or check an existing) VPA recommender in Off mode against real recent usage data first, confirming the recommended values reflect a stable, representative usage pattern (not a brief anomaly) before applying corrected requests — acting on a single cost snapshot without confirming the underlying usage data is stable risks under-sizing just as badly as the original overprovisioning.

Problem 3: A GPU pod requesting 2 GPUs stays Pending even though kubectl describe nodes shows 5 GPUs "free" across the cluster in total, and the Cluster Autoscaler has not added any new nodes. What's actually happening, and what two structural fixes address it?

Answer: The 5 free GPUs are fragmented as single units across separate nodes — none of them has 2 free GPUs together, so the pod's 2-GPU request can't be satisfied by any single node, even though the cluster isn't out of GPU capacity in aggregate. Because the cluster genuinely has free GPUs somewhere, the Cluster Autoscaler often doesn't recognize this as a scaling trigger at all. The two structural fixes are: enforcing consistent GPU request sizes within a node pool so fragmentation of this shape can't occur, and/or splitting single-GPU and multi-GPU workloads into separate, dedicated node pools so a multi-GPU pod is never competing for space against a scattered set of single-GPU allocations.

Problem 4: A platform team raises a node pool's ceiling and HPA maxReplicas ahead of a planned product launch expected to triple checkout-service traffic, but the launch still produces a partial outage. Postmortem finds checkout-service itself scaled correctly and had plenty of headroom. What capacity-planning step was most likely skipped?

Answer: The team almost certainly capacity-planned checkout-service in isolation without extending the same review to its real dependencies (catalog-service, inventory-service, or an external database/API) — a customer-facing service scaling correctly under 3x load still produces user-visible failures if a dependency it calls on every request wasn't sized or load-tested for the same multiplier. The fix is treating a known-event capacity plan as covering the whole dependency chain, confirmed via an actual load test against the raised ceilings before the event, not just the one service that happens to be customer-facing.

Summary and What's Next#

Capacity planning is the deliberate, forward-looking layer that decides what autoscaling is allowed to scale into — right-sizing requests from real usage data first, since every later calculation inherits that number's accuracy; sizing cluster-wide headroom explicitly for failure-domain loss, burst absorption, and upgrades rather than picking one arbitrary percentage; matching node pool instance types to a workload's real CPU:memory ratio and request-size consistency to avoid bin-packing fragmentation; forecasting growth trends to get ahead of demand instead of always reacting one cycle behind it; and treating GPU pools, multi-tenant aggregation, and control-plane capacity as their own specific cases within the same overall discipline. None of it holds without a recurring, owned review process — the single habit that keeps a cluster's shape honest against its actual, continuously-changing workload instead of drifting silently until a bill or an outage forces the conversation.

Part 17 moves from planning how much cluster to run to a deep look at what actually moves packets through it: Cilium and eBPF-based networking. It builds directly on Part 3's base networking model and the brief Cilium Gateway mention in Part 8, going deep on the eBPF data plane itself, kube-proxy replacement, identity -based network policy, Hubble observability, and Cilium's sidecar-less service mesh — the layer every capacity number in this chapter ultimately has to move traffic across.