# Capacity Planning & Performance — Part 3: Autoscaling & Load Testing

> **Series:** Capacity Planning & Performance (3 of 3)
> **Part 1:** `01-scaling-strategies.md` — Scaling Strategies
> **Part 2:** `02-queuing-theory-and-capacity-math.md` — Queuing Theory & Capacity Math
> **Part 3:** This file — Autoscaling & Load Testing
> **Questions:** `questions.md`

## Table of Contents

1. [From Manual Planning to Automatic Reaction](#from-manual-planning-to-automatic-reaction)
2. [Reactive vs Predictive Autoscaling](#reactive-vs-predictive-autoscaling)
3. [Horizontal Pod Autoscaler (HPA) — How It Actually Works](#horizontal-pod-autoscaler-hpa--how-it-actually-works)
4. [A Full Worked HPA Example](#a-full-worked-hpa-example)
5. [Scaling on Custom and External Metrics](#scaling-on-custom-and-external-metrics)
6. [Vertical Pod Autoscaler (VPA)](#vertical-pod-autoscaler-vpa)
7. [Cluster Autoscaler — Scaling the Nodes Themselves](#cluster-autoscaler--scaling-the-nodes-themselves)
8. [The Cold-Start Problem](#the-cold-start-problem)
9. [Scale-Down Is Just as Important as Scale-Up](#scale-down-is-just-as-important-as-scale-up)
10. [Predictive Autoscaling](#predictive-autoscaling)
11. [Why Load Testing Exists](#why-load-testing-exists)
12. [The Types of Performance Tests](#the-types-of-performance-tests)
13. [Load Testing in Practice: k6](#load-testing-in-practice-k6)
14. [Load Testing in Practice: Locust](#load-testing-in-practice-locust)
15. [Reading Load Test Results](#reading-load-test-results)
16. [Testing in Production, Safely](#testing-in-production-safely)
17. [Common Mistakes](#common-mistakes)
18. [Worked Practice Problems](#worked-practice-problems)
19. [Summary — The Complete Capacity Planning Series](#summary--the-complete-capacity-planning-series)

---

## From Manual Planning to Automatic Reaction

Parts 1 and 2 covered how to scale and how to calculate how much capacity you need. But traffic doesn't wait for a human to run the numbers and provision servers — it changes minute by minute. **Autoscaling** is the automation that reacts to real, live demand without a human in the loop, and **load testing** is how you prove that automation (and the system underneath it) actually works before real users find out the hard way.

```mermaid
graph LR
    A["Capacity planning<br/>(Parts 1-2): figure out<br/>roughly how much you'll<br/>need, in advance"] --> B["Autoscaling (this Part):<br/>automatically adjust in<br/>REAL TIME as actual<br/>demand changes"]
    B --> C["Load testing (this Part):<br/>PROVE both of the above<br/>actually work, before real<br/>traffic tests them for you"]
```

---

## Reactive vs Predictive Autoscaling

```mermaid
graph TD
    Reactive["REACTIVE autoscaling:<br/>watch a live metric (CPU,<br/>request rate) and scale<br/>AFTER it crosses a<br/>threshold"] --> ReactiveNote["Simple, works well for<br/>most cases — but there's<br/>always a LAG between<br/>demand rising and new<br/>capacity actually coming online"]

    Predictive["PREDICTIVE autoscaling:<br/>scale AHEAD of time, based<br/>on a forecast (e.g. 'traffic<br/>always spikes at 9am')"] --> PredictiveNote["Removes the lag for<br/>KNOWN, repeating patterns —<br/>but can't predict genuinely<br/>novel spikes"]
```

Most real systems use reactive autoscaling as the default, foundational layer, and this tutorial spends most of its time there — it's simpler, more broadly applicable, and is what you'll actually configure and operate day to day.

---

## Horizontal Pod Autoscaler (HPA) — How It Actually Works

In Kubernetes, the **Horizontal Pod Autoscaler (HPA)** is the standard reactive autoscaling mechanism — it watches a metric and adjusts the number of running pod replicas to keep that metric near a target.

```mermaid
sequenceDiagram
    participant Metrics as Metrics Server
    participant HPA as HPA Controller
    participant Deploy as Deployment

    loop Every 15 seconds (default)
        HPA->>Metrics: What's the current<br/>average CPU usage<br/>across all pods?
        Metrics-->>HPA: 85% (target is 50%)
        HPA->>HPA: Calculate desired replicas
        HPA->>Deploy: Scale from 4 to 7 replicas
    end
```

### The Actual Scaling Formula

HPA's core calculation is simple and worth knowing by heart:

```
desired replicas = ceil( current replicas × (current metric value / target metric value) )
```

**Worked example:** 4 replicas currently running, average CPU usage is 85%, target is 50%.

```
desired replicas = ceil( 4 × (85 / 50) ) = ceil( 4 × 1.7 ) = ceil(6.8) = 7
```

HPA scales from 4 to 7 replicas. **This exact formula is a commonly asked, concrete interview question** — being able to compute it live is a strong, specific signal.

---

## A Full Worked HPA Example

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-service-hpa
  namespace: checkout
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-service
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60
```

```bash
# Apply it
kubectl apply -f checkout-service-hpa.yaml

# Watch it react live during a load test
kubectl get hpa checkout-service-hpa -n checkout --watch
```

### Why `behavior.scaleDown.stabilizationWindowSeconds` Matters So Much

This single setting deserves its own callout, because it directly solves a real, common problem: **flapping** — rapidly scaling up and back down over and over as a metric bounces around its target.

```mermaid
graph TD
    A["No stabilization window"] --> B["CPU briefly dips below<br/>target for 20 seconds"]
    B --> C["HPA scales DOWN<br/>immediately"]
    C --> D["CPU immediately spikes<br/>back up (load didn't<br/>actually go away)"]
    D --> E["HPA scales UP again"]
    E --> F["🔁 Constant flapping —<br/>wasted churn, briefly<br/>degraded performance<br/>during every scale-down/<br/>scale-up cycle"]
```

**Setting a 300-second (5-minute) stabilization window for scale-down** means HPA looks at the metric's behavior over the last 5 minutes and only scales down based on the calculation that would result in the *fewest* replicas removed — effectively requiring the metric to stay genuinely low for a sustained period before HPA commits to removing capacity. **Scale-up deliberately uses no stabilization window (or a very short one)** — you want to react to a genuine spike immediately, since the cost of scaling up too slowly (real user impact) is much higher than the cost of scaling up too eagerly (some wasted capacity for a few minutes).

---

## Scaling on Custom and External Metrics

CPU/memory utilization is the default, but it's frequently the **wrong** signal to scale on — a genuinely important, sometimes-missed point.

```mermaid
graph TD
    Q["What's the RIGHT metric<br/>to scale a service on?"] --> A["CPU-bound work<br/>(heavy computation)"] --> ACPU["CPU utilization —<br/>a reasonable proxy"]
    Q --> B["I/O-bound work<br/>(mostly waiting on a<br/>database/network)"] --> BOther["CPU is a POOR proxy —<br/>the service can be<br/>completely overwhelmed<br/>while CPU sits low, because<br/>it's stuck WAITING, not<br/>computing"]
    Q --> C["Queue-consuming service"] --> CQueue["Queue depth / consumer<br/>lag — directly reflects<br/>the actual backlog"]
```

**A strong, senior-level interview line:** "CPU utilization is the default HPA metric, but it's frequently the wrong one — an I/O-bound service waiting on a slow downstream dependency can be completely saturated with almost no CPU usage at all, and HPA would never scale it up, exactly the same trap as relying on CPU alone in the USE method from the Monitoring Methodologies series. I'd scale on the metric that actually reflects real load for that specific service — request rate, queue depth, or a custom application-level metric — not blindly default to CPU."

```yaml
# Scaling on a custom Prometheus metric (via the Prometheus Adapter)
# instead of CPU — e.g. queue depth for a message consumer
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-processor-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-processor
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: External
      external:
        metric:
          name: rabbitmq_queue_messages_ready
          selector:
            matchLabels:
              queue: orders
        target:
          type: AverageValue
          averageValue: "30"
```

This says: "keep the average number of ready-to-process messages per replica around 30 — add more replicas if the backlog grows, remove them as it shrinks."

---

## Vertical Pod Autoscaler (VPA)

A different, complementary tool: instead of adding more pod *replicas* (horizontal), **VPA adjusts the CPU/memory requests and limits of individual pods** (vertical) — automatically right-sizing them based on observed real usage.

```mermaid
graph LR
    A["Pod requested 2 CPU,<br/>but VPA observes it only<br/>ever actually uses 0.4 CPU"] --> B["VPA recommends/applies<br/>a smaller request —<br/>freeing up cluster capacity<br/>for other workloads"]
```

**Important, commonly-tested caveat:** VPA and HPA scaling on the *same* metric (e.g., both watching CPU) can conflict — VPA resizing a pod's CPU request changes the denominator HPA's utilization percentage is calculated against, potentially causing unstable interactions between the two. **The standard, safe practice: use VPA for right-sizing requests/limits (often in a "recommendation only" mode reviewed by a human, rather than fully automatic), and use HPA for reactive scaling — avoid pointing both at the exact same metric for the exact same workload simultaneously.**

---

## Cluster Autoscaler — Scaling the Nodes Themselves

HPA and VPA operate at the *pod* level — but pods need somewhere to actually run. If a cluster's nodes are all full, HPA can create as many new pod replicas as it wants; they'll simply sit `Pending`, unable to be scheduled anywhere. **Cluster Autoscaler** solves this one layer down, by adding or removing actual worker nodes (VMs) from the cluster.

```mermaid
flowchart TD
    HPA["HPA scales checkout-service<br/>from 5 to 20 pods"] --> Sched["Kubernetes Scheduler tries<br/>to place the new pods"]
    Sched --> Full{"Enough node<br/>capacity available?"}
    Full -->|Yes| Placed["✅ New pods scheduled<br/>and running"]
    Full -->|"No — nodes are full"| Pending["⚠️ New pods stuck<br/>'Pending'"]
    Pending --> CA["Cluster Autoscaler notices<br/>Pending pods with no room"]
    CA --> NewNode["Provisions a NEW node<br/>from the cloud provider<br/>(e.g. a new EC2 instance)"]
    NewNode --> Sched
```

**Why this three-layer relationship (HPA → Cluster Autoscaler → cloud provider) is a genuinely important thing to be able to draw out in an interview:** it's a very common question to ask "what happens if HPA wants to scale up but there's no room?" — the answer demonstrates you understand that Kubernetes autoscaling isn't one single mechanism, it's a **layered system**, and each layer only solves its own specific piece of the puzzle.

---

## The Cold-Start Problem

Autoscaling isn't instantaneous — there's real latency between "demand increased" and "new capacity is actually serving traffic," and this latency compounds across the layers above.

```mermaid
gantt
    dateFormat X
    axisFormat %Ss
    title Cold-Start Latency Chain (seconds from spike detected)
    section HPA
    Detects spike, decides to scale     :a1, 0, 15
    section Scheduler
    New pod scheduled                    :a2, 15, 5
    section Container
    Image pull + container start         :a3, 20, 20
    section App
    Application warmup (JIT, connection pools, cache fill) :a4, 40, 15
    section Ready
    Actually serving traffic              :milestone, 55, 0
```

**In this realistic example, ~55 seconds pass between the spike starting and new capacity actually helping** — and if a new *node* also needs to be provisioned (Cluster Autoscaler), that can add another 1-3 minutes on top, depending on the cloud provider. **This is exactly why headroom (Part 2) still matters even with autoscaling in place** — autoscaling reduces how much static buffer you need, but it doesn't eliminate the need entirely, because it can't react in zero time.

**Practical mitigations worth naming:**
- Keep `minReplicas` high enough to absorb the cold-start window's worth of extra traffic on its own.
- Use smaller, faster-starting container images (connects to the Dockerfile hardening discussion in the DevSecOps series — smaller images pull faster).
- Consider **over-provisioning** a small buffer of already-running "pause" pods with low priority, specifically so the cluster already has spare node capacity ready before it's needed (a known technique sometimes called "cluster overprovisioning").

---

## Scale-Down Is Just as Important as Scale-Up

Scaling up gets most of the attention (it's what prevents outages), but scaling back down correctly matters just as much for cost and stability.

```mermaid
graph TD
    A["Aggressive scale-down"] --> A1["✅ Saves money fast"]
    A --> A2["❌ Risk of flapping (see above),<br/>and risk of removing capacity<br/>right before a SECOND<br/>wave of traffic arrives"]

    B["Conservative scale-down"] --> B1["✅ More stable, less flapping<br/>risk"]
    B --> B2["❌ Pay for unused capacity<br/>longer than strictly necessary"]
```

This is exactly the same headroom/cost tradeoff from Part 2, just applied to the *speed* of removing capacity rather than the *amount* of capacity provisioned upfront.

---

## Predictive Autoscaling

For traffic with **known, repeating patterns** (a daily 9am spike, a weekly Monday-morning surge, a predictable seasonal event), predictive/scheduled autoscaling removes the cold-start lag entirely by scaling *ahead* of the known pattern, rather than reacting after it starts.

```bash
# A simple scheduled scaling approach — e.g. using a CronJob to
# pre-scale BEFORE a known daily traffic pattern, then letting
# reactive HPA take over for anything beyond the predicted baseline
kubectl patch hpa checkout-service-hpa -n checkout \
  --patch '{"spec":{"minReplicas": 15}}'
# (scheduled to run at 08:50, ahead of a reliable 09:00 traffic spike)
```

**A strong interview framing:** "I'd use predictive/scheduled scaling as a complement to reactive HPA, not a replacement — scheduled scaling handles the *known* patterns (raising the floor, i.e. `minReplicas`, ahead of a predictable spike), while reactive HPA still handles anything above that baseline, including genuinely unpredictable spikes scheduled scaling could never anticipate."

---

## Why Load Testing Exists

Every autoscaling configuration and capacity plan in this whole series is a **hypothesis** until it's actually tested under real load. Load testing is how you validate that hypothesis safely, on your own schedule, instead of finding out it was wrong during a real traffic spike.

```mermaid
graph LR
    A["Capacity plan +<br/>autoscaling config<br/>(a HYPOTHESIS)"] --> B["Load test<br/>(deliberately generate<br/>realistic traffic)"] --> C["Either CONFIRMED<br/>(handles it fine) or a<br/>REAL gap found — safely,<br/>on your own schedule"]
```

This is directly the same underlying philosophy as chaos engineering from the Incident Management series — **test your assumptions deliberately, before reality tests them for you.**

---

## The Types of Performance Tests

A commonly-tested vocabulary distinction — knowing these four terms precisely is high-value.

```mermaid
graph TD
    Types[Performance Test Types] --> Load["LOAD test:<br/>expected, realistic traffic —<br/>'can we handle a normal day?'"]
    Types --> Stress["STRESS test:<br/>push traffic BEYOND expected<br/>levels until something breaks —<br/>'where's our actual ceiling?'"]
    Types --> Spike["SPIKE test:<br/>a SUDDEN, sharp jump in<br/>traffic — 'can we survive a<br/>viral moment or a flash sale?'"]
    Types --> Soak["SOAK test:<br/>moderate load sustained for<br/>a LONG time (hours/days) —<br/>'do we leak memory or<br/>degrade slowly over time?'"]
```

| Test Type | Question It Answers | What It Catches |
|---|---|---|
| **Load test** | Can we handle expected, realistic traffic? | Basic capacity gaps under normal conditions |
| **Stress test** | Where's our actual breaking point? | The real ceiling, and *how* the system fails (gracefully or catastrophically) |
| **Spike test** | Can we survive a sudden, sharp traffic jump? | Autoscaling reaction speed, cold-start problems (above) |
| **Soak test** | Does anything degrade slowly over a long period? | Memory leaks, connection pool exhaustion, disk filling up, log rotation issues |

**Why the soak test specifically catches things the others can't:** a memory leak that adds 1MB per hour is completely invisible in a 10-minute load test — it only shows up after hours or days of sustained operation, which is exactly why soak tests exist as their own distinct category rather than "just run the load test longer."

---

## Load Testing in Practice: k6

**k6** (by Grafana Labs) is one of the most widely used modern load testing tools — scripts are written in JavaScript, and it's designed to integrate cleanly into CI/CD pipelines.

```javascript
// load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },   // ramp up to 100 virtual users
    { duration: '5m', target: 100 },   // stay at 100 for 5 minutes
    { duration: '2m', target: 500 },   // spike to 500 (spike test)
    { duration: '3m', target: 500 },
    { duration: '2m', target: 0 },     // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<400'],   // p95 latency must stay under 400ms
    http_req_failed: ['rate<0.01'],     // error rate must stay under 1%
  },
};

export default function () {
  const res = http.get('https://staging.example.com/api/checkout');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time OK': (r) => r.timings.duration < 500,
  });
  sleep(1);
}
```

```bash
k6 run load-test.js

# Run against a staging environment with custom output for dashboards
k6 run --out prometheus-remote load-test.js
```

**Why the `thresholds` block matters so much:** it turns a load test into an automated **pass/fail gate**, exactly like the security scanning gates from the DevSecOps series — instead of a human eyeballing a results graph, the test itself fails (non-zero exit code) if p95 latency or error rate breach the defined threshold, which is what makes it possible to run load tests automatically as part of a CI/CD pipeline before every release.

---

## Load Testing in Practice: Locust

**Locust** is another very popular option, written in Python, notable for defining user behavior as realistic, weighted "tasks" rather than a single flat script.

```python
# locustfile.py
from locust import HttpUser, task, between

class CheckoutUser(HttpUser):
    wait_time = between(1, 3)

    @task(3)
    def browse_products(self):
        self.client.get("/api/products")

    @task(1)
    def checkout(self):
        self.client.post("/api/checkout", json={"item_id": 42, "qty": 1})
```

```bash
# Run with a web UI for interactive control
locust -f locustfile.py --host https://staging.example.com

# Run headless, for CI, with a fixed user count and spawn rate
locust -f locustfile.py --host https://staging.example.com \
  --users 500 --spawn-rate 50 --run-time 10m --headless
```

**Why the `@task(3)` vs `@task(1)` weighting matters:** it models *realistic* user behavior — in this example, browsing happens 3x more often than checking out, matching real-world usage patterns far more accurately than a load test that hits every endpoint with equal, artificial frequency.

---

## Reading Load Test Results

A results summary from a tool like k6 typically looks something like this — knowing how to read it is as important as knowing how to run the test:

```
     http_req_duration..............: avg=142ms  min=45ms  med=118ms max=2.1s  p(95)=380ms  p(99)=890ms
     http_req_failed.................: 0.42%  ✓ 2098  ✗ 9
     http_reqs.......................: 5250   87.5/s

     ✓ status is 200
     ✓ response time OK
```

**What to actually check, in order:**
1. **Did the thresholds pass or fail?** (the automated pass/fail gate)
2. **p95/p99, not just avg** — exactly the same "averages hide the real story" lesson from the Monitoring Methodologies series, now applied to a load test's own results.
3. **Error rate** — even a small non-zero error rate under load can reveal a real capacity ceiling being approached.
4. **Did the SYSTEM's own dashboards (RED/USE, from the Monitoring Methodologies series) tell the same story as the load test tool's own numbers?** A genuinely important cross-check — if the load test reports fine results but the target system's own dashboards show it was struggling, something about the test itself may be misrepresenting real conditions (e.g., hitting a cache that wouldn't exist for real, more varied traffic).

---

## Testing in Production, Safely

Staging environments are useful but often don't perfectly match production scale, data volume, or traffic patterns. Some organizations deliberately test in production too — carefully.

```mermaid
graph TD
    Techniques["Safe Production<br/>Testing Techniques"] --> T1["Canary load testing:<br/>send synthetic load ONLY<br/>to a small canary deployment,<br/>never the full fleet"]
    Techniques --> T2["Shadow traffic:<br/>mirror a COPY of real<br/>production traffic to a<br/>new system, without<br/>affecting real users at all"]
    Techniques --> T3["Dark launches:<br/>real code runs in production<br/>but its output is discarded,<br/>not shown to users — tests<br/>real capacity/behavior safely"]
```

**Why this connects directly back to the Reliability & Architecture Patterns series' resilience patterns:** production load testing should always be run with circuit breakers, rate limits, and a clear kill switch ready — exactly the resilience patterns from that tutorial — specifically so a production load test that goes wrong can be stopped immediately without becoming a real, unplanned incident.

---

## Common Mistakes

| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Defaulting HPA to CPU utilization for an I/O-bound service | The service can be fully saturated with low CPU, so HPA never scales it up | Scale on the metric that actually reflects real load — queue depth, request rate, custom application metrics |
| No stabilization window on scale-down | Causes flapping — rapid, wasteful scale-down/scale-up cycles | Set a meaningful `stabilizationWindowSeconds` (commonly a few minutes) for scale-down |
| Assuming autoscaling eliminates the need for any static headroom | Autoscaling has real reaction latency (the cold-start chain) — it reduces but doesn't eliminate the need for buffer | Keep a sensible `minReplicas` floor sized to absorb the cold-start window |
| Only running short load tests | Misses slow-developing issues like memory leaks or connection pool exhaustion | Include soak tests (sustained load over hours) as a distinct test category |
| Trusting only average latency/error rate from a load test | Hides the same tail-latency problems averages always hide | Always check p95/p99 and the target system's own dashboards, not just the load tool's summary averages |
| Load testing with unrealistic, uniform traffic patterns | Doesn't reflect real user behavior, can produce misleadingly good (or bad) results | Model realistic, weighted user behavior (like Locust's task weighting) instead of hitting every endpoint equally |

---

## Worked Practice Problems

**Problem 1:** An HPA is configured with a target CPU utilization of 50%. It currently has 6 replicas running, and the metrics server reports current average CPU utilization at 120%. What will HPA calculate as the desired replica count?

*Answer:* Using the formula `desired = ceil(current × (current metric / target metric))`: `ceil(6 × (120/50)) = ceil(6 × 2.4) = ceil(14.4) = 15` replicas.

**Problem 2:** A service scaling on CPU utilization shows HPA never scaling above its minimum replica count, even though users are reporting real timeouts and slow responses during peak hours. What's the most likely explanation, and what would you check?

*Answer:* This is a strong signal the service is I/O-bound, not CPU-bound — it's likely spending most of its time waiting on a slow downstream dependency (a database, an external API) rather than actively computing, so CPU utilization stays low even while the service is genuinely overwhelmed and unable to serve requests promptly. I'd check request queue depth, active connection counts, and downstream dependency latency (using the RED/USE dashboards from the Monitoring Methodologies series) to identify the real bottleneck metric, and reconfigure HPA to scale on that instead of CPU.

**Problem 3:** A team's load test shows p95 latency staying comfortably under their 400ms threshold throughout a 10-minute test, so they conclude the system is ready for a major product launch. What's missing from their validation, and what would you recommend adding?

*Answer:* A 10-minute load test can't catch anything that develops slowly over time — memory leaks, gradually exhausted connection pools, disk filling up from logs, or degrading cache hit rates as data grows. I'd recommend adding a soak test (sustained moderate load over several hours, ideally matching the expected duration of the actual launch event) specifically to catch these slow-developing issues before the real launch does, in addition to the load test they've already run.

---

## Summary — The Complete Capacity Planning Series

- **Autoscaling** automates the reactive side of capacity management — HPA scales pod replica count based on a live metric, using the formula `desired = ceil(current × (current metric / target metric))`.
- **Choose the right scaling metric deliberately** — CPU utilization is the default but frequently wrong for I/O-bound services, exactly the same USE-method trap of trusting utilization alone from the Monitoring Methodologies series.
- Kubernetes autoscaling is **layered**: HPA (pod count) depends on the Scheduler having room, which depends on Cluster Autoscaler adding nodes when needed — understanding all three layers together is a common, high-value interview topic.
- **Cold-start latency is real** — autoscaling reduces, but never fully eliminates, the need for some static headroom, because new capacity always takes real time (scheduling, image pull, app warmup) to actually start helping.
- **Predictive/scheduled scaling** complements reactive HPA for known, repeating traffic patterns, removing the cold-start lag specifically for predictable spikes.
- **Load, stress, spike, and soak tests** each answer a different question — soak tests specifically catch slow-developing problems (memory leaks, resource exhaustion) that shorter tests structurally cannot.
- Tools like **k6** and **Locust** let you define realistic, weighted traffic patterns and set automated pass/fail thresholds, turning load testing into a genuine CI/CD gate rather than a one-off manual exercise.
- **Read load test results the same way you'd read any latency data** — check p95/p99, not just averages, and cross-check against the target system's own real dashboards, not just the load tool's self-reported summary.

This completes the **Capacity Planning & Performance** series. See `questions.md` in this folder for the full interview question bank covering all three parts.