Autoscaling & Load Testing

Table of Contents#

1. From Manual Planning to Automatic Reaction
2. Reactive vs Predictive Autoscaling
3. Horizontal Pod Autoscaler (HPA) — How It Actually Works
4. A Full Worked HPA Example
5. Scaling on Custom and External Metrics
6. Vertical Pod Autoscaler (VPA)
7. Cluster Autoscaler — Scaling the Nodes Themselves
8. The Cold-Start Problem
9. Scale-Down Is Just as Important as Scale-Up
10. Predictive Autoscaling
11. Why Load Testing Exists
12. The Types of Performance Tests
13. Load Testing in Practice: k6
14. Load Testing in Practice: Locust
15. Reading Load Test Results
16. Testing in Production, Safely
17. Common Mistakes
18. Worked Practice Problems
19. Summary — The Complete Capacity Planning Series

---

From Manual Planning to Automatic Reaction#

Parts 1 and 2 covered how to scale and how to calculate how much capacity you need. But traffic doesn't wait for a human to run the numbers and provision servers — it changes minute by minute. Autoscaling is the automation that reacts to real, live demand without a human in the loop, and load testing is how you prove that automation (and the system underneath it) actually works before real users find out the hard way.

Diagram

---

Reactive vs Predictive Autoscaling#

Diagram

Most real systems use reactive autoscaling as the default, foundational layer, and this tutorial spends most of its time there — it's simpler, more broadly applicable, and is what you'll actually configure and operate day to day.

---

Horizontal Pod Autoscaler (HPA) — How It Actually Works#

In Kubernetes, the Horizontal Pod Autoscaler (HPA) is the standard reactive autoscaling mechanism — it watches a metric and adjusts the number of running pod replicas to keep that metric near a target.

Diagram

The Actual Scaling Formula#

HPA's core calculation is simple and worth knowing by heart:

desired replicas = ceil( current replicas × (current metric value / target metric value) )

Worked example: 4 replicas currently running, average CPU usage is 85%, target is 50%.

desired replicas = ceil( 4 × (85 / 50) ) = ceil( 4 × 1.7 ) = ceil(6.8) = 7

HPA scales from 4 to 7 replicas. This exact formula is a commonly asked, concrete interview question — being able to compute it live is a strong, specific signal.

---

A Full Worked HPA Example#

Copy
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-service-hpa
  namespace: checkout
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-service
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

Copy
# Apply it
kubectl apply -f checkout-service-hpa.yaml

# Watch it react live during a load test
kubectl get hpa checkout-service-hpa -n checkout --watch

Why behavior.scaleDown.stabilizationWindowSeconds Matters So Much#

This single setting deserves its own callout, because it directly solves a real, common problem: flapping — rapidly scaling up and back down over and over as a metric bounces around its target.

Diagram

Setting a 300-second (5-minute) stabilization window for scale-down means HPA looks at the metric's behavior over the last 5 minutes and only scales down based on the calculation that would result in the fewest replicas removed — effectively requiring the metric to stay genuinely low for a sustained period before HPA commits to removing capacity. Scale-up deliberately uses no stabilization window (or a very short one) — you want to react to a genuine spike immediately, since the cost of scaling up too slowly (real user impact) is much higher than the cost of scaling up too eagerly (some wasted capacity for a few minutes).

---

Scaling on Custom and External Metrics#

CPU/memory utilization is the default, but it's frequently the wrong signal to scale on — a genuinely important, sometimes-missed point.

Diagram

A strong, senior-level interview line: "CPU utilization is the default HPA metric, but it's frequently the wrong one — an I/O-bound service waiting on a slow downstream dependency can be completely saturated with almost no CPU usage at all, and HPA would never scale it up, exactly the same trap as relying on CPU alone in the USE method from the Monitoring Methodologies series. I'd scale on the metric that actually reflects real load for that specific service — request rate, queue depth, or a custom application-level metric — not blindly default to CPU."

Copy
# Scaling on a custom Prometheus metric (via the Prometheus Adapter)
# instead of CPU — e.g. queue depth for a message consumer
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-processor-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-processor
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: External
      external:
        metric:
          name: rabbitmq_queue_messages_ready
          selector:
            matchLabels:
              queue: orders
        target:
          type: AverageValue
          averageValue: "30"

This says: "keep the average number of ready-to-process messages per replica around 30 — add more replicas if the backlog grows, remove them as it shrinks."

---

Vertical Pod Autoscaler (VPA)#

A different, complementary tool: instead of adding more pod replicas (horizontal), VPA adjusts the CPU/memory requests and limits of individual pods (vertical) — automatically right-sizing them based on observed real usage.

Diagram

Important, commonly-tested caveat: VPA and HPA scaling on the same metric (e.g., both watching CPU) can conflict — VPA resizing a pod's CPU request changes the denominator HPA's utilization percentage is calculated against, potentially causing unstable interactions between the two. The standard, safe practice: use VPA for right-sizing requests/limits (often in a "recommendation only" mode reviewed by a human, rather than fully automatic), and use HPA for reactive scaling — avoid pointing both at the exact same metric for the exact same workload simultaneously.

---

Cluster Autoscaler — Scaling the Nodes Themselves#

HPA and VPA operate at the pod level — but pods need somewhere to actually run. If a cluster's nodes are all full, HPA can create as many new pod replicas as it wants; they'll simply sit Pending, unable to be scheduled anywhere. Cluster Autoscaler solves this one layer down, by adding or removing actual worker nodes (VMs) from the cluster.

Diagram

Why this three-layer relationship (HPA → Cluster Autoscaler → cloud provider) is a genuinely important thing to be able to draw out in an interview: it's a very common question to ask "what happens if HPA wants to scale up but there's no room?" — the answer demonstrates you understand that Kubernetes autoscaling isn't one single mechanism, it's a layered system, and each layer only solves its own specific piece of the puzzle.

---

The Cold-Start Problem#

Autoscaling isn't instantaneous — there's real latency between "demand increased" and "new capacity is actually serving traffic," and this latency compounds across the layers above.

Diagram

In this realistic example, ~55 seconds pass between the spike starting and new capacity actually helping — and if a new node also needs to be provisioned (Cluster Autoscaler), that can add another 1-3 minutes on top, depending on the cloud provider. This is exactly why headroom (Part 2) still matters even with autoscaling in place — autoscaling reduces how much static buffer you need, but it doesn't eliminate the need entirely, because it can't react in zero time.

Practical mitigations worth naming:

• Keep minReplicas high enough to absorb the cold-start window's worth of extra traffic on its own.
• Use smaller, faster-starting container images (connects to the Dockerfile hardening discussion in the DevSecOps series — smaller images pull faster).
• Consider over-provisioning a small buffer of already-running "pause" pods with low priority, specifically so the cluster already has spare node capacity ready before it's needed (a known technique sometimes called "cluster overprovisioning").

---

Scale-Down Is Just as Important as Scale-Up#

Scaling up gets most of the attention (it's what prevents outages), but scaling back down correctly matters just as much for cost and stability.

Diagram

This is exactly the same headroom/cost tradeoff from Part 2, just applied to the speed of removing capacity rather than the amount of capacity provisioned upfront.

---

Predictive Autoscaling#

For traffic with known, repeating patterns (a daily 9am spike, a weekly Monday-morning surge, a predictable seasonal event), predictive/scheduled autoscaling removes the cold-start lag entirely by scaling ahead of the known pattern, rather than reacting after it starts.

Copy
# A simple scheduled scaling approach — e.g. using a CronJob to
# pre-scale BEFORE a known daily traffic pattern, then letting
# reactive HPA take over for anything beyond the predicted baseline
kubectl patch hpa checkout-service-hpa -n checkout \
  --patch '{"spec":{"minReplicas": 15}}'
# (scheduled to run at 08:50, ahead of a reliable 09:00 traffic spike)

A strong interview framing: "I'd use predictive/scheduled scaling as a complement to reactive HPA, not a replacement — scheduled scaling handles the known patterns (raising the floor, i.e. minReplicas, ahead of a predictable spike), while reactive HPA still handles anything above that baseline, including genuinely unpredictable spikes scheduled scaling could never anticipate."

---

Why Load Testing Exists#

Every autoscaling configuration and capacity plan in this whole series is a hypothesis until it's actually tested under real load. Load testing is how you validate that hypothesis safely, on your own schedule, instead of finding out it was wrong during a real traffic spike.

Diagram

This is directly the same underlying philosophy as chaos engineering from the Incident Management series — test your assumptions deliberately, before reality tests them for you.

---

The Types of Performance Tests#

A commonly-tested vocabulary distinction — knowing these four terms precisely is high-value.

Diagram

Why the soak test specifically catches things the others can't: a memory leak that adds 1MB per hour is completely invisible in a 10-minute load test — it only shows up after hours or days of sustained operation, which is exactly why soak tests exist as their own distinct category rather than "just run the load test longer."

---

Load Testing in Practice: k6#

k6 (by Grafana Labs) is one of the most widely used modern load testing tools — scripts are written in JavaScript, and it's designed to integrate cleanly into CI/CD pipelines.

Copy
// load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },   // ramp up to 100 virtual users
    { duration: '5m', target: 100 },   // stay at 100 for 5 minutes
    { duration: '2m', target: 500 },   // spike to 500 (spike test)
    { duration: '3m', target: 500 },
    { duration: '2m', target: 0 },     // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<400'],   // p95 latency must stay under 400ms
    http_req_failed: ['rate<0.01'],     // error rate must stay under 1%
  },
};

export default function () {
  const res = http.get('https://staging.example.com/api/checkout');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time OK': (r) => r.timings.duration < 500,
  });
  sleep(1);
}

Copy
k6 run load-test.js

# Run against a staging environment with custom output for dashboards
k6 run --out prometheus-remote load-test.js

Why the thresholds block matters so much: it turns a load test into an automated pass/fail gate, exactly like the security scanning gates from the DevSecOps series — instead of a human eyeballing a results graph, the test itself fails (non-zero exit code) if p95 latency or error rate breach the defined threshold, which is what makes it possible to run load tests automatically as part of a CI/CD pipeline before every release.

---

Load Testing in Practice: Locust#

Locust is another very popular option, written in Python, notable for defining user behavior as realistic, weighted "tasks" rather than a single flat script.

Copy
# locustfile.py
from locust import HttpUser, task, between

class CheckoutUser(HttpUser):
    wait_time = between(1, 3)

    @task(3)
    def browse_products(self):
        self.client.get("/api/products")

    @task(1)
    def checkout(self):
        self.client.post("/api/checkout", json={"item_id": 42, "qty": 1})

Copy
# Run with a web UI for interactive control
locust -f locustfile.py --host https://staging.example.com

# Run headless, for CI, with a fixed user count and spawn rate
locust -f locustfile.py --host https://staging.example.com \
  --users 500 --spawn-rate 50 --run-time 10m --headless

Why the @task(3) vs @task(1) weighting matters: it models realistic user behavior — in this example, browsing happens 3x more often than checking out, matching real-world usage patterns far more accurately than a load test that hits every endpoint with equal, artificial frequency.

---

Reading Load Test Results#

A results summary from a tool like k6 typically looks something like this — knowing how to read it is as important as knowing how to run the test:

     http_req_duration..............: avg=142ms  min=45ms  med=118ms max=2.1s  p(95)=380ms  p(99)=890ms
     http_req_failed.................: 0.42%  ✓ 2098  ✗ 9
     http_reqs.......................: 5250   87.5/s

     ✓ status is 200
     ✓ response time OK

What to actually check, in order:

1. Did the thresholds pass or fail? (the automated pass/fail gate)
2. p95/p99, not just avg — exactly the same "averages hide the real story" lesson from the Monitoring Methodologies series, now applied to a load test's own results.
3. Error rate — even a small non-zero error rate under load can reveal a real capacity ceiling being approached.
4. Did the SYSTEM's own dashboards (RED/USE, from the Monitoring Methodologies series) tell the same story as the load test tool's own numbers? A genuinely important cross-check — if the load test reports fine results but the target system's own dashboards show it was struggling, something about the test itself may be misrepresenting real conditions (e.g., hitting a cache that wouldn't exist for real, more varied traffic).

---

Testing in Production, Safely#

Staging environments are useful but often don't perfectly match production scale, data volume, or traffic patterns. Some organizations deliberately test in production too — carefully.

Diagram

Why this connects directly back to the Reliability & Architecture Patterns series' resilience patterns: production load testing should always be run with circuit breakers, rate limits, and a clear kill switch ready — exactly the resilience patterns from that tutorial — specifically so a production load test that goes wrong can be stopped immediately without becoming a real, unplanned incident.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: An HPA is configured with a target CPU utilization of 50%. It currently has 6 replicas running, and the metrics server reports current average CPU utilization at 120%. What will HPA calculate as the desired replica count?

Answer: Using the formula desired = ceil(current × (current metric / target metric)): ceil(6 × (120/50)) = ceil(6 × 2.4) = ceil(14.4) = 15 replicas.

Problem 2: A service scaling on CPU utilization shows HPA never scaling above its minimum replica count, even though users are reporting real timeouts and slow responses during peak hours. What's the most likely explanation, and what would you check?

Answer: This is a strong signal the service is I/O-bound, not CPU-bound — it's likely spending most of its time waiting on a slow downstream dependency (a database, an external API) rather than actively computing, so CPU utilization stays low even while the service is genuinely overwhelmed and unable to serve requests promptly. I'd check request queue depth, active connection counts, and downstream dependency latency (using the RED/USE dashboards from the Monitoring Methodologies series) to identify the real bottleneck metric, and reconfigure HPA to scale on that instead of CPU.

Problem 3: A team's load test shows p95 latency staying comfortably under their 400ms threshold throughout a 10-minute test, so they conclude the system is ready for a major product launch. What's missing from their validation, and what would you recommend adding?

Answer: A 10-minute load test can't catch anything that develops slowly over time — memory leaks, gradually exhausted connection pools, disk filling up from logs, or degrading cache hit rates as data grows. I'd recommend adding a soak test (sustained moderate load over several hours, ideally matching the expected duration of the actual launch event) specifically to catch these slow-developing issues before the real launch does, in addition to the load test they've already run.

---

Summary — The Complete Capacity Planning Series#

• Autoscaling automates the reactive side of capacity management — HPA scales pod replica count based on a live metric, using the formula desired = ceil(current × (current metric / target metric)).
• Choose the right scaling metric deliberately — CPU utilization is the default but frequently wrong for I/O-bound services, exactly the same USE-method trap of trusting utilization alone from the Monitoring Methodologies series.
• Kubernetes autoscaling is layered: HPA (pod count) depends on the Scheduler having room, which depends on Cluster Autoscaler adding nodes when needed — understanding all three layers together is a common, high-value interview topic.
• Cold-start latency is real — autoscaling reduces, but never fully eliminates, the need for some static headroom, because new capacity always takes real time (scheduling, image pull, app warmup) to actually start helping.
• Predictive/scheduled scaling complements reactive HPA for known, repeating traffic patterns, removing the cold-start lag specifically for predictable spikes.
• Load, stress, spike, and soak tests each answer a different question — soak tests specifically catch slow-developing problems (memory leaks, resource exhaustion) that shorter tests structurally cannot.
• Tools like k6 and Locust let you define realistic, weighted traffic patterns and set automated pass/fail thresholds, turning load testing into a genuine CI/CD gate rather than a one-off manual exercise.
• Read load test results the same way you'd read any latency data — check p95/p99, not just averages, and cross-check against the target system's own real dashboards, not just the load tool's self-reported summary.

This completes the Capacity Planning & Performance series. See questions.md in this folder for the full interview question bank covering all three parts.