# Networking Deep Dive — Part 2: Load Balancing & Traffic Management

> **Series:** Networking Deep Dive (2 of 5)
> **Part 1:** `01-routing-bgp-and-internet-architecture.md` — Routing, BGP & the Internet's Architecture
> **Part 2:** This file — Load Balancing & Traffic Management
> **Part 3:** `03-cloud-network-architecture.md` — Cloud Network Architecture
> **Part 4:** `04-tls-http-evolution-and-secure-transport.md` — TLS, mTLS & the HTTP Protocol Evolution
> **Part 5:** `05-cdn-edge-and-service-mesh-dataplanes.md` — CDN, Edge Networking & Service Mesh Data Planes
> **Questions:** `questions.md`

## Table of Contents

1. [Why Load Balancing Is a Separate Layer From Routing](#why-load-balancing-is-a-separate-layer-from-routing)
2. [L4 vs. L7 Load Balancing — the Fundamental Tradeoff](#l4-vs-l7-load-balancing-the-fundamental-tradeoff)
3. [Load Balancing Algorithms, One at a Time](#load-balancing-algorithms-one-at-a-time)
4. [Consistent Hashing — Why Caches and Sharded Stores Need a Different Algorithm](#consistent-hashing-why-caches-and-sharded-stores-need-a-different-algorithm)
5. [Health Checks — Active, Passive, and Why They Lie](#health-checks-active-passive-and-why-they-lie)
6. [Connection Draining — Removing a Backend Without Dropping Live Requests](#connection-draining-removing-a-backend-without-dropping-live-requests)
7. [Session Affinity — the Tradeoff Against Even Distribution](#session-affinity-the-tradeoff-against-even-distribution)
8. [TLS Termination Location — Edge, Re-Encrypt, or Passthrough](#tls-termination-location-edge-re-encrypt-or-passthrough)
9. [Cloud-Managed Load Balancers — ALB, NLB, and Their Equivalents](#cloud-managed-load-balancers-alb-nlb-and-their-equivalents)
10. [Running Your Own — HAProxy and nginx](#running-your-own-haproxy-and-nginx)
11. [Envoy — the Modern Programmable Data Plane](#envoy-the-modern-programmable-data-plane)
12. [Circuit Breaking and Outlier Detection](#circuit-breaking-and-outlier-detection)
13. [Global Server Load Balancing — Spreading Load Across Regions](#global-server-load-balancing-spreading-load-across-regions)
14. [Rate Limiting and Traffic Shaping at the Load Balancer](#rate-limiting-and-traffic-shaping-at-the-load-balancer)
15. [Retries, Timeouts, and the Retry Storm Problem](#retries-timeouts-and-the-retry-storm-problem)
16. [Weighted Traffic Shifting — Canary and Blue-Green at the Load-Balancing Layer](#weighted-traffic-shifting-canary-and-blue-green-at-the-load-balancing-layer)
17. [Full Worked Scenario: Redesigning checkout-service's Load Balancing After a Cascading Failure](#full-worked-scenario-redesigning-checkout-services-load-balancing-after-a-cascading-failure)
18. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
19. [Worked Practice Problems](#worked-practice-problems)
20. [Summary and What's Next](#summary-and-whats-next)

---

## Why Load Balancing Is a Separate Layer From Routing

**Part 1 answered "how does traffic find its way to my network at all" — this chapter answers the very next
question: once traffic arrives, how does it get spread correctly across the many backend instances actually
running your service.** These are genuinely different problems, solved by different mechanisms, and
conflating them is a common source of confused architecture diagrams. BGP and routing (Part 1) operate on
*networks* — prefixes, ASes, physical paths — with no concept of "this specific request" or "this specific
backend process." Load balancing operates one layer up: given that traffic has already arrived at your
front door, which of `checkout-service`'s twelve currently-running pods should actually handle *this*
request, right now, given their current health and load.

Throughout this chapter, the throughline returns to `checkout-service`, `catalog-service`, and
`inventory-service` — the same three-service platform from Part 1, now viewed from inside one region: a
fleet of backend instances behind a load balancer, not yet the multi-region picture Part 1's anycast
example covered.

```mermaid
flowchart LR
    Internet["Internet traffic<br/>(Part 1's job: get it HERE)"] --> LB["Load balancer<br/>(THIS chapter's job:<br/>pick a backend)"]
    LB --> Pod1["checkout-service<br/>pod 1"]
    LB --> Pod2["checkout-service<br/>pod 2"]
    LB --> Pod3["checkout-service<br/>pod 3"]
    LB -.->|"unhealthy — skipped"| Pod4["checkout-service<br/>pod 4"]

    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c
    class Internet,LB info
    class Pod1,Pod2,Pod3 ok
    class Pod4 crit
```

*Routing (Part 1) is a solved problem by the time traffic reaches the load balancer's front interface —
this chapter starts from that point.*

## L4 vs. L7 Load Balancing — the Fundamental Tradeoff

**The single most important design decision in load balancing is which layer of the stack the balancer
operates at, because it determines what the balancer is allowed to see and how expensive each decision is
to make.**

| | **L4 (transport layer)** | **L7 (application layer)** |
|---|---|---|
| What it sees | IP addresses, ports, TCP/UDP — never inspects payload | Full HTTP request: method, path, headers, cookies, body |
| Decision granularity | Per-connection | Per-request — a single TCP connection can carry many independent routing decisions (HTTP/2/3 multiplexing) |
| Routing logic possible | Which backend gets this *connection* | Path-based routing (`/api/checkout/*` → checkout-service), header-based routing, retries, request rewriting |
| Can terminate TLS | No — passes encrypted bytes through unmodified | Yes — must decrypt to read the request, then usually re-encrypts to the backend |
| Performance overhead | Very low — effectively just NAT/forwarding | Higher — parses, may buffer, may re-encrypt |
| Client IP visibility | Preserved by default in many implementations | Requires explicit forwarding (`X-Forwarded-For`, PROXY protocol) since the L7 balancer is now itself the TCP peer |

**Neither is strictly "better" — they solve different problems, and a production architecture frequently
uses both, layered.** An L4 balancer is the right tool when the traffic isn't HTTP at all (a raw TCP
database protocol, gRPC over HTTP/2 handled below the balancer, a UDP-based protocol), when preserving the
original client IP matters more than routing intelligence, or when the sheer packet-forwarding performance
of skipping payload inspection matters at extreme scale. An L7 balancer is the right tool the moment routing
needs to depend on *what's actually being requested* — which is most of the time for a modern HTTP API
surface, and is a strict prerequisite for path-based routing, canary/weighted traffic splitting by header,
or per-route retry policy.

⚙️ Concretely: `checkout-service`'s public API and `catalog-service`'s public API can share a single L7
load balancer that inspects the request path and routes `/checkout/*` to one target group and `/catalog/*`
to another — something an L4 balancer has no way to do, since it never looks past the IP/port headers.

## Load Balancing Algorithms, One at a Time

Once a request/connection has arrived at a load balancer with multiple healthy backends, the balancer must
pick one. Each algorithm optimizes for a different assumption about the workload:

| Algorithm | How it picks | Best fit | Weak point |
|---|---|---|---|
| **Round robin** | Cycles through backends in fixed order | Uniform, short-lived, stateless requests where every backend has identical capacity | Ignores actual current load — a backend mid-processing a slow request still gets the next one on schedule |
| **Weighted round robin** | Round robin, but backends with a higher weight receive proportionally more requests | Heterogeneous backend capacity (some instances are bigger/faster) | Weights are usually static — doesn't adapt to real-time load changes |
| **Least connections** | Sends the next request to whichever backend currently has the fewest open connections | Long-lived or variable-duration requests, where connection count is a real proxy for load | Assumes all connections are equally expensive — a backend with few connections each doing heavy work can still be the most loaded one |
| **Least response time / least outstanding requests** | Combines connection count with observed recent latency | The workloads least-connections handles poorly | More overhead to compute; needs continuous latency sampling |
| **Random / power of two choices** | Picks two backends at random, sends the request to whichever has fewer active connections | Extremely large backend fleets, where full state tracking across every instance is itself a bottleneck | Slightly less optimal than true least-connections, but scales far better with fleet size |
| **IP hash / consistent hash** | Hashes a client attribute (source IP, a cookie, a request key) to deterministically pick the same backend for the same client | Session affinity without a session store; cache/shard locality | Uneven distribution if the hashed attribute isn't evenly spread (e.g. many users behind one corporate NAT IP) |

> [!TIP]
> **Best practice: default to least-connections (or "power of two choices" at very large fleet sizes) unless
> you have a specific, verified reason to reach for round robin.** Round robin's core assumption — every
> request costs roughly the same amount of backend work — is false for almost any real API surface, where
> request cost varies by endpoint, payload size, and cache hit/miss. Least-connections adapts to real,
> observed load instead of assuming it away, and the added computational cost is negligible at normal
> production scale.

```mermaid
sequenceDiagram
    participant LB as Load Balancer
    participant P1 as Pod 1 (2 active conns)
    participant P2 as Pod 2 (7 active conns)
    participant P3 as Pod 3 (1 active conn)

    Note over LB: New request arrives
    LB->>LB: Round robin - next in rotation is Pod 2
    Note over LB,P2: Sent to the BUSIEST pod anyway - bad luck of the rotation
    Note over LB: Same request, with least-connections instead
    LB->>P3: Least connections - Pod 3 has fewest active conns
    Note over LB,P3: Correctly avoids the already-busy pod
```

*Same three backends, same instant — round robin's blind rotation can hand a request to the busiest
backend purely by scheduling coincidence; least-connections cannot.*

## Consistent Hashing — Why Caches and Sharded Stores Need a Different Algorithm

**A naive hash-based balancer — `hash(key) % number_of_backends` — has a catastrophic failure mode the
moment the backend count changes: adding or removing even one backend reshuffles the modulo result for
*almost every* key, not just the keys that logically belong to the changed backend.** For a stateless web
request this is a non-issue (any healthy backend can serve it). For a caching layer or a sharded data store
— where "which backend has this key's data" is the entire point — a naive rehash on every scaling event
means a near-total cache wipe or a full data reshuffle, exactly when the system is least able to absorb the
extra load.

**Consistent hashing solves this by hashing both the backends and the keys onto the same fixed conceptual
ring**, and assigning each key to the nearest backend clockwise on that ring. Adding or removing one backend
only reassigns the keys that fell between it and its immediate neighbor on the ring — a small, bounded
fraction of the total keyspace, not a full reshuffle.

```mermaid
flowchart TD
    subgraph Ring["Consistent Hash Ring"]
        direction TB
        A["Backend A"] --> K1["Key 1"] --> K2["Key 2"] --> B["Backend B"] --> K3["Key 3"] --> C["Backend C"] --> K4["Key 4"] --> A
    end
    Note["Adding Backend D between B and C<br/>only moves Key 3 - not Key 1, Key 2, or Key 4"]

    classDef accent fill:#f0e9fb,stroke:#6d43c0,color:#10161c
    classDef muted fill:#eaeef1,stroke:#c3ccd4,color:#10161c
    class A,B,C accent
    class K1,K2,K3,K4,Note muted
```

*Each key belongs to the next backend clockwise on the ring — a topology change only disturbs the keys
adjacent to the change, bounding the blast radius of any scaling event.*

**Envoy's `ring_hash` and Google's Maglev algorithm are the two production-grade implementations most
platform teams will actually encounter**, rather than hand-rolling one. Maglev (originally published by
Google, and adopted widely since, including inside Envoy as an alternative to ring hash) trades a small
amount of theoretical rebalancing optimality for dramatically faster lookup and construction time, and is
generally the recommended default for a large backend fleet over the classic ring-hash approach.

⚠️ **From the trenches**: `inventory-service`'s platform team ran a Redis-backed cache in front of a
frequently-queried product catalog, load-balanced across cache nodes with a plain modulo hash rather than
consistent hashing — a decision made early, before cache-node scaling was a regular event. During a routine
capacity add (3 nodes → 4), cache hit rate collapsed from 94% to under 10% for roughly fifteen minutes, and
`inventory-service`'s own database — never sized for that query volume — briefly became the bottleneck for
the entire platform, including `checkout-service` requests that depend on inventory data. The immediate
cause was the modulo hash reassigning nearly every key to a different node the instant the fourth node
joined; the underlying condition was that nobody had revisited the caching layer's hashing strategy since
it was first stood up, because cache scaling had always previously happened during low-traffic maintenance
windows small enough that the hit-rate dip went unnoticed. Migrating to consistent hashing (Maglev, via the
caching layer's Envoy sidecar) turned the next scaling event's hit-rate dip into a low-single-digit-percent
blip, fully absorbed without any downstream impact.

## Health Checks — Active, Passive, and Why They Lie

A load balancer only routes to backends it believes are healthy — the mechanism that belief is built on is
worth understanding in detail, because a wrong belief (routing to a backend that's actually broken, or
withholding traffic from one that's actually fine) is one of the most common real production incidents in
this whole chapter's territory.

| Type | Mechanism | Detects |
|---|---|---|
| **Active health check** | The balancer itself periodically sends a synthetic probe (a TCP connect, an HTTP GET to `/healthz`) on its own schedule, independent of real traffic | Backend fully down, or explicitly failing its own health endpoint |
| **Passive health check** | The balancer observes *real* request outcomes (connection refused, timeout, a run of 5xx responses) and ejects a backend based on live traffic patterns | Degraded-but-not-fully-down backends that would still pass a trivial synthetic probe |

**Neither type alone is sufficient in production.** An active check hitting a trivially cheap `/healthz`
endpoint can report "healthy" while the backend's *actual* request-handling path is completely broken — a
classic case is a health endpoint hardcoded to return 200 regardless of downstream dependency state, while
the real application logic depends on a database connection pool that's exhausted. A passive check alone
can't proactively catch a backend that's about to fail before it actually serves bad traffic to real users,
and needs a reasonable sample size before it can distinguish "genuinely unhealthy" from "one unlucky
request." Production load balancers (ALB, Envoy, HAProxy) run both simultaneously: active checks for
baseline liveness, passive/outlier detection (covered later in this chapter) for real-traffic-driven
ejection.

> [!IMPORTANT]
> **A health check endpoint that always returns 200 is worse than no health check at all — it gives false
> confidence.** The health check should exercise a meaningful, representative slice of what the backend
> actually needs to work (a real, cheap query against the actual database connection pool the app uses, not
> a separate hardcoded-success handler) — otherwise the load balancer will happily keep sending real traffic
> to a backend that cannot actually serve it.

⚠️ **From the trenches**: `checkout-service`'s `/healthz` endpoint was implemented, correctly by the letter
of most guidance, as a lightweight endpoint that didn't touch the database — "keep health checks fast and
cheap" is common (and generally correct) advice. During an incident where the connection pool to the
payment-processing database was fully exhausted, every real checkout request failed with a pool-timeout
error, while `/healthz` — which never touched that pool — kept returning 200 the entire time. The load
balancer, working exactly as designed, kept routing full traffic to backends that were 100% unable to
complete a real checkout. The fix wasn't "make health checks slow and heavy" (the original instinct is
correct in general) — it was adding one cheap, specific signal: a fast, non-blocking check of the
connection pool's *current available connection count* (an in-memory counter, not a real query), which
directly reflects the actual failure mode without adding real database load to every health probe. The
lesson generalizes: a health check needs to be cheap *and* representative of the actual dependency most
likely to fail — those two goals aren't in tension nearly as often as teams assume.

## Connection Draining — Removing a Backend Without Dropping Live Requests

**Removing a backend from a load balancer's rotation — for a deploy, a scale-down, or a failed health check
— has two very different ways to happen: an abrupt removal that kills in-flight connections immediately, or
a graceful drain that stops sending *new* traffic while letting existing connections finish naturally.**
Every production load balancer supports a configurable **deregistration delay** (AWS's term;
"connection draining" and "graceful shutdown" are the equivalent concepts elsewhere): once a backend is
marked for removal, the balancer stops routing new requests to it immediately, but keeps existing
connections open and lets them complete, up to a maximum timeout, before forcibly terminating anything
still outstanding.

```mermaid
sequenceDiagram
    participant LB as Load Balancer
    participant Pod as checkout-service Pod (being replaced)
    participant NewPod as New Pod

    Note over Pod: Deploy triggers replacement
    LB->>Pod: Mark for removal - deregistration starts
    Note over LB: STOP sending new requests to Pod
    LB->>NewPod: New requests go here instead
    Note over Pod: Existing in-flight requests<br/>finish naturally (up to timeout)
    Pod->>LB: Last in-flight request completes
    LB->>Pod: Now safe to fully terminate
```

*The window between "marked for removal" and "fully terminated" is exactly the deregistration delay — set
too short, and it kills legitimate slow requests; set too long, and deploys/scale-downs take
correspondingly longer to fully complete.*

> [!TIP]
> **Best practice: size the deregistration delay to comfortably exceed your slowest legitimate request's
> p99 duration, not your median.** A deregistration delay tuned to the median request time will routinely
> and silently truncate the slowest few percent of real requests during every single deploy — a real,
> recurring, self-inflicted source of user-visible errors that only ever happen during deploys, which makes
> them notoriously easy to misdiagnose as "something about the new version" when the actual cause is
> draining being too aggressive for the *old* version's tail latency.

## Session Affinity — the Tradeoff Against Even Distribution

**Session affinity (sticky sessions) deliberately routes every request from the same client to the same
backend, trading perfectly even load distribution for the ability to keep state local to one instance.**
The two common mechanisms:

- **Cookie-based affinity**: the load balancer sets (or the application sets, and the balancer reads) a
  cookie identifying which backend a client was routed to, and honors it on every subsequent request.
- **IP-hash affinity**: the balancer hashes the client's source IP to deterministically pick a backend —
  no cookie needed, but breaks down when many clients share one visible IP (a corporate NAT, a mobile
  carrier's CGNAT pool), all landing on the same backend regardless of actual load.

The right call depends entirely on whether the backend actually needs local state. A genuinely stateless
service (most well-designed modern APIs, including `checkout-service` and `catalog-service` in this
series's own architecture, which push session state to a shared cache/database rather than in-process
memory) has no need for affinity at all, and enabling it anyway only adds an uneven-distribution risk for no
benefit. A backend that legitimately holds in-memory state per client (a WebSocket connection with
server-side buffered state, an in-process shopping-cart cache used specifically to avoid a database round
trip) genuinely needs it — but the better long-term architectural fix, where practical, is usually to
externalize that state (a shared Redis-backed session store) so *any* backend can serve *any* request,
removing the need for affinity entirely and restoring even load distribution.

## TLS Termination Location — Edge, Re-Encrypt, or Passthrough

Chapter 4 of this series covers the TLS handshake and certificate mechanics themselves in depth — this
section covers the load-balancing-layer decision that has to be made regardless of those details: **where,
physically, does encrypted traffic actually get decrypted, relative to the load balancer?** Three
architectures are in common production use, and the choice affects security posture, observability, and
operational overhead in ways worth understanding before defaulting to whichever one a cloud console makes
easiest.

| Architecture | Where decryption happens | What the LB can see | Tradeoff |
|---|---|---|---|
| **Edge termination** | At the load balancer itself; traffic to backends is plain HTTP | Full request content — enables L7 routing, header inspection, WAF rules | Traffic inside the VPC/cluster is unencrypted — acceptable only when the internal network is itself trusted (a private subnet with tight security groups) |
| **Re-encryption (TLS bridging)** | Decrypted at the LB, then re-encrypted with a (often internal-CA-issued) certificate to the backend | Full L7 visibility, plus encryption maintained end-to-end | Doubles the TLS handshake/CPU cost; requires managing a second certificate (often via a service mesh's internal CA, covered in Chapter 5) |
| **Passthrough** | The LB never decrypts at all — forwards the raw encrypted TCP stream (SNI-based routing is still possible without decrypting, since SNI is sent in cleartext during the handshake) | No visibility into request content — L4 routing decisions only | Backend does its own TLS termination; the only option when true end-to-end encryption with no intermediate decryption point is a hard requirement |

> [!IMPORTANT]
> **Edge-terminate-only is a common, real security gap, not a theoretical one.** A "defense in depth"
> mindset — assuming a compromised host *inside* the VPC is a real, planned-for scenario, not just an
> external-attacker one — argues for re-encryption to the backend even when the internal network is
> nominally trusted, specifically so a compromised or misconfigured internal component can't passively
> sniff plaintext application traffic. This is precisely the justification behind a service mesh's default
> mTLS-everywhere posture, covered in full in Chapter 5 — re-encryption at the load-balancing layer is the
> same idea, applied one layer earlier in the request path.

`checkout-service`'s platform team runs re-encryption specifically for its payment-processing traffic path
(PCI-DSS scope requires it) while accepting edge-termination for `catalog-service`'s public product-browsing
API, where the internal traffic carries no sensitive payload — a deliberate, risk-scoped choice rather than
a uniform policy applied everywhere regardless of what's actually being protected.

## Cloud-Managed Load Balancers — ALB, NLB, and Their Equivalents

Every major cloud provider offers managed L4 and L7 load balancers, sparing a platform team from operating
the balancer software itself. AWS's naming is the one most engineers encounter first and worth knowing in
detail, since the same conceptual split (a "dumb," extremely fast L4 balancer vs. a smarter, HTTP-aware L7
one) recurs under different names on every cloud:

| | **Application Load Balancer (ALB)** | **Network Load Balancer (NLB)** |
|---|---|---|
| Layer | L7 | L4 |
| Protocol awareness | HTTP/HTTPS/gRPC — reads the request | TCP/UDP/TLS-passthrough — no payload inspection |
| Path/host-based routing | Yes — the primary reason to reach for it | No |
| Client IP preservation | Requires `X-Forwarded-For` header | Preserved natively |
| Static IP | No — DNS name only, backed by rotating IPs | Yes — one static IP per AZ, a prerequisite for some anycast/allowlisting setups |
| Health checks | HTTP-aware (expects a status code) | TCP or HTTP, faster detection intervals |
| Typical use | Standard public web/API traffic | Extreme low-latency needs, non-HTTP protocols, or when a static IP is a hard requirement |

GCP's equivalent split is the External HTTP(S) Load Balancer (L7, itself built on a globally-distributed
anycast frontend — a direct, managed application of Part 1's anycast mechanism, and notably a single global
resource rather than a per-region one, unlike AWS's ALB) versus the External TCP/UDP Network Load Balancer
(L4, regional, backed by Maglev — the same consistent-hashing algorithm covered earlier in this chapter,
originally built at Google specifically for this product before Envoy adopted it too). Azure's split is the
Application Gateway (L7, with an optional integrated Web Application Firewall tier) versus the Azure Load
Balancer (L4, itself available in both a regional and a cross-region/"global" tier since Microsoft added
global-tier support). The conceptual split — and the decision criteria in the table above — carries across
all three clouds even though the product names and exact regional-vs-global defaults differ; Chapter 3 of
this series covers where each of these sits inside the surrounding VPC/VNet architecture in more depth.

> [!NOTE]
> **GCP's L7 load balancer being global by default, versus AWS's ALB being regional, is a real
> architectural difference, not just naming.** A single GCP HTTP(S) Load Balancer resource can front
> backends in multiple regions directly, with GCP's own anycast frontend handling the geographic routing —
> functionally folding part of this chapter's GSLB discussion (below) into the load balancer itself. On
> AWS, the equivalent multi-region routing is a separate, composed layer (Global Accelerator or Route 53,
> in front of per-region ALBs) — neither approach is strictly better, but assuming AWS's regional model
> when reading GCP documentation (or vice versa) is a common, confusing mistake for engineers moving
> between the two.

⚙️ **A subtlety worth knowing before it surprises you**: an NLB placed *in front of* an ALB (a real,
supported pattern — used to get NLB's static IP for an allowlist while keeping ALB's L7 routing behind it)
means the ALB's health checks now see the NLB as the "client," not the real end user, and the NLB's own
health checks against the ALB need to be configured on the correct port and protocol independently — a
misconfigured health check at this specific junction is a recurring, hard-to-spot cause of an otherwise
perfectly healthy ALB silently receiving zero traffic.

## Running Your Own — HAProxy and nginx

Cloud-managed load balancers cover the overwhelming majority of production needs, but a platform team still
frequently runs a **self-managed L7 proxy layer** — inside a Kubernetes cluster as an Ingress controller, in
front of a set of VMs, or specifically for capabilities a managed LB doesn't expose. **HAProxy** and
**nginx** are the two long-standing, battle-tested choices, and both remain in wide production use even as
Envoy (next section) has captured much of the newer service-mesh-adjacent use case.

- **HAProxy** is purpose-built as a load balancer first — its configuration model is organized directly
  around frontends, backends, and the load-balancing algorithm applied between them (`balance leastconn`,
  `balance roundrobin`, and several more, directly matching this chapter's algorithm section). It's
  frequently the choice when load balancing itself, plus fine-grained L4/L7 traffic control, is the primary
  job.
- **nginx** started as a web server and reverse proxy and grew load-balancing capability on top of that
  foundation — a natural fit when a team already needs nginx's other capabilities (static file serving, TLS
  termination, request rewriting) and wants load balancing as one more configured feature rather than a
  separate dedicated component.

```bash
# A representative HAProxy backend definition — direct application of this
# chapter's algorithm and health-check concepts as real configuration
backend checkout_service
    balance leastconn
    option httpchk GET /healthz
    http-check expect status 200
    server pod1 10.0.1.10:8080 check inter 5s fall 3 rise 2
    server pod2 10.0.1.11:8080 check inter 5s fall 3 rise 2
    server pod3 10.0.1.12:8080 check inter 5s fall 3 rise 2
```

*`fall 3` / `rise 2` directly implements the "don't flap on a single unlucky probe" lesson from the health
checks section — three consecutive failures before a backend is marked down, two consecutive successes
before it's trusted again.*

## Envoy — the Modern Programmable Data Plane

**Envoy is a purpose-built L4/L7 proxy, originally created at Lyft and now a graduated CNCF project, that
has become the de facto standard data plane underneath most modern service mesh and API gateway products —
Istio, AWS App Mesh, and many managed API gateways all run Envoy underneath, rather than building their own
proxy from scratch.** What distinguishes Envoy from HAProxy/nginx isn't raw load-balancing capability (it
implements everything covered so far, plus consistent hashing via ring-hash and Maglev) — it's Envoy's
**xDS API**: a dynamic configuration protocol that lets an external control plane push routing, endpoint,
and policy changes to a fleet of Envoy proxies in near-real-time, without a config reload or restart.

```mermaid
flowchart TD
    ControlPlane["Control plane<br/>(e.g. Istiod)"] -->|"xDS - push config"| E1["Envoy proxy 1"]
    ControlPlane -->|"xDS - push config"| E2["Envoy proxy 2"]
    ControlPlane -->|"xDS - push config"| E3["Envoy proxy 3"]
    E1 --> Backend1["Backend"]
    E2 --> Backend2["Backend"]
    E3 --> Backend3["Backend"]

    classDef accent fill:#f0e9fb,stroke:#6d43c0,color:#10161c
    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    class ControlPlane accent
    class E1,E2,E3,Backend1,Backend2,Backend3 info
```

*Every Envoy instance is a dumb executor of whatever config the control plane pushes it — the intelligence
lives centrally, the enforcement runs at the edge of every workload. Chapter 5 of this series covers this
same architecture in full as the foundation of a service mesh data plane.*

This dynamic-configuration model is what makes Envoy the natural fit for environments where the backend
fleet changes constantly (Kubernetes pods scaling up and down, canary deployments shifting traffic
percentages by the minute) — a static config file reload, the traditional HAProxy/nginx model, becomes a
real operational bottleneck at that pace of change, while xDS pushes are designed for exactly this
continuous-update pattern.

### Load Balancer or API Gateway? Where the Line Actually Blurs

An L7 load balancer (ALB, Envoy, nginx) and a dedicated **API gateway** product (Kong, Apigee, AWS API
Gateway) overlap heavily in marketing material and genuinely confuse the distinction in practice — both
route HTTP requests to backends, both can do rate limiting, both can inspect headers. The meaningful
difference is one of **scope and ownership model**, not raw technical capability:

| | **L7 Load Balancer** | **Dedicated API Gateway** |
|---|---|---|
| Primary job | Distribute traffic across a healthy backend fleet | Manage the *developer-facing contract* of an API — auth, quotas, versioning, developer portal |
| Authentication | Usually delegates entirely to the backend or a separate auth layer | Frequently owns API-key/OAuth validation directly, as a first-class feature |
| Per-consumer policy | Coarse (a route, a header match) | Fine-grained, per-API-key or per-tenant policy is a core feature |
| Request/response transformation | Limited (header rewrite, redirects) | A primary feature — protocol translation, payload reshaping, legacy-API adaptation |
| Typical position | Any HTTP traffic, internal or external | Specifically the boundary where external/partner developers consume your API as a product |

**In practice, many production architectures use both, layered**: an L7 load balancer (or Envoy specifically)
handles the raw traffic-distribution job covered throughout this chapter, sitting in front of, or
alongside, a dedicated API gateway that owns the developer-facing API-product concerns — API keys, usage
quotas per partner, a self-service developer portal. A team that only needs internal service-to-service
routing rarely needs a full API gateway product at all; a team exposing a metered, partner-facing API
surface almost always needs one layered on top of, not instead of, its load balancer.

## Circuit Breaking and Outlier Detection

Two related but distinct Envoy (and, in concept, any modern L7 proxy) mechanisms exist specifically to stop
a struggling backend from making an incident *worse*:

- **Circuit breaking** sets hard, static limits — maximum concurrent connections, maximum pending requests,
  maximum concurrent requests, maximum retries — at the level of an entire backend cluster. Once a limit is
  hit, Envoy immediately rejects further traffic (typically a fast `503`) rather than queuing it
  indefinitely or forwarding it to an already-overwhelmed backend fleet. This protects the *backend* from
  being driven further into overload by a traffic spike or a slow-dependency cascade.
- **Outlier detection** continuously watches *individual* backend hosts' real traffic outcomes (a run of
  consecutive 5xx responses, consecutive connection failures, or a success-rate statistical outlier
  relative to the rest of the cluster) and temporarily ejects a misbehaving individual host from the
  load-balancing pool — a form of passive health checking (see above) with automatic, temporary remediation
  built in, rather than just alerting.

```mermaid
stateDiagram-v2
    [*] --> Healthy
    Healthy --> Ejected: Consecutive 5xx threshold exceeded
    Ejected --> Healthy: Ejection interval expires,<br/>host re-admitted for a trial period
    Healthy --> Healthy: Normal traffic, passes checks

    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c
    class Healthy ok
    class Ejected crit
```

*A repeat offender's ejection interval typically grows on each subsequent ejection — Envoy's default
behavior doubles it, up to a configured maximum — so a genuinely chronically failing host gets
progressively longer time-outs rather than flapping in and out of rotation every few seconds.*

Together, circuit breaking and outlier detection turn the load balancer into an **active participant in
incident containment** rather than a passive traffic router: circuit breaking caps the blast radius at the
cluster level before things get worse, and outlier detection surgically removes the specific misbehaving
host(s) causing the problem — both acting automatically, faster than any human on-call engineer could
manually intervene, and without requiring a single line of change in the application itself.

## Global Server Load Balancing — Spreading Load Across Regions

Everything so far in this chapter operates *within* one region or one cluster. **Global Server Load
Balancing (GSLB)** is the practice of load-balancing across geographically distributed regions or
datacenters — the layer that sits directly on top of, and depends on, both this chapter's mechanisms and
Part 1's anycast/BGP concepts. GSLB is implemented one of two ways in production:

1. **Anycast-based** (covered in depth in Part 1): the same IP address is announced from multiple
   locations, and BGP's own path selection decides which location a given client reaches — fast,
   BGP-convergence-speed failover, but coarse-grained (you don't control precisely *how much* traffic each
   region gets, only which one is topologically "closest").
2. **DNS-based**: an authoritative DNS server (often a dedicated GSLB/traffic-manager product — AWS Route
   53, Cloudflare Load Balancing, NS1) returns a different backend IP depending on the querying resolver's
   geography, measured latency, or each region's current reported health — finer-grained control (can do
   percentage-based weighting, active/passive failover with health checks), at the cost of depending on DNS
   resolver caching behavior and TTLs for how fast a change actually reaches end users. Chapter 5 of this
   series covers DNS-based traffic steering and its CDN-adjacent use cases in full depth.

> [!NOTE]
> These two approaches are frequently combined rather than treated as an either/or choice — anycast for the
> coarse, fast, network-level failover, with a DNS-based layer underneath it providing finer application-aware
> control within whichever anycast-selected region a client landed in. Part 1's worked scenario
> (`checkout-service` going multi-region with AWS Global Accelerator) is exactly this: Global Accelerator
> provides the anycast layer, while the ALBs it forwards to inside each region provide this chapter's own
> L7 load-balancing intelligence underneath.

## Rate Limiting and Traffic Shaping at the Load Balancer

**Load balancing decides *which* backend handles a request; rate limiting decides *whether* a request gets
handled at all, right now.** The two are complementary layers of the same traffic-management job, and the
air-traffic-control analogy this site's content generally draws on is a genuinely good fit here: a load
balancer is the ground controller assigning each aircraft to a gate, while a rate limiter is the tower
deciding how many aircraft are allowed to be in the approach pattern simultaneously, independent of which
gate each one is ultimately headed to.

Two algorithms cover the overwhelming majority of real production rate limiting:

| Algorithm | How it works | Behavior under burst |
|---|---|---|
| **Token bucket** | A bucket holds up to N tokens, refilled at a fixed rate; each request consumes one token, and is rejected if the bucket is empty | Allows a burst up to the bucket's full capacity, then throttles to the steady refill rate |
| **Leaky bucket** | Requests enter a queue (the "bucket") and are processed at a fixed, constant output rate regardless of arrival rate | Smooths bursts into a steady stream — no burst allowance, but no burst-driven backend spike either |

**Token bucket is the far more common production choice** specifically because it tolerates legitimate
burstiness (a user rapid-clicking, a batch job's initial ramp) without penalizing it, while still enforcing
a hard steady-state ceiling — leaky bucket's strict output-smoothing is a better fit for protecting a
downstream system that genuinely cannot tolerate *any* burst at all (a legacy backend with a hard concurrent-
connection ceiling), which is a narrower and less common real-world requirement.

**A second, orthogonal dimension is scope: local vs. global rate limiting.** A single Envoy instance
enforcing "max 100 requests/second" only knows about *its own* traffic — with ten Envoy instances behind
the same virtual service, the real aggregate limit is effectively 1,000/second, not 100, unless the limiting
decision is centralized. Envoy's **global rate limit service (RLS)** pattern solves this: every proxy
instance calls out to a shared, centralized rate-limiting service (commonly backed by Redis) for the actual
token-bucket decision, so the limit is enforced correctly in aggregate across the whole fleet rather than
per-instance.

```mermaid
flowchart TD
    E1["Envoy 1"] -->|"check/decrement quota"| RLS["Shared Rate Limit Service<br/>(Redis-backed token bucket)"]
    E2["Envoy 2"] -->|"check/decrement quota"| RLS
    E3["Envoy 3"] -->|"check/decrement quota"| RLS
    RLS -->|"allow / deny"| E1
    RLS -->|"allow / deny"| E2
    RLS -->|"allow / deny"| E3

    classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef accent fill:#f0e9fb,stroke:#6d43c0,color:#10161c
    class E1,E2,E3 info
    class RLS accent
```

*Without a shared limiter, each proxy's local 100/s cap silently becomes an aggregate 100×N/s cap across
an N-instance fleet — a common, easy-to-miss gap between an intended and an actual limit.*

> [!TIP]
> **Best practice: rate-limit at multiple granularities simultaneously, not just one global ceiling.** A
> single global cap protects the backend fleet overall but does nothing to stop one abusive or buggy client
> from consuming the *entire* budget and starving every other client. Production rate-limiting configs
> typically layer a per-API-key or per-client-IP limit underneath a coarser global limit — `catalog-service`'s
> public search API, for example, enforces both "no single API key over 50 req/s" and "no more than 5,000
> req/s in aggregate," so one misbehaving integration partner can't exhaust capacity meant for everyone else.

A third algorithm worth knowing, common in Redis-backed shared limiters specifically: the **sliding window
counter** — a hybrid that keeps two adjacent fixed windows (the current and previous minute, say) and
computes a weighted count across both, proportional to how far into the current window the request arrived.
This avoids fixed-window counting's own well-known edge case, where a client can burst up to the limit at
the very end of one window and again at the very start of the next, briefly achieving nearly double the
intended rate across the boundary — a real gap a pure fixed-window implementation has and a sliding window
closes, at the cost of slightly more bookkeeping than either token bucket or a plain fixed window. This is
the approach several major CDN and API-gateway providers document publicly as their default rate-limiting
implementation specifically because it gives token-bucket-like burst tolerance within a window while still
bounding the worst-case rate across any window boundary — a genuinely better default than either pure
algorithm alone for a public-facing API surface with unpredictable, bursty client behavior.

## Retries, Timeouts, and the Retry Storm Problem

**A naive retry policy — "if a request fails, immediately retry it" — is one of the most common ways a
minor, localized backend hiccup turns into a full cascading outage**, and it's worth understanding exactly
why before configuring retries on any production load balancer or service mesh.

The failure mechanism: a backend cluster becomes momentarily slow (a GC pause, a brief spike in database
latency) and starts timing out a small fraction of requests. Every client (or every upstream proxy) with a
naive "retry once immediately on failure" policy now sends a **second** request for each timed-out one —
which lands on the *same already-struggling* backend fleet, adding load precisely when it has the least
spare capacity to absorb it. If the retry itself times out and is retried again, the effective request rate
against a fleet that's already degraded can multiply several times over from its original level, turning a
transient blip into the thing that actually pushes the fleet into full overload.

```mermaid
sequenceDiagram
    participant Client
    participant LB as Load Balancer
    participant Backend as Already-slow Backend

    Client->>LB: Request 1
    LB->>Backend: Forward
    Note over Backend: Slow - times out
    LB->>Backend: Naive retry - SAME already-slow backend
    Note over Backend: Also times out - MORE load added
    LB->>Backend: Second naive retry
    Note over Backend: Backend now fully overloaded -<br/>the retries themselves caused the collapse
```

*Each retry adds load to a backend that's already the bottleneck — with no backoff and no budget, retries
amplify exactly the failure they were meant to paper over.*

The standard production mitigations, layered together:

- **Exponential backoff with jitter**: each retry waits progressively longer than the last (doubling is
  common), with a small random jitter added to each wait so a large number of simultaneously-retrying
  clients don't all retry in synchronized waves that themselves look like a traffic spike.
- **Retry budgets**: cap total retry volume as a percentage of *original* request volume (Envoy's default
  is 20%) — once the budget is exhausted, further retries are suppressed entirely rather than allowed to
  keep compounding, directly preventing the multiplicative blow-up shown above.
- **Retry only idempotent operations, or only specific failure classes**: a `POST /checkout/charge` request
  that failed *after* the payment was actually processed but *before* the response reached the client is
  extremely dangerous to blindly retry — it can double-charge the customer. Retry policy should be scoped
  to genuinely safe failure modes (connection refused before any request reached the backend at all) and
  genuinely idempotent operations, not applied uniformly to every request type.

> [!WARNING]
> **From the trenches**: a platform team enabled Envoy's automatic retry-on-5xx for every route by default,
> including `checkout-service`'s payment-charge endpoint, reasoning "retries improve reliability" without
> auditing which specific endpoints were safe to retry. During a brief database failover event, a small
> percentage of charge requests succeeded on the backend but the response was lost in transit before
> reaching the proxy — which Envoy correctly treated as a failure and, per policy, retried. The underlying
> condition that turned this from "briefly elevated latency" into "a real financial-reconciliation incident"
> was that the charge endpoint had no idempotency-key mechanism to let the retried request safely detect
> "this charge already happened" — so a meaningful number of customers were charged twice. The fix had two
> parts: excluding non-idempotent, financially-sensitive endpoints from automatic retry policy entirely, and
> — the more durable fix — adding proper idempotency-key support to the charge endpoint itself, so that even
> a legitimate, necessary retry becomes provably safe rather than merely rare.

## Weighted Traffic Shifting — Canary and Blue-Green at the Load-Balancing Layer

The same mechanism that spreads load across a fleet for redundancy can be repurposed to gradually shift
traffic between two *different versions* of a service — the load-balancing layer's own contribution to a
safe deployment strategy, distinct from (but frequently paired with) the CI/CD pipeline mechanics this
site's Automation, CI/CD & GitOps series covers from the deployment-orchestration side.

- **Weighted target groups**: an ALB, an Envoy route, or an Istio `VirtualService` can split traffic
  between two backend target groups by percentage (95%/5%, then 80%/20%, and so on) — the direct
  load-balancing-layer implementation of a canary rollout, letting a small, real slice of production
  traffic exercise the new version before it receives 100%.
- **Header/cookie-based routing**: rather than a random percentage split, route based on a specific request
  attribute — an internal employee's session cookie, a `X-Canary: true` header set by a specific test
  client — so a canary version can be validated by a *known*, controlled set of requests before any random
  slice of real customer traffic reaches it at all.

```mermaid
flowchart LR
    Traffic["Incoming traffic"] --> Split{"Weighted split"}
    Split -->|"95%"| Stable["checkout-service v1.4<br/>(stable)"]
    Split -->|"5%"| Canary["checkout-service v1.5<br/>(canary)"]

    classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c
    class Stable ok
    class Canary warn
```

*The load balancer, not the deployment pipeline, is what actually enforces the traffic split — the
pipeline's job is triggering the weight change and watching the canary's health signal before advancing it.*

⚙️ **Worked example**: `checkout-service`'s platform team ships a rewrite of the discount-code validation
logic. Rather than a straight blue-green cutover, they configure a weighted ALB target-group split starting
at 99%/1%, paired with the same outlier-detection and health-check signals covered earlier in this chapter
watching the canary target group specifically. Because the canary's error rate is monitored *independently*
from the stable version's aggregate rate, a defect affecting only the new discount logic (a bug present in
only 1% of traffic) produces a clearly visible, statistically significant spike in the canary's own narrow
slice — instead of being diluted into invisibility inside a combined 100%-traffic error-rate graph, which is
exactly the failure mode a naive "just deploy and watch the overall dashboard" approach runs into.

## Full Worked Scenario: Redesigning checkout-service's Load Balancing After a Cascading Failure

`checkout-service` runs on Kubernetes behind an ALB, with a Kubernetes Service (backed by `kube-proxy`,
itself doing L4 load balancing inside the cluster — a mechanism covered from the kernel side in this site's
Linux & Networking Fundamentals series) fronting the actual pods. During a Black Friday traffic spike, one
availability zone's pods began responding slowly due to a noisy-neighbor CPU contention issue on the
underlying nodes — not fully down, just meaningfully slower than the other two AZs' pods.

**What went wrong initially**: the ALB was configured with plain round-robin distribution and only an
active health check hitting a cheap `/healthz` endpoint (the exact anti-pattern covered earlier in this
chapter) — which the degraded pods kept passing, since `/healthz` never touched the slow, contended
resource. Round robin kept sending a full, equal share of traffic to the degraded AZ regardless of its
actual response times, and because ALB (at the time) had no outlier-detection mechanism comparable to
Envoy's, there was no automatic way for the load balancer itself to notice and route around the problem.
The result: roughly a third of all checkout requests during the peak traffic window experienced elevated
latency, some breaching the payment gateway's own timeout and failing outright, while the other two-thirds
of capacity sat comparatively underutilized.

**The redesign**, applying this chapter's concepts directly:

1. Switched the ALB's target group algorithm from round robin to **least outstanding requests** — ALB's
   name for a least-connections-family algorithm — so a degraded, slower AZ naturally receives less new
   traffic simply because its in-flight request count stays elevated relative to the healthy AZs, with zero
   manual intervention needed.
2. Deployed an **Envoy-based internal proxy layer** between the ALB and the Kubernetes Service specifically
   to get outlier detection — something the ALB itself didn't support at the time — so a genuinely
   struggling pod (not just an AZ-level slowdown) gets automatically, temporarily ejected from rotation the
   moment its error/latency profile diverges from its peers.
3. Replaced the shallow `/healthz` active health check with one that reports a real, lightweight
   representative signal (the connection-pool-availability pattern from earlier in this chapter),
   specifically so a resource-contention-driven degradation shows up as an unhealthy signal rather than
   silently passing.
4. Tuned circuit breaking on the Envoy layer to cap maximum concurrent requests per backend pod at a level
   informed by prior load-test data, so a future traffic spike degrades gracefully (fast, explicit 503s for
   the excess) rather than queuing indefinitely and taking down the whole fleet's latency together.

The following Black Friday, the same noisy-neighbor contention issue recurred on a different AZ — the
underlying infrastructure problem was never actually fixed, deliberately, since diagnosing and correcting
node-level noisy-neighbor scheduling was a separate, longer-running platform initiative. This time, least
outstanding requests plus outlier detection redirected traffic away from the degraded pods within seconds
of the first elevated-latency samples, and the incident that had previously been a customer-visible,
multi-hour checkout failure became a fully automatic, unremarkable non-event that only showed up as a minor
blip in the AZ-level latency dashboard — the direct, measurable payoff of treating load balancing as an
active reliability mechanism rather than a passive traffic splitter.

## Common Mistakes and Interview Traps

| Mistake | Why it's wrong | What to say instead |
|---|---|---|
| "Round robin is the default and simplest, so it's always a safe choice" | It assumes every request costs the same amount of backend work, which is false for almost any real API | Least-connections (or power-of-two-choices at scale) adapts to real load; round robin doesn't |
| "A passing health check means the backend is fully working" | A shallow health check can pass while the actual request-serving path is broken | The health check must exercise a representative, cheap signal of the *real* dependency most likely to fail |
| "L7 load balancing is strictly better than L4" | L7 adds real overhead (TLS termination, payload parsing) and can't handle non-HTTP protocols at all | Pick based on whether routing needs to depend on request content — not by default preference |
| "Consistent hashing and session affinity are the same thing" | Consistent hashing is about mapping *keys* to backends efficiently under scaling; session affinity is about keeping one *client's* traffic on one backend | Session affinity can be *implemented* using consistent/IP hashing, but the goals (cache locality vs. per-client stickiness) are distinct |
| "Circuit breaking and outlier detection do the same job" | Circuit breaking is a static, cluster-wide cap; outlier detection is dynamic, per-host ejection based on observed behavior | They're complementary — circuit breaking caps overall damage, outlier detection removes the specific bad actor |

## Worked Practice Problems

**Problem 1**: A team runs a WebSocket-based real-time notification service behind a load balancer using
plain round robin with no session affinity. Users report their connection "resets" every time the service
scales up or a deploy happens, even though individual pods are healthy throughout. What's the most likely
cause, and what's the standard fix if the application genuinely can't be made stateless?

*Answer*: Round robin, with no session affinity, has no concept of "this client already has an established
connection to a specific pod" — a scaling event or deploy changing the backend set, combined with any
reconnect logic on the client side, can easily land a reconnecting client on a *different* pod than before.
If the WebSocket connection carries meaningful in-memory server-side state (not just a stateless pass-through),
losing that pod affinity effectively resets the session even though nothing actually "broke." The standard
fix, if externalizing the state entirely isn't immediately feasible, is enabling session affinity (typically
cookie-based, since WebSocket upgrade requests still start as a normal HTTP request the balancer can inspect)
so a reconnecting client is routed back to the same pod when possible — though the better long-term
architectural answer, per this chapter's session-affinity section, is externalizing that state so any pod
can serve any client.

**Problem 2**: An Envoy-fronted service has circuit breaking configured with a maximum of 100 concurrent
requests per backend cluster, and outlier detection configured to eject a host after 5 consecutive 5xx
responses. During a traffic spike, the cluster starts returning fast 503s well before individual hosts show
signs of being unhealthy. Is this a misconfiguration, or expected behavior — and what's the actual tradeoff
being made?

*Answer*: Expected behavior, not a bug — this is exactly what circuit breaking is designed to do. The 100
concurrent-request cap is a deliberate, static ceiling meant to protect the backend cluster from being
driven into overload in the first place; once traffic exceeds it, Envoy rejects the excess immediately with
a fast 503 rather than queuing requests indefinitely or letting them pile up against already-saturated
backends (which would likely convert a manageable overload into a much worse cascading failure with slow
timeouts instead of fast, cheap rejections). The tradeoff is real: some legitimate requests get rejected
during the spike that a queue-everything approach might have eventually served — but a fast, explicit
failure a client can retry against is generally far preferable operationally to a slow-motion pile-up that
degrades every request, including ones that would otherwise have succeeded quickly.

**Problem 3**: A platform team migrates a Redis-backed cache layer from 8 nodes to 12 nodes, load-balanced
with a naive `hash(key) % node_count` scheme. What's the expected impact, and what change would have made
this a non-event?

*Answer*: Changing the node count from 8 to 12 changes the modulo divisor, which reassigns the vast majority
of keys to a different node than before — for most keys, `hash(key) % 8` and `hash(key) % 12` land on
completely different results, not just for the "new" capacity. The practical impact is a near-total cache
miss storm immediately after the scaling event, with all that traffic falling through to the origin data
store, which was very likely never sized to absorb a near-100%-miss-rate traffic pattern. Migrating to
consistent hashing (a ring-hash or Maglev implementation, as covered earlier in this chapter) before the
scaling event would have bounded the reassignment to only the keys adjacent to the newly added nodes on the
hash ring — a small, absorbable fraction of the total keyspace — turning what was a customer-visible
incident into an unremarkable, gradual cache warm-up.

## Summary and What's Next

Load balancing is where a request's fate is actually decided, one hop past the routing layer Part 1
covered — the algorithm, the health-check design, and the draining/circuit-breaking policy chosen here
determine whether a degraded backend quietly gets routed around or actively makes an incident worse. Envoy's
xDS-driven, dynamically-configurable model is also the direct foundation for the service mesh data plane
this series returns to in Part 5.

**Part 3** moves from the traffic-management layer back down to the infrastructure underneath it — how a
cloud VPC/VNet is actually structured, how subnets, route tables, NAT gateways, and security groups fit
together, and how multiple VPCs (and multiple clouds) get connected without every pair needing a direct
link.