Load Balancing & Traffic Management

Table of Contents#

1. Why Load Balancing Is a Separate Layer From Routing
2. L4 vs. L7 Load Balancing — the Fundamental Tradeoff
3. Load Balancing Algorithms, One at a Time
4. Consistent Hashing — Why Caches and Sharded Stores Need a Different Algorithm
5. Health Checks — Active, Passive, and Why They Lie
6. Connection Draining — Removing a Backend Without Dropping Live Requests
7. Session Affinity — the Tradeoff Against Even Distribution
8. TLS Termination Location — Edge, Re-Encrypt, or Passthrough
9. Cloud-Managed Load Balancers — ALB, NLB, and Their Equivalents
10. Running Your Own — HAProxy and nginx
11. Envoy — the Modern Programmable Data Plane
12. Circuit Breaking and Outlier Detection
13. Global Server Load Balancing — Spreading Load Across Regions
14. Rate Limiting and Traffic Shaping at the Load Balancer
15. Retries, Timeouts, and the Retry Storm Problem
16. Weighted Traffic Shifting — Canary and Blue-Green at the Load-Balancing Layer
17. Full Worked Scenario: Redesigning checkout-service's Load Balancing After a Cascading Failure
18. Common Mistakes and Interview Traps
19. Worked Practice Problems
20. Summary and What's Next

Why Load Balancing Is a Separate Layer From Routing#

Part 1 answered "how does traffic find its way to my network at all" — this chapter answers the very next question: once traffic arrives, how does it get spread correctly across the many backend instances actually running your service. These are genuinely different problems, solved by different mechanisms, and conflating them is a common source of confused architecture diagrams. BGP and routing (Part 1) operate on networks — prefixes, ASes, physical paths — with no concept of "this specific request" or "this specific backend process." Load balancing operates one layer up: given that traffic has already arrived at your front door, which of checkout-service's twelve currently-running pods should actually handle this request, right now, given their current health and load.

Throughout this chapter, the throughline returns to checkout-service, catalog-service, and inventory-service — the same three-service platform from Part 1, now viewed from inside one region: a fleet of backend instances behind a load balancer, not yet the multi-region picture Part 1's anycast example covered.

Routing (Part 1) is a solved problem by the time traffic reaches the load balancer's front interface — this chapter starts from that point.

L4 vs. L7 Load Balancing — the Fundamental Tradeoff#

The single most important design decision in load balancing is which layer of the stack the balancer operates at, because it determines what the balancer is allowed to see and how expensive each decision is to make.

Neither is strictly "better" — they solve different problems, and a production architecture frequently uses both, layered. An L4 balancer is the right tool when the traffic isn't HTTP at all (a raw TCP database protocol, gRPC over HTTP/2 handled below the balancer, a UDP-based protocol), when preserving the original client IP matters more than routing intelligence, or when the sheer packet-forwarding performance of skipping payload inspection matters at extreme scale. An L7 balancer is the right tool the moment routing needs to depend on what's actually being requested — which is most of the time for a modern HTTP API surface, and is a strict prerequisite for path-based routing, canary/weighted traffic splitting by header, or per-route retry policy.

⚙️ Concretely: checkout-service's public API and catalog-service's public API can share a single L7 load balancer that inspects the request path and routes /checkout/* to one target group and /catalog/* to another — something an L4 balancer has no way to do, since it never looks past the IP/port headers.

Load Balancing Algorithms, One at a Time#

Once a request/connection has arrived at a load balancer with multiple healthy backends, the balancer must pick one. Each algorithm optimizes for a different assumption about the workload:

Same three backends, same instant — round robin's blind rotation can hand a request to the busiest backend purely by scheduling coincidence; least-connections cannot.

Consistent Hashing — Why Caches and Sharded Stores Need a Different Algorithm#

A naive hash-based balancer — hash(key) % number_of_backends — has a catastrophic failure mode the moment the backend count changes: adding or removing even one backend reshuffles the modulo result for almost every key, not just the keys that logically belong to the changed backend. For a stateless web request this is a non-issue (any healthy backend can serve it). For a caching layer or a sharded data store — where "which backend has this key's data" is the entire point — a naive rehash on every scaling event means a near-total cache wipe or a full data reshuffle, exactly when the system is least able to absorb the extra load.

Consistent hashing solves this by hashing both the backends and the keys onto the same fixed conceptual ring, and assigning each key to the nearest backend clockwise on that ring. Adding or removing one backend only reassigns the keys that fell between it and its immediate neighbor on the ring — a small, bounded fraction of the total keyspace, not a full reshuffle.

Each key belongs to the next backend clockwise on the ring — a topology change only disturbs the keys adjacent to the change, bounding the blast radius of any scaling event.

Envoy's ring_hash and Google's Maglev algorithm are the two production-grade implementations most platform teams will actually encounter, rather than hand-rolling one. Maglev (originally published by Google, and adopted widely since, including inside Envoy as an alternative to ring hash) trades a small amount of theoretical rebalancing optimality for dramatically faster lookup and construction time, and is generally the recommended default for a large backend fleet over the classic ring-hash approach.

⚠️ From the trenches: inventory-service's platform team ran a Redis-backed cache in front of a frequently-queried product catalog, load-balanced across cache nodes with a plain modulo hash rather than consistent hashing — a decision made early, before cache-node scaling was a regular event. During a routine capacity add (3 nodes → 4), cache hit rate collapsed from 94% to under 10% for roughly fifteen minutes, and inventory-service's own database — never sized for that query volume — briefly became the bottleneck for the entire platform, including checkout-service requests that depend on inventory data. The immediate cause was the modulo hash reassigning nearly every key to a different node the instant the fourth node joined; the underlying condition was that nobody had revisited the caching layer's hashing strategy since it was first stood up, because cache scaling had always previously happened during low-traffic maintenance windows small enough that the hit-rate dip went unnoticed. Migrating to consistent hashing (Maglev, via the caching layer's Envoy sidecar) turned the next scaling event's hit-rate dip into a low-single-digit-percent blip, fully absorbed without any downstream impact.

Health Checks — Active, Passive, and Why They Lie#

A load balancer only routes to backends it believes are healthy — the mechanism that belief is built on is worth understanding in detail, because a wrong belief (routing to a backend that's actually broken, or withholding traffic from one that's actually fine) is one of the most common real production incidents in this whole chapter's territory.

Neither type alone is sufficient in production. An active check hitting a trivially cheap /healthz endpoint can report "healthy" while the backend's actual request-handling path is completely broken — a classic case is a health endpoint hardcoded to return 200 regardless of downstream dependency state, while the real application logic depends on a database connection pool that's exhausted. A passive check alone can't proactively catch a backend that's about to fail before it actually serves bad traffic to real users, and needs a reasonable sample size before it can distinguish "genuinely unhealthy" from "one unlucky request." Production load balancers (ALB, Envoy, HAProxy) run both simultaneously: active checks for baseline liveness, passive/outlier detection (covered later in this chapter) for real-traffic-driven ejection.

⚠️ From the trenches: checkout-service's /healthz endpoint was implemented, correctly by the letter of most guidance, as a lightweight endpoint that didn't touch the database — "keep health checks fast and cheap" is common (and generally correct) advice. During an incident where the connection pool to the payment-processing database was fully exhausted, every real checkout request failed with a pool-timeout error, while /healthz — which never touched that pool — kept returning 200 the entire time. The load balancer, working exactly as designed, kept routing full traffic to backends that were 100% unable to complete a real checkout. The fix wasn't "make health checks slow and heavy" (the original instinct is correct in general) — it was adding one cheap, specific signal: a fast, non-blocking check of the connection pool's current available connection count (an in-memory counter, not a real query), which directly reflects the actual failure mode without adding real database load to every health probe. The lesson generalizes: a health check needs to be cheap and representative of the actual dependency most likely to fail — those two goals aren't in tension nearly as often as teams assume.

Connection Draining — Removing a Backend Without Dropping Live Requests#

Removing a backend from a load balancer's rotation — for a deploy, a scale-down, or a failed health check — has two very different ways to happen: an abrupt removal that kills in-flight connections immediately, or a graceful drain that stops sending new traffic while letting existing connections finish naturally. Every production load balancer supports a configurable deregistration delay (AWS's term; "connection draining" and "graceful shutdown" are the equivalent concepts elsewhere): once a backend is marked for removal, the balancer stops routing new requests to it immediately, but keeps existing connections open and lets them complete, up to a maximum timeout, before forcibly terminating anything still outstanding.

The window between "marked for removal" and "fully terminated" is exactly the deregistration delay — set too short, and it kills legitimate slow requests; set too long, and deploys/scale-downs take correspondingly longer to fully complete.

Session Affinity — the Tradeoff Against Even Distribution#

Session affinity (sticky sessions) deliberately routes every request from the same client to the same backend, trading perfectly even load distribution for the ability to keep state local to one instance. The two common mechanisms:

• Cookie-based affinity: the load balancer sets (or the application sets, and the balancer reads) a cookie identifying which backend a client was routed to, and honors it on every subsequent request.
• IP-hash affinity: the balancer hashes the client's source IP to deterministically pick a backend — no cookie needed, but breaks down when many clients share one visible IP (a corporate NAT, a mobile carrier's CGNAT pool), all landing on the same backend regardless of actual load.

The right call depends entirely on whether the backend actually needs local state. A genuinely stateless service (most well-designed modern APIs, including checkout-service and catalog-service in this series's own architecture, which push session state to a shared cache/database rather than in-process memory) has no need for affinity at all, and enabling it anyway only adds an uneven-distribution risk for no benefit. A backend that legitimately holds in-memory state per client (a WebSocket connection with server-side buffered state, an in-process shopping-cart cache used specifically to avoid a database round trip) genuinely needs it — but the better long-term architectural fix, where practical, is usually to externalize that state (a shared Redis-backed session store) so any backend can serve any request, removing the need for affinity entirely and restoring even load distribution.

TLS Termination Location — Edge, Re-Encrypt, or Passthrough#

Chapter 4 of this series covers the TLS handshake and certificate mechanics themselves in depth — this section covers the load-balancing-layer decision that has to be made regardless of those details: where, physically, does encrypted traffic actually get decrypted, relative to the load balancer? Three architectures are in common production use, and the choice affects security posture, observability, and operational overhead in ways worth understanding before defaulting to whichever one a cloud console makes easiest.

checkout-service's platform team runs re-encryption specifically for its payment-processing traffic path (PCI-DSS scope requires it) while accepting edge-termination for catalog-service's public product-browsing API, where the internal traffic carries no sensitive payload — a deliberate, risk-scoped choice rather than a uniform policy applied everywhere regardless of what's actually being protected.

Cloud-Managed Load Balancers — ALB, NLB, and Their Equivalents#

Every major cloud provider offers managed L4 and L7 load balancers, sparing a platform team from operating the balancer software itself. AWS's naming is the one most engineers encounter first and worth knowing in detail, since the same conceptual split (a "dumb," extremely fast L4 balancer vs. a smarter, HTTP-aware L7 one) recurs under different names on every cloud:

GCP's equivalent split is the External HTTP(S) Load Balancer (L7, itself built on a globally-distributed anycast frontend — a direct, managed application of Part 1's anycast mechanism, and notably a single global resource rather than a per-region one, unlike AWS's ALB) versus the External TCP/UDP Network Load Balancer (L4, regional, backed by Maglev — the same consistent-hashing algorithm covered earlier in this chapter, originally built at Google specifically for this product before Envoy adopted it too). Azure's split is the Application Gateway (L7, with an optional integrated Web Application Firewall tier) versus the Azure Load Balancer (L4, itself available in both a regional and a cross-region/"global" tier since Microsoft added global-tier support). The conceptual split — and the decision criteria in the table above — carries across all three clouds even though the product names and exact regional-vs-global defaults differ; Chapter 3 of this series covers where each of these sits inside the surrounding VPC/VNet architecture in more depth.

⚙️ A subtlety worth knowing before it surprises you: an NLB placed in front of an ALB (a real, supported pattern — used to get NLB's static IP for an allowlist while keeping ALB's L7 routing behind it) means the ALB's health checks now see the NLB as the "client," not the real end user, and the NLB's own health checks against the ALB need to be configured on the correct port and protocol independently — a misconfigured health check at this specific junction is a recurring, hard-to-spot cause of an otherwise perfectly healthy ALB silently receiving zero traffic.

Running Your Own — HAProxy and nginx#

Cloud-managed load balancers cover the overwhelming majority of production needs, but a platform team still frequently runs a self-managed L7 proxy layer — inside a Kubernetes cluster as an Ingress controller, in front of a set of VMs, or specifically for capabilities a managed LB doesn't expose. HAProxy and nginx are the two long-standing, battle-tested choices, and both remain in wide production use even as Envoy (next section) has captured much of the newer service-mesh-adjacent use case.

• HAProxy is purpose-built as a load balancer first — its configuration model is organized directly around frontends, backends, and the load-balancing algorithm applied between them (balance leastconn, balance roundrobin, and several more, directly matching this chapter's algorithm section). It's frequently the choice when load balancing itself, plus fine-grained L4/L7 traffic control, is the primary job.
• nginx started as a web server and reverse proxy and grew load-balancing capability on top of that foundation — a natural fit when a team already needs nginx's other capabilities (static file serving, TLS termination, request rewriting) and wants load balancing as one more configured feature rather than a separate dedicated component.

# A representative HAProxy backend definition — direct application of this
# chapter's algorithm and health-check concepts as real configuration
backend checkout_service
    balance leastconn
    option httpchk GET /healthz
    http-check expect status 200
    server pod1 10.0.1.10:8080 check inter 5s fall 3 rise 2
    server pod2 10.0.1.11:8080 check inter 5s fall 3 rise 2
    server pod3 10.0.1.12:8080 check inter 5s fall 3 rise 2

fall 3 / rise 2 directly implements the "don't flap on a single unlucky probe" lesson from the health checks section — three consecutive failures before a backend is marked down, two consecutive successes before it's trusted again.

Envoy — the Modern Programmable Data Plane#

Envoy is a purpose-built L4/L7 proxy, originally created at Lyft and now a graduated CNCF project, that has become the de facto standard data plane underneath most modern service mesh and API gateway products — Istio, AWS App Mesh, and many managed API gateways all run Envoy underneath, rather than building their own proxy from scratch. What distinguishes Envoy from HAProxy/nginx isn't raw load-balancing capability (it implements everything covered so far, plus consistent hashing via ring-hash and Maglev) — it's Envoy's xDS API: a dynamic configuration protocol that lets an external control plane push routing, endpoint, and policy changes to a fleet of Envoy proxies in near-real-time, without a config reload or restart.

Every Envoy instance is a dumb executor of whatever config the control plane pushes it — the intelligence lives centrally, the enforcement runs at the edge of every workload. Chapter 5 of this series covers this same architecture in full as the foundation of a service mesh data plane.

This dynamic-configuration model is what makes Envoy the natural fit for environments where the backend fleet changes constantly (Kubernetes pods scaling up and down, canary deployments shifting traffic percentages by the minute) — a static config file reload, the traditional HAProxy/nginx model, becomes a real operational bottleneck at that pace of change, while xDS pushes are designed for exactly this continuous-update pattern.

Load Balancer or API Gateway? Where the Line Actually Blurs#

An L7 load balancer (ALB, Envoy, nginx) and a dedicated API gateway product (Kong, Apigee, AWS API Gateway) overlap heavily in marketing material and genuinely confuse the distinction in practice — both route HTTP requests to backends, both can do rate limiting, both can inspect headers. The meaningful difference is one of scope and ownership model, not raw technical capability:

In practice, many production architectures use both, layered: an L7 load balancer (or Envoy specifically) handles the raw traffic-distribution job covered throughout this chapter, sitting in front of, or alongside, a dedicated API gateway that owns the developer-facing API-product concerns — API keys, usage quotas per partner, a self-service developer portal. A team that only needs internal service-to-service routing rarely needs a full API gateway product at all; a team exposing a metered, partner-facing API surface almost always needs one layered on top of, not instead of, its load balancer.

Circuit Breaking and Outlier Detection#

Two related but distinct Envoy (and, in concept, any modern L7 proxy) mechanisms exist specifically to stop a struggling backend from making an incident worse:

• Circuit breaking sets hard, static limits — maximum concurrent connections, maximum pending requests, maximum concurrent requests, maximum retries — at the level of an entire backend cluster. Once a limit is hit, Envoy immediately rejects further traffic (typically a fast 503) rather than queuing it indefinitely or forwarding it to an already-overwhelmed backend fleet. This protects the backend from being driven further into overload by a traffic spike or a slow-dependency cascade.
• Outlier detection continuously watches individual backend hosts' real traffic outcomes (a run of consecutive 5xx responses, consecutive connection failures, or a success-rate statistical outlier relative to the rest of the cluster) and temporarily ejects a misbehaving individual host from the load-balancing pool — a form of passive health checking (see above) with automatic, temporary remediation built in, rather than just alerting.

A repeat offender's ejection interval typically grows on each subsequent ejection — Envoy's default behavior doubles it, up to a configured maximum — so a genuinely chronically failing host gets progressively longer time-outs rather than flapping in and out of rotation every few seconds.

Together, circuit breaking and outlier detection turn the load balancer into an active participant in incident containment rather than a passive traffic router: circuit breaking caps the blast radius at the cluster level before things get worse, and outlier detection surgically removes the specific misbehaving host(s) causing the problem — both acting automatically, faster than any human on-call engineer could manually intervene, and without requiring a single line of change in the application itself.

Global Server Load Balancing — Spreading Load Across Regions#

Everything so far in this chapter operates within one region or one cluster. Global Server Load Balancing (GSLB) is the practice of load-balancing across geographically distributed regions or datacenters — the layer that sits directly on top of, and depends on, both this chapter's mechanisms and Part 1's anycast/BGP concepts. GSLB is implemented one of two ways in production:

1. Anycast-based (covered in depth in Part 1): the same IP address is announced from multiple locations, and BGP's own path selection decides which location a given client reaches — fast, BGP-convergence-speed failover, but coarse-grained (you don't control precisely how much traffic each region gets, only which one is topologically "closest").
2. DNS-based: an authoritative DNS server (often a dedicated GSLB/traffic-manager product — AWS Route 53, Cloudflare Load Balancing, NS1) returns a different backend IP depending on the querying resolver's geography, measured latency, or each region's current reported health — finer-grained control (can do percentage-based weighting, active/passive failover with health checks), at the cost of depending on DNS resolver caching behavior and TTLs for how fast a change actually reaches end users. Chapter 5 of this series covers DNS-based traffic steering and its CDN-adjacent use cases in full depth.

Rate Limiting and Traffic Shaping at the Load Balancer#

Load balancing decides which backend handles a request; rate limiting decides whether a request gets handled at all, right now. The two are complementary layers of the same traffic-management job, and the air-traffic-control analogy this site's content generally draws on is a genuinely good fit here: a load balancer is the ground controller assigning each aircraft to a gate, while a rate limiter is the tower deciding how many aircraft are allowed to be in the approach pattern simultaneously, independent of which gate each one is ultimately headed to.

Two algorithms cover the overwhelming majority of real production rate limiting:

Token bucket is the far more common production choice specifically because it tolerates legitimate burstiness (a user rapid-clicking, a batch job's initial ramp) without penalizing it, while still enforcing a hard steady-state ceiling — leaky bucket's strict output-smoothing is a better fit for protecting a downstream system that genuinely cannot tolerate any burst at all (a legacy backend with a hard concurrent- connection ceiling), which is a narrower and less common real-world requirement.

A second, orthogonal dimension is scope: local vs. global rate limiting. A single Envoy instance enforcing "max 100 requests/second" only knows about its own traffic — with ten Envoy instances behind the same virtual service, the real aggregate limit is effectively 1,000/second, not 100, unless the limiting decision is centralized. Envoy's global rate limit service (RLS) pattern solves this: every proxy instance calls out to a shared, centralized rate-limiting service (commonly backed by Redis) for the actual token-bucket decision, so the limit is enforced correctly in aggregate across the whole fleet rather than per-instance.

Without a shared limiter, each proxy's local 100/s cap silently becomes an aggregate 100×N/s cap across an N-instance fleet — a common, easy-to-miss gap between an intended and an actual limit.

A third algorithm worth knowing, common in Redis-backed shared limiters specifically: the sliding window counter — a hybrid that keeps two adjacent fixed windows (the current and previous minute, say) and computes a weighted count across both, proportional to how far into the current window the request arrived. This avoids fixed-window counting's own well-known edge case, where a client can burst up to the limit at the very end of one window and again at the very start of the next, briefly achieving nearly double the intended rate across the boundary — a real gap a pure fixed-window implementation has and a sliding window closes, at the cost of slightly more bookkeeping than either token bucket or a plain fixed window. This is the approach several major CDN and API-gateway providers document publicly as their default rate-limiting implementation specifically because it gives token-bucket-like burst tolerance within a window while still bounding the worst-case rate across any window boundary — a genuinely better default than either pure algorithm alone for a public-facing API surface with unpredictable, bursty client behavior.

Retries, Timeouts, and the Retry Storm Problem#

A naive retry policy — "if a request fails, immediately retry it" — is one of the most common ways a minor, localized backend hiccup turns into a full cascading outage, and it's worth understanding exactly why before configuring retries on any production load balancer or service mesh.

The failure mechanism: a backend cluster becomes momentarily slow (a GC pause, a brief spike in database latency) and starts timing out a small fraction of requests. Every client (or every upstream proxy) with a naive "retry once immediately on failure" policy now sends a second request for each timed-out one — which lands on the same already-struggling backend fleet, adding load precisely when it has the least spare capacity to absorb it. If the retry itself times out and is retried again, the effective request rate against a fleet that's already degraded can multiply several times over from its original level, turning a transient blip into the thing that actually pushes the fleet into full overload.

Each retry adds load to a backend that's already the bottleneck — with no backoff and no budget, retries amplify exactly the failure they were meant to paper over.

The standard production mitigations, layered together:

• Exponential backoff with jitter: each retry waits progressively longer than the last (doubling is common), with a small random jitter added to each wait so a large number of simultaneously-retrying clients don't all retry in synchronized waves that themselves look like a traffic spike.
• Retry budgets: cap total retry volume as a percentage of original request volume (Envoy's default is 20%) — once the budget is exhausted, further retries are suppressed entirely rather than allowed to keep compounding, directly preventing the multiplicative blow-up shown above.
• Retry only idempotent operations, or only specific failure classes: a POST /checkout/charge request that failed after the payment was actually processed but before the response reached the client is extremely dangerous to blindly retry — it can double-charge the customer. Retry policy should be scoped to genuinely safe failure modes (connection refused before any request reached the backend at all) and genuinely idempotent operations, not applied uniformly to every request type.

Weighted Traffic Shifting — Canary and Blue-Green at the Load-Balancing Layer#

The same mechanism that spreads load across a fleet for redundancy can be repurposed to gradually shift traffic between two different versions of a service — the load-balancing layer's own contribution to a safe deployment strategy, distinct from (but frequently paired with) the CI/CD pipeline mechanics this site's Automation, CI/CD & GitOps series covers from the deployment-orchestration side.

• Weighted target groups: an ALB, an Envoy route, or an Istio VirtualService can split traffic between two backend target groups by percentage (95%/5%, then 80%/20%, and so on) — the direct load-balancing-layer implementation of a canary rollout, letting a small, real slice of production traffic exercise the new version before it receives 100%.
• Header/cookie-based routing: rather than a random percentage split, route based on a specific request attribute — an internal employee's session cookie, a X-Canary: true header set by a specific test client — so a canary version can be validated by a known, controlled set of requests before any random slice of real customer traffic reaches it at all.

The load balancer, not the deployment pipeline, is what actually enforces the traffic split — the pipeline's job is triggering the weight change and watching the canary's health signal before advancing it.

⚙️ Worked example: checkout-service's platform team ships a rewrite of the discount-code validation logic. Rather than a straight blue-green cutover, they configure a weighted ALB target-group split starting at 99%/1%, paired with the same outlier-detection and health-check signals covered earlier in this chapter watching the canary target group specifically. Because the canary's error rate is monitored independently from the stable version's aggregate rate, a defect affecting only the new discount logic (a bug present in only 1% of traffic) produces a clearly visible, statistically significant spike in the canary's own narrow slice — instead of being diluted into invisibility inside a combined 100%-traffic error-rate graph, which is exactly the failure mode a naive "just deploy and watch the overall dashboard" approach runs into.

Full Worked Scenario: Redesigning checkout-service's Load Balancing After a Cascading Failure#

checkout-service runs on Kubernetes behind an ALB, with a Kubernetes Service (backed by kube-proxy, itself doing L4 load balancing inside the cluster — a mechanism covered from the kernel side in this site's Linux & Networking Fundamentals series) fronting the actual pods. During a Black Friday traffic spike, one availability zone's pods began responding slowly due to a noisy-neighbor CPU contention issue on the underlying nodes — not fully down, just meaningfully slower than the other two AZs' pods.

What went wrong initially: the ALB was configured with plain round-robin distribution and only an active health check hitting a cheap /healthz endpoint (the exact anti-pattern covered earlier in this chapter) — which the degraded pods kept passing, since /healthz never touched the slow, contended resource. Round robin kept sending a full, equal share of traffic to the degraded AZ regardless of its actual response times, and because ALB (at the time) had no outlier-detection mechanism comparable to Envoy's, there was no automatic way for the load balancer itself to notice and route around the problem. The result: roughly a third of all checkout requests during the peak traffic window experienced elevated latency, some breaching the payment gateway's own timeout and failing outright, while the other two-thirds of capacity sat comparatively underutilized.

The redesign, applying this chapter's concepts directly:

1. Switched the ALB's target group algorithm from round robin to least outstanding requests — ALB's name for a least-connections-family algorithm — so a degraded, slower AZ naturally receives less new traffic simply because its in-flight request count stays elevated relative to the healthy AZs, with zero manual intervention needed.
2. Deployed an Envoy-based internal proxy layer between the ALB and the Kubernetes Service specifically to get outlier detection — something the ALB itself didn't support at the time — so a genuinely struggling pod (not just an AZ-level slowdown) gets automatically, temporarily ejected from rotation the moment its error/latency profile diverges from its peers.
3. Replaced the shallow /healthz active health check with one that reports a real, lightweight representative signal (the connection-pool-availability pattern from earlier in this chapter), specifically so a resource-contention-driven degradation shows up as an unhealthy signal rather than silently passing.
4. Tuned circuit breaking on the Envoy layer to cap maximum concurrent requests per backend pod at a level informed by prior load-test data, so a future traffic spike degrades gracefully (fast, explicit 503s for the excess) rather than queuing indefinitely and taking down the whole fleet's latency together.

The following Black Friday, the same noisy-neighbor contention issue recurred on a different AZ — the underlying infrastructure problem was never actually fixed, deliberately, since diagnosing and correcting node-level noisy-neighbor scheduling was a separate, longer-running platform initiative. This time, least outstanding requests plus outlier detection redirected traffic away from the degraded pods within seconds of the first elevated-latency samples, and the incident that had previously been a customer-visible, multi-hour checkout failure became a fully automatic, unremarkable non-event that only showed up as a minor blip in the AZ-level latency dashboard — the direct, measurable payoff of treating load balancing as an active reliability mechanism rather than a passive traffic splitter.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A team runs a WebSocket-based real-time notification service behind a load balancer using plain round robin with no session affinity. Users report their connection "resets" every time the service scales up or a deploy happens, even though individual pods are healthy throughout. What's the most likely cause, and what's the standard fix if the application genuinely can't be made stateless?

Answer: Round robin, with no session affinity, has no concept of "this client already has an established connection to a specific pod" — a scaling event or deploy changing the backend set, combined with any reconnect logic on the client side, can easily land a reconnecting client on a different pod than before. If the WebSocket connection carries meaningful in-memory server-side state (not just a stateless pass-through), losing that pod affinity effectively resets the session even though nothing actually "broke." The standard fix, if externalizing the state entirely isn't immediately feasible, is enabling session affinity (typically cookie-based, since WebSocket upgrade requests still start as a normal HTTP request the balancer can inspect) so a reconnecting client is routed back to the same pod when possible — though the better long-term architectural answer, per this chapter's session-affinity section, is externalizing that state so any pod can serve any client.

Problem 2: An Envoy-fronted service has circuit breaking configured with a maximum of 100 concurrent requests per backend cluster, and outlier detection configured to eject a host after 5 consecutive 5xx responses. During a traffic spike, the cluster starts returning fast 503s well before individual hosts show signs of being unhealthy. Is this a misconfiguration, or expected behavior — and what's the actual tradeoff being made?

Answer: Expected behavior, not a bug — this is exactly what circuit breaking is designed to do. The 100 concurrent-request cap is a deliberate, static ceiling meant to protect the backend cluster from being driven into overload in the first place; once traffic exceeds it, Envoy rejects the excess immediately with a fast 503 rather than queuing requests indefinitely or letting them pile up against already-saturated backends (which would likely convert a manageable overload into a much worse cascading failure with slow timeouts instead of fast, cheap rejections). The tradeoff is real: some legitimate requests get rejected during the spike that a queue-everything approach might have eventually served — but a fast, explicit failure a client can retry against is generally far preferable operationally to a slow-motion pile-up that degrades every request, including ones that would otherwise have succeeded quickly.

Problem 3: A platform team migrates a Redis-backed cache layer from 8 nodes to 12 nodes, load-balanced with a naive hash(key) % node_count scheme. What's the expected impact, and what change would have made this a non-event?

Answer: Changing the node count from 8 to 12 changes the modulo divisor, which reassigns the vast majority of keys to a different node than before — for most keys, hash(key) % 8 and hash(key) % 12 land on completely different results, not just for the "new" capacity. The practical impact is a near-total cache miss storm immediately after the scaling event, with all that traffic falling through to the origin data store, which was very likely never sized to absorb a near-100%-miss-rate traffic pattern. Migrating to consistent hashing (a ring-hash or Maglev implementation, as covered earlier in this chapter) before the scaling event would have bounded the reassignment to only the keys adjacent to the newly added nodes on the hash ring — a small, absorbable fraction of the total keyspace — turning what was a customer-visible incident into an unremarkable, gradual cache warm-up.

Summary and What's Next#

Load balancing is where a request's fate is actually decided, one hop past the routing layer Part 1 covered — the algorithm, the health-check design, and the draining/circuit-breaking policy chosen here determine whether a degraded backend quietly gets routed around or actively makes an incident worse. Envoy's xDS-driven, dynamically-configurable model is also the direct foundation for the service mesh data plane this series returns to in Part 5.

Part 3 moves from the traffic-management layer back down to the infrastructure underneath it — how a cloud VPC/VNet is actually structured, how subnets, route tables, NAT gateways, and security groups fit together, and how multiple VPCs (and multiple clouds) get connected without every pair needing a direct link.