High Availability & Load Balancing

Table of Contents#

1. Why This Topic Matters
2. What "High Availability" Actually Means
3. The Core Idea: Redundancy
4. Active-Passive vs Active-Active
5. Failover — How the Switch Actually Happens
6. Single Points of Failure (SPOFs)
7. Redundancy at Every Layer
8. Availability Zones vs Regions
9. What Is a Load Balancer, Really?
10. Layer 4 vs Layer 7 Load Balancing
11. Load Balancing Algorithms
12. Health Checks — How the Load Balancer Knows Who's Alive
13. Consistent Hashing — Solving a Real Problem
14. DNS-Based Load Balancing and Global Traffic
15. Putting It Together: A Full Request's Journey
16. Common Mistakes
17. Worked Practice Problems
18. Summary and What's Next

---

Why This Topic Matters#

Think about the last time a website "just worked" even though, somewhere behind the scenes, a server crashed, a network cable got unplugged, or an entire data center lost power. That's not luck — it's design. This tutorial is about the handful of ideas that make that possible: having backups ready to go, spreading traffic across many machines instead of one, and knowing what to do the instant something breaks.

If SLIs/SLOs (from the SRE Fundamentals series) tell you how reliable you need to be, this tutorial is about the actual building blocks that get you there.

Diagram

---

What "High Availability" Actually Means#

High Availability (HA) simply means: the system keeps working even when a piece of it breaks. Not "nothing ever breaks" — things break all the time, in any system big enough to matter. HA is about making sure a single broken piece doesn't take the whole thing down.

A simple analogy: think of a restaurant with one chef. If that chef gets sick, the restaurant closes for the day. Now imagine the same restaurant with three chefs on rotation — if one gets sick, the other two cover, and customers never even notice. That's the entire idea behind HA: don't depend on just one of anything that matters.

Diagram

---

The Core Idea: Redundancy#

Redundancy just means having more than one of something important, so that if one fails, another can take over. It sounds almost too simple to be a whole engineering discipline — but almost every HA pattern is some specific flavor of redundancy applied to a specific part of the system.

The tricky part isn't "have a backup" — it's how do you know when to use the backup, and how do you switch to it without anyone noticing? That's what the rest of this tutorial is about.

---

Active-Passive vs Active-Active#

There are two basic ways to arrange your redundant copies.

Active-Passive (a.k.a. Active-Standby)#

One copy does all the work (the "active" one). The other copy sits there, ready, doing nothing — until the active one fails, at which point the passive copy takes over.

Diagram

Analogy: a spare tire in your car trunk. It does nothing 99.9% of the time, but the moment you get a flat, you swap it in.

• Pro: simple to reason about — only one copy is ever actually serving traffic, so there's no risk of the two copies disagreeing with each other.
• Con: you're paying for a whole second server that sits idle almost all the time — wasted capacity.

Active-Active#

Both (or all) copies are handling real traffic, all the time, simultaneously. If one fails, the others just absorb its share of the load.

Diagram

Analogy: three chefs cooking at the same time (back to our restaurant), instead of two chefs sitting at home waiting for the third to call in sick.

• Pro: no wasted capacity — every server is doing useful work, all the time.
• Con: harder to build — if the servers share data (like a database), you now have to worry about keeping that data consistent across multiple active copies at once (this connects directly to the CAP Theorem tutorial, Part 3).

Quick Comparison#

Interview tip: a very common follow-up question is "which would you use for a stateless API vs. a database?" Answer: active-active for stateless services (easy — any server can handle any request), and active-passive is much more common for databases specifically because keeping multiple databases simultaneously writable and consistent is genuinely hard (again, this is CAP Theorem territory).

---

Failover — How the Switch Actually Happens#

Failover is the actual mechanical process of detecting a failure and redirecting traffic away from the broken piece.

Diagram

Three Things That Determine How Painful a Failover Is#

1. Detection time — how long until the system notices something's wrong? (Usually a few failed health checks in a row, to avoid overreacting to one blip.)
2. Decision time — how confident do we need to be before flipping the switch? (Flip too eagerly and you get "flapping" — switching back and forth on transient issues. Flip too cautiously and users suffer longer.)
3. Switch-over time — once decided, how long until traffic is actually flowing to the new active copy? (DNS changes can take minutes to propagate; a load balancer health check removing a bad target can take seconds.)

Diagram

A concrete, often-tested number: DNS-based failover can take minutes because of DNS caching (clients and resolvers cache the old IP address based on its TTL). This is exactly why load-balancer-based failover (where the LB itself detects the failure and simply stops routing to the bad server) is usually much faster than DNS-based failover, and why DNS failover is typically reserved for larger-scale events (like an entire region going down) rather than a single failed server.

---

Single Points of Failure (SPOFs)#

A Single Point of Failure is any one component that, if it breaks, takes the whole system down — because nothing is backing it up.

Diagram

The key insight interviewers look for: redundancy at one layer doesn't help if there's a SPOF at another layer. Having 10 redundant app servers is worthless if they all point at a single, non-redundant database, or if they're all reachable only through a single, non-redundant load balancer. You have to hunt for SPOFs at every layer of the stack, not just the layer you happened to think about first.

A Practical SPOF-Hunting Checklist#

Diagram

Run this question against: load balancers, DNS providers, databases, message queues, caches, the network path itself, the power supply, even the team (the "bus factor" — what if the one person who understands this system is unavailable?).

---

Redundancy at Every Layer#

A realistic, full-stack view of where redundancy typically needs to exist:

Diagram

Interview-ready summary line: "High availability isn't one technique — it's the discipline of asking 'what happens if this specific piece dies' at every single layer, from DNS all the way down to the database, and making sure the answer is never 'everything stops.'"

---

Availability Zones vs Regions#

A very common piece of cloud vocabulary, worth being crisp about.

Diagram

• Availability Zone (AZ): one or more physically distinct data centers within the same broad geographic area, with independent power, cooling, and networking — but low-latency, high-bandwidth connections between AZs in the same region.
• Region: a geographically distinct area (e.g., "US East" vs "Europe West") containing multiple AZs.

The tradeoff to always name explicitly: multi-AZ is relatively cheap (low latency between AZs, often included in standard architecture) and protects against the vast majority of real failures. Multi-region is much more expensive and architecturally complex (cross-region data replication has real latency and consistency costs — see the CAP Theorem tutorial), and is usually reserved for services with extremely strict availability requirements or specific regulatory/data-residency needs.

---

What Is a Load Balancer, Really?#

At its core, a load balancer does one job: it receives traffic and decides which backend server should handle each request.

Diagram

Why not just let clients talk directly to servers? Two big reasons:

1. Spreading load evenly so no single server gets overwhelmed while others sit idle.
2. Hiding failures — if Server 2 dies, the load balancer simply stops sending it traffic; clients never need to know Server 2 ever existed.

---

Layer 4 vs Layer 7 Load Balancing#

This is one of the most commonly tested "networking basics" questions in SRE interviews.

Diagram

Analogy: Layer 4 is like a mail sorter who only looks at the zip code on an envelope and sends it to the right regional office — fast, but doesn't read what's inside. Layer 7 is like a receptionist who actually reads your request ("I'm here for a dentist appointment") and sends you to exactly the right room — slower per person, but much smarter routing.

A great interview answer to "when would you use L4 vs L7": "L4 when I need raw speed and don't need content-aware routing — e.g., a generic TCP service. L7 when I need to route based on the actual request — e.g., sending /api/* to one backend and /static/* to a CDN-backed bucket, or doing SSL termination centrally instead of on every backend server."

---

Load Balancing Algorithms#

Once the load balancer decides that a request needs to go to some backend, it needs a rule for which one.

Diagram

---

Health Checks — How the Load Balancer Knows Who's Alive#

A load balancer is only as good as its ability to tell healthy servers from broken ones — this is what makes the "automatically stop sending traffic to a dead server" behavior work.

Diagram

Two Kinds of Health Checks#

A subtle but important interview point: a shallow health check can lie — the web server process might be "up" and responding to /health, but if its database connection is broken, it can't actually serve real traffic. A good health check should verify the dependencies that actually matter for serving real requests, not just "is the process alive" — but it also shouldn't check too much (e.g., checking a slow, rarely-used downstream dependency could cause a healthy server to be wrongly marked as down). This is a genuine design tradeoff worth naming explicitly if asked.

---

Consistent Hashing — Solving a Real Problem#

This deserves its own spotlight because it's a frequent, specific interview question: "why not just use plain hashing (hash(key) % N) to decide which server/shard handles a given key?"

The Problem With Plain Hashing#

Diagram

Concrete example: with 4 servers, key_id % 4 decides the owner. Add a 5th server, and now it's key_id % 5 — almost every single key gets remapped to a different server than before, even though only one server was added. For a cache, this means nearly every cached item suddenly looks like a "miss" on its new server, hammering the origin database all at once.

The Consistent Hashing Fix#

Diagram

Diagram

Why this matters in plain terms: with consistent hashing, adding a 4th server only reshuffles the small handful of keys that happen to fall right next to it on the ring — everything else stays exactly where it was. This is why consistent hashing is the standard technique behind distributed caches (Memcached client libraries), CDNs, and sharded databases — it lets you scale up or down without a massive, disruptive reshuffle every time.

---

DNS-Based Load Balancing and Global Traffic#

For routing traffic across entire regions (not just servers within one data center), DNS itself often does the load balancing.

Diagram

• GeoDNS: returns a different IP address depending on where the request is coming from, routing users to their nearest region.
• DNS-based failover: if health checks (run by the DNS provider itself, e.g., Route 53 health checks) detect a whole region is down, DNS stops returning that region's IP, redirecting new lookups to a healthy region.

The known limitation, worth naming: DNS results get cached by resolvers and clients based on a TTL (time-to-live). A short TTL means faster failover but more DNS query volume; a long TTL means slower failover (some clients keep using the old, dead IP until their cache expires) but less DNS load. This is a genuine, explicit tradeoff teams tune deliberately.

---

Putting It Together: A Full Request's Journey#

Diagram

This single diagram ties every concept in this tutorial into one flow: DNS handles global/regional routing, the load balancer handles server-level routing using an algorithm, health checks ensure only actually working servers receive traffic, and redundancy exists at every layer along the way.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: Your service has 10 redundant, active-active application servers behind a load balancer, but a recent postmortem revealed a full outage anyway. Investigation shows all 10 servers were healthy the entire time. What's the most likely category of root cause, and how would you find it?

Answer: If all 10 redundant app servers were healthy, the SPOF is almost certainly somewhere else in the stack that isn't redundant — the load balancer itself (if there's only one instance), DNS, the shared database, a shared cache, or a shared upstream dependency all 10 servers call. I'd systematically walk every layer in the "redundancy at every layer" diagram and ask "if this one thing died, would we have stayed up?" until I find the layer where the answer is no.

Problem 2: A caching cluster of 8 Memcached nodes uses plain hash(key) % 8 to decide ownership. The team wants to add a 9th node to handle growing load. What will happen, and what should they use instead?

Answer: Changing from % 8 to % 9 reassigns the vast majority of keys to different nodes than before, even though only one node was added — nearly every cache entry becomes a "miss" on its new node simultaneously, hammering the origin database with a stampede of requests it wasn't prepared for. They should use consistent hashing instead, so adding the 9th node only reassigns the small slice of keys nearest to it on the hash ring, leaving the rest of the cache warm and undisturbed.

Problem 3: A team's load balancer uses a shallow health check (GET /health just returns 200 OK if the process is running). During an incident, the database went down, but the load balancer kept sending traffic to all app servers, and every single request failed. What went wrong, and how would you fix the health check?

Answer: The shallow health check only confirmed the app process itself was alive — it never verified the app could actually reach its critical dependency (the database), so the load balancer had no way to know every server was actually broken. Fix: make /health also verify the database connection (a "deep" health check) so the load balancer can detect this condition. Caveat worth naming: if every server's health check depends on the same broken database, they'll all get marked unhealthy simultaneously, which just turns a partial outage into a "no servers available" total outage at the load balancer — a real tradeoff to think through (sometimes the better answer is a fast-failing circuit breaker in front of the DB call, covered in Part 2, rather than taking the whole server out of rotation).

---

Summary and What's Next#

• High Availability = the system keeps working even when a piece of it breaks — achieved almost entirely through redundancy: never depend on just one of anything important.
• Active-passive (one working copy, one standby) is simpler; active-active (all copies working simultaneously) is more resource-efficient but harder, especially for stateful systems.
• Failover speed depends on detection time, decision time, and switch-over time — DNS-based failover is notably slower than load-balancer-level failover, because of DNS caching (TTLs).
• Hunt for Single Points of Failure at every layer — redundant app servers don't help if they all share one non-redundant database or load balancer.
• Availability Zones protect against data-center-level failures cheaply; multi-region protects against region-level failures but is expensive and complex — use it deliberately, not by default.
• Layer 4 load balancing is fast but "dumb" (IP/port only); Layer 7 is smarter (can route by URL/headers) but does more work per request.
• Load balancing algorithms range from simple (round robin) to load-aware (least connections) to sticky (IP hash) — consistent hashing solves the specific, important problem of minimizing reshuffling when the number of servers changes.
• Health checks determine which servers actually receive traffic — shallow checks (is the process alive) can miss real problems that deep checks (can it actually serve requests) catch, but deep checks bring their own tradeoffs.

Continue to Part 2 (02-resilience-patterns.md) to see what happens after a request reaches a healthy server — how to protect the system when a downstream dependency starts failing, using circuit breakers, retries, timeouts, and rate limiting.