# Reliability & Architecture Patterns — Part 1: High Availability & Load Balancing

> **Series:** Reliability & Architecture Patterns (1 of 3)
> **Part 1:** This file — High Availability & Load Balancing
> **Part 2:** `02-resilience-patterns.md` — Circuit Breakers, Retries, Timeouts, Rate Limiting
> **Part 3:** `03-cap-theorem-and-consistency.md` — CAP Theorem & Consistency Models
> **Questions:** `questions.md`

## Table of Contents

1. [Why This Topic Matters](#why-this-topic-matters)
2. [What "High Availability" Actually Means](#what-high-availability-actually-means)
3. [The Core Idea: Redundancy](#the-core-idea-redundancy)
4. [Active-Passive vs Active-Active](#active-passive-vs-active-active)
5. [Failover — How the Switch Actually Happens](#failover--how-the-switch-actually-happens)
6. [Single Points of Failure (SPOFs)](#single-points-of-failure-spofs)
7. [Redundancy at Every Layer](#redundancy-at-every-layer)
8. [Availability Zones vs Regions](#availability-zones-vs-regions)
9. [What Is a Load Balancer, Really?](#what-is-a-load-balancer-really)
10. [Layer 4 vs Layer 7 Load Balancing](#layer-4-vs-layer-7-load-balancing)
11. [Load Balancing Algorithms](#load-balancing-algorithms)
12. [Health Checks — How the Load Balancer Knows Who's Alive](#health-checks--how-the-load-balancer-knows-whos-alive)
13. [Consistent Hashing — Solving a Real Problem](#consistent-hashing--solving-a-real-problem)
14. [DNS-Based Load Balancing and Global Traffic](#dns-based-load-balancing-and-global-traffic)
15. [Putting It Together: A Full Request's Journey](#putting-it-together-a-full-requests-journey)
16. [Common Mistakes](#common-mistakes)
17. [Worked Practice Problems](#worked-practice-problems)
18. [Summary and What's Next](#summary-and-whats-next)

---

## Why This Topic Matters

Think about the last time a website "just worked" even though, somewhere behind the scenes, a server crashed, a network cable got unplugged, or an entire data center lost power. That's not luck — it's *design*. This tutorial is about the handful of ideas that make that possible: having backups ready to go, spreading traffic across many machines instead of one, and knowing what to do the instant something breaks.

If SLIs/SLOs (from the SRE Fundamentals series) tell you *how reliable you need to be*, this tutorial is about the actual building blocks that get you there.

```mermaid
mindmap
  root((Reliability<br/>Building Blocks))
    Redundancy
      Don't have just one of anything important
    Load Balancing
      Spread the work around
    Failover
      Switch to the backup automatically
    Health Checks
      Know who's actually alive
```

---

## What "High Availability" Actually Means

High Availability (HA) simply means: **the system keeps working even when a piece of it breaks.** Not "nothing ever breaks" — things break all the time, in any system big enough to matter. HA is about making sure a single broken piece doesn't take the whole thing down.

**A simple analogy:** think of a restaurant with one chef. If that chef gets sick, the restaurant closes for the day. Now imagine the same restaurant with three chefs on rotation — if one gets sick, the other two cover, and customers never even notice. That's the entire idea behind HA: **don't depend on just one of anything that matters.**

```mermaid
graph LR
    A["Single chef<br/>(single point of failure)"] -->|"chef gets sick"| B["Restaurant closes<br/>❌"]
    C["Three chefs on rotation<br/>(redundancy)"] -->|"one chef gets sick"| D["Other two cover,<br/>restaurant stays open<br/>✅"]
```

---

## The Core Idea: Redundancy

**Redundancy** just means having more than one of something important, so that if one fails, another can take over. It sounds almost too simple to be a whole engineering discipline — but almost every HA pattern is some specific *flavor* of redundancy applied to a specific part of the system.

| What You're Protecting | The Redundant Backup |
|---|---|
| A single server | A second (or third, fourth...) identical server |
| A single database | A replica database that mirrors the primary |
| A single data center | A second data center in a different location |
| A single network path | A second network provider/route |
| A single power source | A backup generator / battery |

The tricky part isn't "have a backup" — it's **how do you know when to use the backup, and how do you switch to it without anyone noticing?** That's what the rest of this tutorial is about.

---

## Active-Passive vs Active-Active

There are two basic ways to arrange your redundant copies.

### Active-Passive (a.k.a. Active-Standby)

One copy does all the work (the "active" one). The other copy sits there, ready, doing nothing — until the active one fails, at which point the passive copy takes over.

```mermaid
graph LR
    User[User Traffic] --> Active["Active Server<br/>(handling everything)"]
    Passive["Passive Server<br/>(idle, waiting)"] -.->|"standing by"| Active

    Active -->|"fails!"| Fail[❌]
    Fail -.->|"failover"| Passive2["Passive becomes Active<br/>✅ now handling traffic"]
```

**Analogy:** a spare tire in your car trunk. It does nothing 99.9% of the time, but the moment you get a flat, you swap it in.

- **Pro:** simple to reason about — only one copy is ever actually serving traffic, so there's no risk of the two copies disagreeing with each other.
- **Con:** you're paying for a whole second server that sits idle almost all the time — wasted capacity.

### Active-Active

Both (or all) copies are handling real traffic, all the time, simultaneously. If one fails, the others just absorb its share of the load.

```mermaid
graph LR
    User[User Traffic] --> LB[Load Balancer]
    LB --> A1["Server A<br/>(active)"]
    LB --> A2["Server B<br/>(active)"]
    LB --> A3["Server C<br/>(active)"]

    A2 -->|"fails!"| Fail[❌]
    Fail -.-> Redistribute["Load Balancer stops<br/>sending traffic to B,<br/>A and C absorb the extra load"]
```

**Analogy:** three chefs cooking at the same time (back to our restaurant), instead of two chefs sitting at home waiting for the third to call in sick.

- **Pro:** no wasted capacity — every server is doing useful work, all the time.
- **Con:** harder to build — if the servers share data (like a database), you now have to worry about keeping that data consistent across multiple active copies at once (this connects directly to the CAP Theorem tutorial, Part 3).

### Quick Comparison

| | Active-Passive | Active-Active |
|---|---|---|
| Resource efficiency | Lower (standby sits idle) | Higher (everyone works) |
| Complexity | Lower | Higher (data consistency across active copies) |
| Failover speed | Depends on detection + switch-over time | Often near-instant (just stop routing to the dead one) |
| Common use case | Databases (primary/replica), simpler systems | Stateless web/API servers, CDNs |

**Interview tip:** a very common follow-up question is "which would you use for a stateless API vs. a database?" Answer: **active-active for stateless services** (easy — any server can handle any request), and **active-passive is much more common for databases** specifically because keeping multiple databases simultaneously writable and consistent is genuinely hard (again, this is CAP Theorem territory).

---

## Failover — How the Switch Actually Happens

**Failover** is the actual mechanical process of detecting a failure and redirecting traffic away from the broken piece.

```mermaid
sequenceDiagram
    participant Monitor as Health Monitor
    participant Primary as Primary Server
    participant Standby as Standby Server
    participant LB as Load Balancer / DNS

    Monitor->>Primary: "Are you healthy?" (repeated check)
    Primary-->>Monitor: "Yes" ✅
    Note over Monitor,Primary: ... time passes ...
    Monitor->>Primary: "Are you healthy?"
    Primary--xMonitor: (no response) ❌
    Monitor->>Primary: Retry (avoid a false alarm)
    Primary--xMonitor: (still no response)
    Monitor->>Standby: Promote to active
    Standby->>Standby: Takes over the primary role
    Monitor->>LB: Update routing to point at Standby
    LB->>Standby: New traffic now goes here
```

### Three Things That Determine How Painful a Failover Is

1. **Detection time** — how long until the system *notices* something's wrong? (Usually a few failed health checks in a row, to avoid overreacting to one blip.)
2. **Decision time** — how confident do we need to be before flipping the switch? (Flip too eagerly and you get "flapping" — switching back and forth on transient issues. Flip too cautiously and users suffer longer.)
3. **Switch-over time** — once decided, how long until traffic is actually flowing to the new active copy? (DNS changes can take minutes to propagate; a load balancer health check removing a bad target can take seconds.)

```mermaid
graph LR
    A[Failure happens] --> B["Detection time<br/>(seconds to minutes)"]
    B --> C["Decision time<br/>(avoid false alarms)"]
    C --> D["Switch-over time<br/>(DNS: slow, LB: fast)"]
    D --> E[Users back to normal]
```

**A concrete, often-tested number:** DNS-based failover can take minutes because of DNS caching (clients and resolvers cache the old IP address based on its TTL). This is exactly why load-balancer-based failover (where the LB itself detects the failure and simply stops routing to the bad server) is usually much faster than DNS-based failover, and why DNS failover is typically reserved for larger-scale events (like an entire region going down) rather than a single failed server.

---

## Single Points of Failure (SPOFs)

A **Single Point of Failure** is any one component that, if it breaks, takes the whole system down — because nothing is backing it up.

```mermaid
flowchart TD
    User --> LB["Load Balancer<br/>(only ONE instance!)"]
    LB --> App1[App Server 1]
    LB --> App2[App Server 2]
    App1 --> DB["Database<br/>(only ONE instance!)"]
    App2 --> DB

    LB -.->|"SPOF! If this dies,<br/>EVERYTHING dies,<br/>even though App1/App2<br/>are individually redundant"| SPOF1[⚠️]
    DB -.->|"SPOF! Redundant app<br/>servers don't help if they<br/>all depend on ONE database"| SPOF2[⚠️]
```

**The key insight interviewers look for:** redundancy at one layer doesn't help if there's a SPOF at another layer. Having 10 redundant app servers is worthless if they all point at a single, non-redundant database, or if they're all reachable only through a single, non-redundant load balancer. **You have to hunt for SPOFs at every layer of the stack**, not just the layer you happened to think about first.

### A Practical SPOF-Hunting Checklist

```mermaid
graph TD
    Q[Ask this about EVERY component] --> A["If this one thing disappeared<br/>right now, would the system<br/>keep working?"]
    A -->|No| SPOF["🚨 It's a SPOF —<br/>needs redundancy"]
    A -->|Yes| OK["✅ Not a SPOF"]
```

Run this question against: load balancers, DNS providers, databases, message queues, caches, the network path itself, the power supply, even the *team* (the "bus factor" — what if the one person who understands this system is unavailable?).

---

## Redundancy at Every Layer

A realistic, full-stack view of where redundancy typically needs to exist:

```mermaid
graph TD
    Internet[Internet] --> DNS["DNS<br/>(multiple providers/records)"]
    DNS --> CDN["CDN / Edge<br/>(globally distributed by design)"]
    CDN --> LB["Load Balancers<br/>(usually 2+, often<br/>themselves behind a<br/>redundant setup)"]
    LB --> App["Application Servers<br/>(N replicas, active-active)"]
    App --> Cache["Cache Layer<br/>(e.g. Redis cluster,<br/>not a single instance)"]
    App --> DB["Database<br/>(primary + replica(s),<br/>often active-passive)"]
    App --> Queue["Message Queue<br/>(clustered, e.g. Kafka<br/>with replication factor > 1)"]
```

**Interview-ready summary line:** "High availability isn't one technique — it's the discipline of asking 'what happens if this specific piece dies' at every single layer, from DNS all the way down to the database, and making sure the answer is never 'everything stops.'"

---

## Availability Zones vs Regions

A very common piece of cloud vocabulary, worth being crisp about.

```mermaid
graph TD
    Region["Region<br/>(e.g. us-east-1)"] --> AZ1["Availability Zone A<br/>(a physically separate<br/>data center)"]
    Region --> AZ2["Availability Zone B<br/>(a different data center,<br/>same region)"]
    Region --> AZ3["Availability Zone C"]

    Region2["Region 2<br/>(e.g. eu-west-1)<br/>— physically far away,<br/>different continent/country"]
```

- **Availability Zone (AZ)**: one or more physically distinct data centers within the same broad geographic area, with independent power, cooling, and networking — but low-latency, high-bandwidth connections between AZs in the same region.
- **Region**: a geographically distinct area (e.g., "US East" vs "Europe West") containing multiple AZs.

| Failure Scenario | Protected By |
|---|---|
| One server dies | Multiple instances within one AZ |
| An entire data center loses power/cooling | Multi-AZ deployment (spread across 2-3 AZs in the region) |
| An entire region goes offline (rare, but happens — natural disaster, major cloud provider outage) | Multi-region deployment |

**The tradeoff to always name explicitly:** multi-AZ is relatively cheap (low latency between AZs, often included in standard architecture) and protects against the vast majority of real failures. Multi-region is much more expensive and architecturally complex (cross-region data replication has real latency and consistency costs — see the CAP Theorem tutorial), and is usually reserved for services with extremely strict availability requirements or specific regulatory/data-residency needs.

---

## What Is a Load Balancer, Really?

At its core, a load balancer does one job: **it receives traffic and decides which backend server should handle each request.**

```mermaid
flowchart LR
    Clients["Many Clients"] --> LB["Load Balancer"]
    LB --> S1[Server 1]
    LB --> S2[Server 2]
    LB --> S3[Server 3]
```

Why not just let clients talk directly to servers? Two big reasons:
1. **Spreading load evenly** so no single server gets overwhelmed while others sit idle.
2. **Hiding failures** — if Server 2 dies, the load balancer simply stops sending it traffic; clients never need to know Server 2 ever existed.

---

## Layer 4 vs Layer 7 Load Balancing

This is one of the most commonly tested "networking basics" questions in SRE interviews.

```mermaid
graph TD
    L4["Layer 4 Load Balancer<br/>(Transport layer — TCP/UDP)"] --> L4a["Looks at: IP address + port"]
    L4 --> L4b["Doesn't look inside the<br/>actual request content"]
    L4 --> L4c["Very fast, simple"]

    L7["Layer 7 Load Balancer<br/>(Application layer — HTTP)"] --> L7a["Looks at: URL path, headers,<br/>cookies, HTTP method"]
    L7 --> L7b["Can make SMART routing<br/>decisions based on content"]
    L7 --> L7c["Slightly slower<br/>(more work per request),<br/>more flexible"]
```

**Analogy:** Layer 4 is like a mail sorter who only looks at the zip code on an envelope and sends it to the right regional office — fast, but doesn't read what's inside. Layer 7 is like a receptionist who actually reads your request ("I'm here for a dentist appointment") and sends you to exactly the right room — slower per person, but much smarter routing.

| | Layer 4 (Transport) | Layer 7 (Application) |
|---|---|---|
| Sees | IP + port only | Full HTTP request (path, headers, cookies, body) |
| Can route by URL path (e.g. `/api` vs `/static`)? | ❌ No | ✅ Yes |
| Can do SSL termination? | Usually not | ✅ Yes, commonly |
| Speed | Faster (less to inspect) | Slightly slower (more to inspect) |
| Common real examples | AWS Network Load Balancer (NLB), raw TCP load balancers | AWS Application Load Balancer (ALB), NGINX, HAProxy (L7 mode), Envoy |

**A great interview answer to "when would you use L4 vs L7":** "L4 when I need raw speed and don't need content-aware routing — e.g., a generic TCP service. L7 when I need to route based on the actual request — e.g., sending `/api/*` to one backend and `/static/*` to a CDN-backed bucket, or doing SSL termination centrally instead of on every backend server."

---

## Load Balancing Algorithms

Once the load balancer decides *that* a request needs to go to *some* backend, it needs a rule for *which one*.

```mermaid
graph TD
    Algo[Load Balancing Algorithms] --> RR[Round Robin]
    Algo --> WRR[Weighted Round Robin]
    Algo --> LC[Least Connections]
    Algo --> LRT[Least Response Time]
    Algo --> IPH[IP Hash]
    Algo --> CH[Consistent Hashing]

    RR --> RR1["Send requests to servers<br/>in a fixed rotation: 1, 2, 3, 1, 2, 3..."]
    WRR --> WRR1["Like Round Robin, but bigger<br/>servers get more turns<br/>(e.g. 3:1 ratio for a bigger box)"]
    LC --> LC1["Send the next request to<br/>whichever server currently has<br/>the FEWEST active connections"]
    LRT --> LRT1["Send to whichever server has<br/>been responding FASTEST recently"]
    IPH --> IPH1["Same client IP always goes<br/>to the same server<br/>(simple 'sticky' routing)"]
    CH --> CH1["A smarter version of IP Hash<br/>that barely reshuffles anything<br/>when servers are added/removed"]
```

| Algorithm | How It Works | Best For | Weakness |
|---|---|---|---|
| **Round Robin** | Rotate through servers in order | Simple, uniform requests, equal-sized servers | Ignores actual server load — a slow server still gets its "turn" |
| **Weighted Round Robin** | Rotate, but bigger servers get proportionally more turns | Mixed server sizes | Still ignores real-time load |
| **Least Connections** | Send to the server with the fewest active connections right now | Requests with very different processing times | Slightly more overhead to track connection counts |
| **Least Response Time** | Send to the server that's been fastest recently | Latency-sensitive services | Needs constant response-time tracking |
| **IP Hash** | Hash the client's IP to consistently pick the same server | Simple session "stickiness" without a shared session store | Uneven load if client IPs aren't evenly distributed; breaks if a server goes down (see next section) |
| **Consistent Hashing** | A hashing scheme designed so that adding/removing a server barely disturbs existing assignments | Caching layers, sharded systems | More complex to implement than plain IP hash |

---

## Health Checks — How the Load Balancer Knows Who's Alive

A load balancer is only as good as its ability to tell healthy servers from broken ones — this is what makes the "automatically stop sending traffic to a dead server" behavior work.

```mermaid
sequenceDiagram
    participant LB as Load Balancer
    participant S1 as Server 1
    participant S2 as Server 2

    loop Every few seconds
        LB->>S1: GET /health
        S1-->>LB: 200 OK ✅
        LB->>S2: GET /health
        S2--xLB: timeout / 500 ❌
    end
    Note over LB,S2: After N consecutive failures,<br/>LB marks Server 2 as unhealthy<br/>and stops routing traffic to it
    Note over LB,S1: LB keeps routing all<br/>traffic to Server 1
```

### Two Kinds of Health Checks

| Type | What It Checks | Example |
|---|---|---|
| **Shallow health check** | "Is the process even running and responding?" | `GET /health` returns 200 if the web server process is alive |
| **Deep health check** | "Can this server actually do its job right now?" | `GET /health` also checks: can I reach the database? Is my cache connection alive? |

**A subtle but important interview point:** a shallow health check can lie — the web server process might be "up" and responding to `/health`, but if its database connection is broken, it can't actually serve real traffic. A good health check should verify the dependencies that actually matter for serving real requests, not just "is the process alive" — but it also shouldn't check *too* much (e.g., checking a slow, rarely-used downstream dependency could cause a healthy server to be wrongly marked as down). This is a genuine design tradeoff worth naming explicitly if asked.

---

## Consistent Hashing — Solving a Real Problem

This deserves its own spotlight because it's a frequent, specific interview question: "why not just use plain hashing (`hash(key) % N`) to decide which server/shard handles a given key?"

### The Problem With Plain Hashing

```mermaid
graph TD
    A["hash(key) % N servers"] --> B["Works fine... until N changes<br/>(a server is added or removed)"]
    B --> C["Now EVERY key's assignment<br/>changes, because the modulo<br/>changed for almost everything"]
    C --> D["Massive, unnecessary<br/>reshuffling — e.g. almost<br/>every cache entry suddenly<br/>maps to a different server<br/>→ cache stampede"]
```

**Concrete example:** with 4 servers, `key_id % 4` decides the owner. Add a 5th server, and now it's `key_id % 5` — almost every single key gets remapped to a *different* server than before, even though only one server was added. For a cache, this means nearly every cached item suddenly looks like a "miss" on its new server, hammering the origin database all at once.

### The Consistent Hashing Fix

```mermaid
graph TD
    Ring["Imagine a circle (a 'hash ring')"] --> Place["Both SERVERS and KEYS are<br/>placed on the ring using a hash function"]
    Place --> Rule["Rule: a key belongs to the<br/>FIRST server found going<br/>clockwise around the ring"]
    Rule --> AddServer["Adding a new server only<br/>affects the small slice of<br/>keys between it and its<br/>nearest neighbor on the ring —<br/>everything else is untouched"]
```

```mermaid
graph LR
    subgraph "Hash Ring"
    S1((Server A)) --- S2((Server B)) --- S3((Server C)) --- S1
    end
    K1[Key 1] -.->|"clockwise to nearest server"| S1
    K2[Key 2] -.-> S2
    K3[Key 3] -.-> S3
```

**Why this matters in plain terms:** with consistent hashing, adding a 4th server only reshuffles the small handful of keys that happen to fall right next to it on the ring — everything else stays exactly where it was. This is why consistent hashing is the standard technique behind distributed caches (Memcached client libraries), CDNs, and sharded databases — it lets you scale up or down without a massive, disruptive reshuffle every time.

---

## DNS-Based Load Balancing and Global Traffic

For routing traffic across entire *regions* (not just servers within one data center), DNS itself often does the load balancing.

```mermaid
flowchart TD
    User["User in Europe"] --> DNS["DNS Resolver<br/>(GeoDNS-aware)"]
    DNS -->|"Returns IP for<br/>nearest healthy region"| EU["eu-west region"]
    User2["User in USA"] --> DNS
    DNS -->|"Returns different IP"| US["us-east region"]
```

- **GeoDNS**: returns a different IP address depending on where the request is coming from, routing users to their nearest region.
- **DNS-based failover**: if health checks (run by the DNS provider itself, e.g., Route 53 health checks) detect a whole region is down, DNS stops returning that region's IP, redirecting new lookups to a healthy region.

**The known limitation, worth naming:** DNS results get **cached** by resolvers and clients based on a TTL (time-to-live). A short TTL means faster failover but more DNS query volume; a long TTL means slower failover (some clients keep using the old, dead IP until their cache expires) but less DNS load. This is a genuine, explicit tradeoff teams tune deliberately.

---

## Putting It Together: A Full Request's Journey

```mermaid
sequenceDiagram
    participant User
    participant DNS as GeoDNS
    participant LB as Regional Load Balancer (L7)
    participant HC as Health Checker
    participant App as App Server (1 of many)
    participant DB as Database (primary)

    User->>DNS: Resolve api.example.com
    DNS-->>User: Returns IP of nearest healthy region
    User->>LB: HTTPS request
    HC->>App: (continuously) health checks
    LB->>App: Routes based on algorithm<br/>(e.g. least connections)<br/>+ only to healthy servers
    App->>DB: Query
    DB-->>App: Result
    App-->>LB: Response
    LB-->>User: Response
```

This single diagram ties every concept in this tutorial into one flow: DNS handles *global/regional* routing, the load balancer handles *server-level* routing using an algorithm, health checks ensure only *actually working* servers receive traffic, and redundancy exists at every layer along the way.

---

## Common Mistakes

| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Adding redundant app servers but leaving a single database | The database becomes the SPOF that undoes all the app-layer redundancy | Hunt for SPOFs at *every* layer, not just the one you thought about first |
| Using a shallow health check only | A server can pass "is the process alive" while its actual dependencies are broken, and still receive real traffic it can't serve | Use a health check that verifies the dependencies that matter for real requests (without checking too deeply and causing false negatives) |
| Assuming DNS failover is instant | DNS caching (TTL) means some clients keep hitting the dead endpoint for minutes | Use load-balancer-level failover for fast, single-server issues; reserve DNS failover for larger, region-level events, and tune TTLs deliberately |
| Using plain `hash % N` for a cache/shard cluster that changes size | Adding/removing one server reshuffles almost everything, causing a stampede | Use consistent hashing instead |
| Treating active-active as a free upgrade over active-passive | Active-active often requires solving hard data-consistency problems across simultaneously-active copies | Only go active-active where the added complexity is actually worth it (typically: stateless services) |
| Confusing Availability Zone redundancy with true Region redundancy | Multi-AZ doesn't protect against a whole-region outage | Explicitly decide, based on real requirements, whether multi-region is actually needed — it's expensive and complex |

---

## Worked Practice Problems

**Problem 1:** Your service has 10 redundant, active-active application servers behind a load balancer, but a recent postmortem revealed a full outage anyway. Investigation shows all 10 servers were healthy the entire time. What's the most likely category of root cause, and how would you find it?

*Answer:* If all 10 redundant app servers were healthy, the SPOF is almost certainly somewhere *else* in the stack that isn't redundant — the load balancer itself (if there's only one instance), DNS, the shared database, a shared cache, or a shared upstream dependency all 10 servers call. I'd systematically walk every layer in the "redundancy at every layer" diagram and ask "if this one thing died, would we have stayed up?" until I find the layer where the answer is no.

**Problem 2:** A caching cluster of 8 Memcached nodes uses plain `hash(key) % 8` to decide ownership. The team wants to add a 9th node to handle growing load. What will happen, and what should they use instead?

*Answer:* Changing from `% 8` to `% 9` reassigns the vast majority of keys to different nodes than before, even though only one node was added — nearly every cache entry becomes a "miss" on its new node simultaneously, hammering the origin database with a stampede of requests it wasn't prepared for. They should use consistent hashing instead, so adding the 9th node only reassigns the small slice of keys nearest to it on the hash ring, leaving the rest of the cache warm and undisturbed.

**Problem 3:** A team's load balancer uses a shallow health check (`GET /health` just returns `200 OK` if the process is running). During an incident, the database went down, but the load balancer kept sending traffic to all app servers, and every single request failed. What went wrong, and how would you fix the health check?

*Answer:* The shallow health check only confirmed the app process itself was alive — it never verified the app could actually reach its critical dependency (the database), so the load balancer had no way to know every server was actually broken. Fix: make `/health` also verify the database connection (a "deep" health check) so the load balancer can detect this condition. Caveat worth naming: if *every* server's health check depends on the same broken database, they'll *all* get marked unhealthy simultaneously, which just turns a partial outage into a "no servers available" total outage at the load balancer — a real tradeoff to think through (sometimes the better answer is a fast-failing circuit breaker in front of the DB call, covered in Part 2, rather than taking the whole server out of rotation).

---

## Summary and What's Next

- **High Availability** = the system keeps working even when a piece of it breaks — achieved almost entirely through **redundancy**: never depend on just one of anything important.
- **Active-passive** (one working copy, one standby) is simpler; **active-active** (all copies working simultaneously) is more resource-efficient but harder, especially for stateful systems.
- **Failover speed** depends on detection time, decision time, and switch-over time — DNS-based failover is notably slower than load-balancer-level failover, because of DNS caching (TTLs).
- Hunt for **Single Points of Failure at every layer** — redundant app servers don't help if they all share one non-redundant database or load balancer.
- **Availability Zones** protect against data-center-level failures cheaply; **multi-region** protects against region-level failures but is expensive and complex — use it deliberately, not by default.
- **Layer 4** load balancing is fast but "dumb" (IP/port only); **Layer 7** is smarter (can route by URL/headers) but does more work per request.
- Load balancing algorithms range from simple (**round robin**) to load-aware (**least connections**) to sticky (**IP hash**) — **consistent hashing** solves the specific, important problem of minimizing reshuffling when the number of servers changes.
- **Health checks** determine which servers actually receive traffic — shallow checks (is the process alive) can miss real problems that deep checks (can it actually serve requests) catch, but deep checks bring their own tradeoffs.

**Continue to Part 2** (`02-resilience-patterns.md`) to see what happens *after* a request reaches a healthy server — how to protect the system when a downstream dependency starts failing, using circuit breakers, retries, timeouts, and rate limiting.