Companion question bank for the 3-part tutorial series in this folder: 01-high-availability-and-load-balancing.md, 02-resilience-patterns.md, 03-cap-theorem-and-consistency.md.

Answers are kept short and plain — expand out loud using the analogies and diagrams from the tutorials.

---

Part 1 Questions: High Availability & Load Balancing

Conceptual#

1. What does "High Availability" actually mean?#

The system keeps working even when a piece of it breaks. Not "nothing ever breaks" — things break constantly at scale. HA is about making sure one broken piece doesn't take down everything else.

2. What's the core idea behind almost every HA pattern?#

Redundancy — never depend on just one of anything important. Like a restaurant with three chefs instead of one: if one gets sick, the other two cover and nobody notices.

3. What's the difference between active-passive and active-active, and when would you use each?#

Active-passive: one copy does all the work, a second sits idle as backup, ready to take over. Simple, but wastes capacity. Active-active: all copies handle real traffic simultaneously — no waste, but harder for anything stateful (multiple copies of a database being written to at once gets genuinely complicated). Use active-active for stateless services (any server can handle any request); active-passive is much more common for databases specifically because keeping multiple databases simultaneously consistent is hard.

4. What three things determine how painful a failover is?#

Detection time (how long until we notice something's wrong), decision time (how sure do we need to be before flipping the switch, to avoid false alarms), and switch-over time (how long until traffic actually reaches the new active copy).

5. Why is DNS-based failover generally slower than load-balancer-based failover?#

DNS results get cached by clients and resolvers based on a TTL. Even after DNS starts returning the new, healthy IP, some clients keep using the old cached IP until their cache expires — which can take minutes. A load balancer, by contrast, can just stop routing to a dead server instantly once its health check fails.

6. What is a Single Point of Failure (SPOF), and why is it dangerous even in a system that looks well-designed?#

Any one component that, if it breaks, takes the whole system down. It's dangerous specifically because redundancy at one layer doesn't help if there's a SPOF at another — e.g., 10 redundant app servers behind one non-redundant load balancer are all worthless the moment that one load balancer dies. You have to check every layer, not just the layer you happened to think about.

7. What's the difference between an Availability Zone and a Region, and what does each protect against?#

An Availability Zone is a physically distinct data center within the same broad area, with low-latency links to other AZs in the same region — protects against one data center losing power/cooling. A Region is a much larger geographic area containing multiple AZs — multi-region deployment protects against an entire region going offline, but is far more expensive and complex.

8. What's the difference between Layer 4 and Layer 7 load balancing?#

Layer 4 only looks at IP address and port — fast, but "dumb," like a mail sorter who only reads the zip code. Layer 7 reads the actual HTTP request — path, headers, cookies — and can make smart routing decisions (like sending /api to one backend and /static to another), like a receptionist who actually listens to what you need. L7 does more work per request and is slightly slower, but far more flexible.

9. Name a few load balancing algorithms and when you'd use each.#

Round robin (simple rotation, works when servers/requests are uniform), least connections (send to whoever's least busy right now, good when request processing time varies a lot), IP hash (same client always hits the same server — simple stickiness), consistent hashing (like IP hash, but barely disturbs existing assignments when servers are added/removed).

10. Why does plain hash(key) % N break down when N (the number of servers) changes?#

Because changing N changes the modulo result for almost every key, remapping nearly everything to a different server — even though only one server was added or removed. For a cache, that means nearly every cached item suddenly looks like a miss on its new server, hammering the origin all at once.

11. How does consistent hashing fix that problem?#

Picture a circle (a "hash ring") with both servers and keys placed on it using a hash function. A key belongs to the first server found going clockwise. Adding a new server only steals the small slice of keys nearest to it on the ring — everything else stays exactly where it was, instead of a massive reshuffle.

12. What's the difference between a shallow and a deep health check?#

Shallow: "is the process even running and responding?" Deep: "can this server actually reach its real dependencies (database, cache) and do its job right now?" A shallow check can lie — the process can be "up" while its database connection is dead, meaning it can't actually serve real traffic even though it looks healthy.

Applied / Scenario#

13. You have 10 redundant app servers, all reported healthy the whole time, but a postmortem shows a full outage anyway. Where do you look?#

Somewhere else in the stack that isn't redundant — the load balancer (if there's only one), DNS, a shared database, a shared cache. Walk every layer and ask "if this one thing died right now, would we still be up?" until you find the layer where the answer is no.

14. A team wants to add a 9th node to an 8-node cache cluster that uses plain hash(key) % 8. What will happen, and what should they use instead?#

Almost every key gets remapped to a different node (since % 8 becomes % 9), causing a stampede of cache misses hitting the origin all at once. They should switch to consistent hashing so adding the 9th node only reassigns the small slice of keys nearest to it.

---

Part 2 Questions: Resilience Patterns

Conceptual#

15. Why should every network call have a timeout?#

Without one, a hung dependency can make your service wait forever — one by one, requests pile up until your service runs out of resources (threads, connections) to handle anything, even requests that had nothing to do with the broken dependency. One slow dependency turning into a total, unrelated outage is a classic, preventable failure mode.

16. Should you always retry a failed request?#

No — only retry failures likely to be transient (timeouts, 503s). Don't retry failures that will fail identically every time, like a malformed request (400) — that just wastes time and resources on a request that can never succeed as-is.

17. What is the idempotency problem with retries, and how do you fix it?#

If a write (like "charge $50") actually succeeds on the server but the response gets lost on the way back, the client thinks it failed and retries — potentially causing the charge to happen twice. Fix: idempotency keys — the client sends a unique ID for the intent, and the server remembers which IDs it's already processed, returning the original result instead of repeating the effect if it sees the same ID again.

18. Why isn't exponential backoff enough on its own — what problem does jitter solve?#

If 1,000 clients fail at the same moment and all use the identical backoff schedule, they'll all retry again at the exact same instant, recreating the exact overload they were trying to avoid. Jitter adds randomness to the wait time so retries spread out smoothly over time instead of arriving in one synchronized spike.

19. Walk me through a "retry storm" from start to finish.#

A dependency gets slightly overloaded and starts responding slowly. Callers retry failed requests immediately, with no backoff — piling MORE load on exactly when the dependency can least handle it. It gets more overloaded, causing even more timeouts and retries. This vicious cycle can turn a minor slowdown into a full outage, entirely caused by well-intentioned but naive retries.

20. Describe the three states of a circuit breaker and why "half-open" exists.#

Closed: normal operation, requests flow through, failures are counted. Open: the breaker has "tripped" — requests fail instantly without even trying, once failures cross a threshold. Half-Open: after a cooldown, a small number of test requests are cautiously let through to check if the dependency has recovered. Half-Open exists because snapping straight back to full traffic the instant the cooldown ends could immediately re-trip a dependency that's only barely recovered — testing with a trickle first is safer.

21. What does a circuit breaker actually protect against that timeouts and retries don't?#

Timeouts bound how long you wait per call; retries try again on failure — but neither stops you from repeatedly wasting time and resources on calls to a dependency that's clearly, currently down. A circuit breaker specifically stops even attempting new calls once it's obvious the dependency is broken, failing instantly instead, which also gives the dependency room to recover instead of getting hammered by ongoing retries.

22. What's the token bucket rate-limiting algorithm, and why is it popular?#

Tokens refill into a bucket at a steady rate up to some max capacity; each request consumes one token, and an empty bucket means the request is rejected. It's popular because it naturally allows brief bursts (using up accumulated tokens from a quiet period) while still enforcing a steady average rate over time — unlike a strict, no-burst leaky bucket.

23. What's the weakness of a fixed-window rate limiter?#

A client can send the full limit right at the very end of one window and the full limit again right at the start of the next — technically obeying the "X per window" rule while bursting far beyond that rate across the boundary in a very short real time span.

24. What are bulkheads, and what real-world failure do they prevent?#

Isolating resources (like connection pools) per-dependency, so one hung dependency can only exhaust its own dedicated pool, not a shared one. Without bulkheads, one broken dependency can consume an entire shared resource pool, breaking calls to completely unrelated, healthy dependencies too — like a ship without watertight compartments sinking entirely from one hull breach.

25. What's the difference between graceful degradation and load shedding?#

Graceful degradation: when a non-essential dependency fails, serve a reduced-but-working experience instead of nothing (e.g., generic recommendations instead of personalized ones). Load shedding: when the whole system is genuinely overwhelmed, deliberately reject some requests so the ones that do get through are served well, instead of trying to serve everyone badly.

Applied / Scenario#

26. A payment service starts timing out. The caller retries 3 times with no backoff, no jitter, no circuit breaker, and 20 minutes later the payment service is fully down. What happened, and how would you fix it?#

Classic retry storm — the naive, immediate retries added load on top of an already-struggling service exactly when it could least handle it, driving it from "slow" to "fully down." Fix: add exponential backoff with jitter, cap total retry attempts, and add a circuit breaker so repeated failures stop generating new load once a clear failure pattern is detected.

27. Three dependencies (Inventory, Pricing, Shipping) share one connection pool of 50. Inventory hangs. What happens, and how do bulkheads fix it?#

All 50 connections can eventually get consumed waiting on the hung Inventory calls, leaving nothing for Pricing or Shipping — even though they're completely healthy. The whole service goes down over one bad dependency. Bulkheads give each dependency its own dedicated slice (e.g., 20/15/15) so Inventory hanging only exhausts its own allocation.

---

Part 3 Questions: CAP Theorem & Consistency Models

Conceptual#

28. In plain English, what does the CAP theorem say?#

When a network partition happens (part of the system can't talk to another part), you can only keep one of two promises: Consistency (only answer if you're sure the answer is correct/up to date) or Availability (always answer, even if the answer might be stale). You can't guarantee both at the same time, during the partition.

29. Why is "pick any 2 of C, A, P" a common but misleading way to state CAP?#

Because for any real system spanning more than one machine, network partitions aren't optional — they will eventually happen. "Not tolerating partitions" isn't a real choice for a genuinely distributed system. So the real, practical decision is always "CP vs AP" — what do you do when a partition happens — not "which 2 of 3 do we pick."

30. Does CAP mean a system sacrifices consistency or availability all the time?#

No — that's the most common misunderstanding. The tradeoff only applies during an actual partition, which is hopefully rare. The rest of the time, a well-designed system delivers both. CAP is really about your system's fallback behavior when something goes wrong, not its everyday behavior.

31. Give a real-world example of a CP system and an AP system, and explain the reasoning.#

CP: a single-primary relational database, or ZooKeeper/etcd — used where being wrong is worse than being unavailable (e.g., a bank balance check before a withdrawal — better to show an error than let someone overdraw). AP: Cassandra or DynamoDB's default mode — used where being unavailable is worse than being briefly stale (e.g., a shopping cart — showing a slightly-stale cart beats refusing to load the page).

32. What is PACELC, and what gap in CAP theorem does it fill?#

PACELC says: if there's a Partition, choose Availability or Consistency (that part is just CAP) — Else (i.e., normal operation, no partition), choose Latency or Consistency. CAP alone says nothing about the everyday tradeoff that exists even with a perfectly healthy network: waiting for strong consistency costs latency, while responding fast risks slightly stale data. This is why systems like DynamoDB let you choose "eventually consistent" (fast) or "strongly consistent" (slower) on every single read, regardless of whether a partition is even happening.

33. List consistency models from strongest to weakest, and briefly describe each.#

Strong (every read reflects the absolute latest write, everywhere, immediately) → Causal (unrelated writes can appear in any order, but causally related ones — like a reply and the comment it replies to — always appear in the correct order) → Read-your-writes/Session (you always see your own recent changes, though other users' changes might lag) → Eventual (all copies converge eventually, with no guarantee about exactly when).

34. What does "read-your-writes" consistency guarantee, and why is it a popular practical compromise?#

It guarantees you'll always see your own most recent writes, even though you might briefly see stale data from other users' writes. It's popular because full strong consistency for everyone is expensive, but "never show me my own stale data" is both cheap to implement and covers the single most common, jarring complaint users have about eventually-consistent systems ("I just changed this, why does it still look old?!").

35. Explain quorum-based consistency (the W + R > N rule) in your own words.#

With N total replicas, a write must succeed on at least W of them, and a read must check at least R of them. If W + R is greater than N, you're mathematically guaranteed every read overlaps with the most recent write on at least one replica — giving strong consistency without needing all N replicas to participate every time. Systems like Cassandra let you tune W and R per operation, trading off latency, availability, and consistency as needed.

36. How does Part 1's active-active pattern connect to the CAP theorem problem?#

Active-active for a stateful system (like a database) means multiple simultaneously-writable copies of the data — and the moment you have that, you've signed up for the CAP theorem's dilemma: what happens when those copies can't talk to each other? Active-passive avoids this problem entirely by only ever having one copy accept writes at a time.

Applied / Scenario#

37. During a network partition, a "add to cart" write succeeds on one Cassandra replica, but a "view cart" read on a different, partitioned-off replica shows an empty cart a moment later. Is this a bug?#

Not necessarily — it's the expected tradeoff of an AP system: Cassandra chose to keep answering (Availability) rather than refuse (Consistency) during the partition, so temporary disagreement between replicas is expected and usually resolves once the partition heals. Whether it's acceptable depends on the use case — tolerable for a shopping cart, but a "payment confirmed" check might warrant a stronger-consistency read specifically for that operation.

38. You have N=5 replicas, want fast reads, and are willing to accept slower writes for guaranteed strong consistency. What W and R would you pick?#

Keep R low for fast reads (R=1). To satisfy W + R > N (5), W must be at least 5 (5+1=6 > 5) — every write must be confirmed by all 5 replicas before succeeding, which is slower and less available for writes, but guarantees every fast, single-replica read always sees the latest data.

---

Quick-Fire / Rapid Recall#