Resilience Patterns

Table of Contents#

1. The Problem These Patterns All Solve
2. Timeouts — The Most Basic Protection
3. Retries — Trying Again, Carefully
4. Exponential Backoff and Jitter
5. The Retry Storm — When "Just Retry" Makes Things Worse
6. Circuit Breakers — Stop Hitting a Broken Thing
7. The Three States of a Circuit Breaker
8. Circuit Breakers in Practice
9. Rate Limiting and Throttling
10. Rate Limiting Algorithms
11. Bulkheads — Containing the Blast Radius
12. Graceful Degradation and Fallbacks
13. Load Shedding
14. Combining All the Patterns — A Full Defense-in-Depth Example
15. Common Mistakes
16. Worked Practice Problems
17. Summary and What's Next

---

The Problem These Patterns All Solve#

Part 1 was about making sure you stay up. This part is about something just as important: making sure that when someone else (a dependency you call) starts failing, their problem doesn't become your problem too.

A simple analogy: imagine you're waiting in line at a coffee shop, and the barista suddenly freezes up and stops serving anyone. If you just stand there waiting forever, you're stuck too — and so is everyone behind you in line. A smart customer gives up after a reasonable wait, and an even smarter shop puts up a sign ("we're out of espresso, cold brew only") so people don't even bother waiting for something that's broken.

That's the whole idea behind this tutorial: don't let a slow or broken dependency take you down with it.

Diagram

This spreading of failure from one broken service to its callers is called a cascading failure, and it's one of the most common causes of "small problem turned into a huge outage" in real production incidents.

---

Timeouts — The Most Basic Protection#

A timeout is simply: "I will not wait longer than X seconds for a response — if it takes longer, I'll give up and handle that as a failure."

Diagram

Why "No Timeout" Is a Real, Common Bug#

If you never set a timeout, your code will wait — literally forever, or until some very long default (some libraries default to no timeout at all, or an unreasonably long one like 5 minutes). If Service B is stuck, every request Service A sends to it also gets stuck, one by one, until Service A runs out of available threads/connections to handle anything else — including requests that had nothing to do with B.

Diagram

A very strong, simple interview line: "Every network call should have a timeout. No exceptions. An unbounded wait on one dependency can starve an entire service of resources and turn a small, contained problem into a full outage."

Choosing a Good Timeout Value#

• Too short: you give up on requests that would have succeeded if you'd just waited a little longer — wasted work, unnecessary retries.
• Too long: a broken dependency ties up your resources for a long time before you even notice something's wrong.
• Best practice: base it on real, measured latency data (e.g., your p99 latency to that dependency, plus a reasonable buffer) — not a guess.

---

Retries — Trying Again, Carefully#

A retry means: if a request fails, try it again, on the assumption the failure might have been temporary (a blip, not a real ongoing problem).

Diagram

Not All Failures Should Be Retried#

This is a genuinely important, often-missed nuance. Retrying only makes sense for failures that might resolve themselves — a network blip, a momentarily overloaded server. Retrying a request that failed because of bad input (like a malformed request, HTTP 400) just wastes time and resources, because the exact same broken request will fail the exact same way every single time.

The Idempotency Problem#

This is a favorite, sharp interview question: "if you retry a request, could it run twice?"

Imagine you send a "charge $50" request. It actually succeeds on the server, but the response gets lost on the way back to you (a network blip on the return trip). From your point of view, it looks like a failure — so you retry. Now the customer has been charged $100.

Diagram

The fix: idempotency keys. The client generates a unique ID for the intent ("charge this specific $50, request ID abc-123") and sends it along with the request. The server remembers which request IDs it has already processed — if it sees the same ID again, it returns the original result instead of charging a second time.

Diagram

Interview-ready summary: "Before adding retries to any call, I ask: is this operation idempotent — safe to run more than once with the same effect as running it once? If not (like a payment charge), I either make it idempotent using a client-generated idempotency key, or I don't retry it blindly."

---

Exponential Backoff and Jitter#

If a dependency is struggling, retrying immediately and repeatedly just piles more load onto an already-struggling system — potentially making things worse, not better.

Diagram

Exponential Backoff#

Instead of retrying instantly, wait a little longer each time: 1 second, then 2, then 4, then 8, and so on — giving the struggling dependency room to recover.

Diagram

Jitter — Why "Exponential Backoff" Alone Isn't Enough#

Here's a subtle problem: if 1,000 clients all failed at the exact same moment (e.g., because the server just crashed), and they all use the exact same backoff schedule (1s, 2s, 4s...), they'll all retry again at the exact same instant — recreating the exact overload they were trying to avoid, just delayed by a few seconds.

Diagram

The fix: add randomness ("jitter") to the wait time. Instead of "wait exactly 1 second," each client waits "1 second, plus or minus a random amount." This spreads the retries out over time instead of bunching them all together.

Diagram

Interview-ready one-liner: "Exponential backoff spreads retries out over time; jitter spreads them out within each time window, so a thousand clients don't all retry at the exact same millisecond." This combination — "exponential backoff with jitter" — is the standard, correct answer whenever retries come up in an interview; AWS has a well-known engineering blog post specifically titled "Exponential Backoff and Jitter" that's worth knowing exists.

---

The Retry Storm — When "Just Retry" Makes Things Worse#

Putting the above pieces together, here's the full failure mode retries can cause if done naively — a genuinely important pattern to be able to explain end-to-end.

Diagram

This is called a retry storm, and it's a real, well-documented cause of major outages (several publicly documented AWS and other cloud provider outages have named uncontrolled retry behavior as a contributing factor). The fix requires all of: reasonable timeouts, exponential backoff with jitter, a cap on the total number of retries, and — critically — circuit breakers (next section), which stop the retries entirely once it's clear the dependency is actually down rather than just briefly blipping.

---

Circuit Breakers — Stop Hitting a Broken Thing#

A circuit breaker is directly borrowed from electrical engineering: when a house's wiring gets an unsafe surge of current, the circuit breaker physically "trips," cutting the circuit before it can start a fire — and someone has to manually (or automatically, after a cooldown) reset it before power flows again.

The software version does the same thing: if calls to a dependency keep failing, the circuit breaker "trips" and stops even attempting new calls for a while — failing instantly instead of wasting time/resources on calls that are very likely to fail anyway.

Diagram

---

The Three States of a Circuit Breaker#

This state machine is one of the most commonly diagrammed concepts in SRE/backend interviews — know it cold.

Diagram

Why "Half-Open" exists, specifically: without it, you'd have only two choices — stay fully broken forever (bad), or snap immediately back to fully "Closed" and unleash a full flood of pent-up traffic onto a dependency that might have just barely recovered (also bad, could immediately re-trip it). Half-Open lets the system test the waters cautiously with a small trickle of traffic before fully trusting the dependency again.

---

Circuit Breakers in Practice#

A worked timeline showing the full lifecycle:

Diagram

Key Configuration Parameters (Real Interview Detail)#

Common real-world libraries worth knowing by name: Hystrix (Netflix's now-legacy but historically very influential circuit breaker library), resilience4j (its modern Java successor), Polly (.NET). Even if you've never used one directly, knowing these names signals real familiarity with the ecosystem.

---

Rate Limiting and Throttling#

Circuit breakers protect you from a struggling dependency. Rate limiting is the mirror image: protecting your own service from being overwhelmed by too many incoming requests — whether from a misbehaving client, a traffic spike, or an actual attack.

Diagram

"Throttling" is often used interchangeably with rate limiting, though it sometimes specifically implies slowing down rather than outright rejecting (e.g., deliberately delaying responses instead of returning an error).

---

Rate Limiting Algorithms#

Diagram

Token Bucket#

Imagine a bucket that holds tokens. Tokens are added at a steady rate (e.g., 10/second) up to a max capacity. Every request consumes one token; if the bucket is empty, the request is rejected.

Diagram

Why it's popular: it naturally allows brief bursts (if the bucket has accumulated tokens from a quiet period, a sudden flurry of requests can be handled all at once, up to the bucket's capacity) while still enforcing a steady average rate over time. This burst-tolerance is often exactly what you want — a client that's mostly quiet but occasionally sends a batch of requests shouldn't be punished the same way as one that's constantly hammering the limit.

Leaky Bucket#

Similar idea, but requests are processed at a strictly constant, steady rate — like a bucket with a small hole in the bottom that leaks at a fixed rate, no matter how fast water is poured in. This smooths bursts into a constant output rate rather than allowing them through.

Fixed Window Counter#

Count requests in fixed time windows (e.g., "max 100 requests per minute, reset at the top of each minute").

Diagram

The known weakness: a client can send 100 requests at 12:00:59 and another 100 at 12:01:00 — 200 requests in 2 seconds, technically obeying the "100 per window" rule but wildly bursting right across the window boundary.

Diagram

Sliding Window Log / Counter#

Fixes the boundary problem by looking at a continuously moving window (e.g., "the last 60 seconds from right now," not "this calendar minute") — much smoother, but more expensive to compute exactly (a "sliding window log" tracks every request's exact timestamp; a "sliding window counter" approximates it more cheaply by blending the current and previous fixed windows proportionally).

Interview tip: "Token bucket" is the single most commonly expected answer when asked "how would you implement rate limiting" — know it well enough to describe and roughly sketch, since it's the most frequently asked of this whole family.

---

Bulkheads — Containing the Blast Radius#

Named after ship design: a ship's hull is divided into separate watertight compartments (bulkheads) so that if one section floods, the whole ship doesn't sink — just that one compartment fills with water while the rest stays dry and afloat.

Diagram

Applied to software: instead of one shared pool of resources (threads, connections) for calls to every dependency, give each dependency its own isolated pool. If Dependency X hangs and exhausts its pool, only calls to X are affected — calls to Y and Z, using their own separate pools, keep working normally.

A simple, memorable interview line: "Bulkheads mean isolating resources per-dependency, so one hung dependency can only sink its own compartment, not the whole ship."

---

Graceful Degradation and Fallbacks#

Sometimes the right answer isn't "fail" — it's "give the user something slightly less good, instead of nothing at all."

Diagram

Real-world examples worth citing:

• Netflix famously falls back to a generic, non-personalized list of popular titles if its personalization/recommendation engine is unavailable — the homepage still works, just less tailored.
• An e-commerce site might disable "customers also bought" widgets during a database slowdown, prioritizing letting people still check out.
• A search feature might fall back from "smart, typo-tolerant search" to "basic exact-match search" if the fancy search index is temporarily unavailable.

The core principle: identify which parts of a page/flow are essential (checkout, login) versus nice-to-have (recommendations, "related items," non-critical widgets), and design so that a failure in a nice-to-have component degrades that one piece gracefully instead of taking down the entire page/flow.

---

Load Shedding#

When a system is genuinely overwhelmed — more traffic than it can possibly serve well — sometimes the healthiest thing to do is deliberately refuse some requests, so the ones that do get through can be served properly, instead of trying to serve everyone badly (or crashing entirely).

Diagram

Analogy: a restaurant that's completely full can either cram in more tables until service for everyone becomes terrible, or it can tell new arrivals "sorry, we're full, try back in 20 minutes" — keeping the experience good for everyone already seated. Load shedding is the software version of that second choice.

Load shedding is often implemented by prioritizing which requests to keep — e.g., dropping low-priority background/batch traffic first while continuing to serve real user-facing requests, or dropping requests from clients that have already exceeded their fair share (tying back to rate limiting).

---

Combining All the Patterns — A Full Defense-in-Depth Example#

A realistic view of how all these patterns stack together in one real call path:

Diagram

A genuinely strong interview answer to "how would you make a call to an external payment API resilient" walks through this exact stack, in this exact order, explaining what each layer protects against — that level of structured, layered thinking is what separates a senior answer from someone who just says "add retries."

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A payment service starts timing out intermittently. The calling service retries every failed request up to 3 times, with no backoff, no jitter, and no circuit breaker. Twenty minutes later, the payment service is fully down, and a postmortem needs a root-cause explanation. What happened?

Answer: Classic retry storm. The payment service was likely just moderately overloaded at first (causing intermittent timeouts), but the immediate, backoff-free retries added extra load on top of the original load precisely when the payment service could least handle it — driving it from "struggling" to "fully down." Without a circuit breaker, the calling service kept hammering it the whole way down instead of backing off once it became clear something was seriously wrong. Fix: add exponential backoff with jitter to the retries, cap the total retry attempts, and add a circuit breaker so repeated failures stop generating new load entirely once a clear pattern of failure is detected.

Problem 2: You're designing a rate limiter for a public API and want to allow occasional legitimate bursts (e.g., a client that batches its work and sends 50 requests at once every few minutes) without allowing sustained abuse. Which algorithm fits best, and why?

Answer: Token bucket — it naturally accumulates tokens during quiet periods (up to the bucket's capacity) and allows a burst to consume them all at once, while still enforcing a steady average rate over time by refilling at a fixed rate. A strict leaky bucket or fixed sliding window would smooth out or reject exactly the legitimate bursty pattern this use case needs to allow.

Problem 3: A service calls three downstream dependencies (Inventory, Pricing, Shipping) using one shared connection pool of 50 connections. Inventory starts hanging. What happens to Pricing and Shipping calls, and how would bulkheads fix it?

Answer: Without bulkheads, all 50 connections can eventually get consumed by hung calls to Inventory, leaving zero connections available for Pricing or Shipping calls — even though those two dependencies are completely healthy. The whole service effectively goes down because of one bad dependency. With bulkheads, each dependency gets its own dedicated slice of the pool (e.g., 20/15/15) — Inventory hanging can exhaust its own 20-connection allocation, but Pricing and Shipping keep working normally using their separate, untouched allocations.

---

Summary and What's Next#

• These patterns exist to stop a failure in one dependency from cascading into a failure of the whole system.
• Timeouts are non-negotiable on every network call — an unbounded wait on a hung dependency can exhaust your entire service's resources.
• Retries should only apply to failures that are actually likely to be transient, and only to operations that are idempotent (or made idempotent via idempotency keys) — otherwise you risk double-effects like duplicate charges.
• Exponential backoff with jitter is the standard, correct pattern for retry timing — backoff spreads retries over time, jitter prevents many clients from retrying in perfect, overload-recreating unison.
• Circuit breakers (Closed → Open → Half-Open) stop calling a dependency once it's clearly broken, failing fast instead of wasting resources, and cautiously test recovery before fully trusting it again.
• Rate limiting (commonly via token bucket) protects your own service from being overwhelmed by too much incoming traffic.
• Bulkheads isolate resources per-dependency so one hung dependency can't exhaust resources needed for calls to healthy dependencies.
• Graceful degradation serves a reduced but working experience when a non-essential dependency fails; load shedding deliberately rejects some traffic when genuinely overloaded, so the requests that do get through are served well.
• A resilient call path typically layers several of these patterns together — rate limiting, bulkheads, circuit breakers, timeouts, and careful retries, in that order.

Continue to Part 3 (03-cap-theorem-and-consistency.md) to understand the deeper theoretical tradeoffs behind why some of these patterns (especially around active-active databases and multi-region systems) are hard in the first place — the CAP theorem and consistency models.