DR Testing & Runbooks

Table of Contents#

1. A Plan You've Never Tested Isn't a Plan
2. The DR Testing Maturity Ladder
3. Level 1: Tabletop Exercises
4. Level 2: Walkthrough / Simulation Tests
5. Level 3: Partial (Component) Failover Tests
6. Level 4: Full Failover Tests
7. Level 5: Chaos Engineering and Unannounced Drills
8. How Often to Actually Test
9. Writing a Real DR Runbook
10. A Full Worked DR Runbook
11. The Communication Plan — Often Forgotten
12. The Dependency Mapping Problem
13. People Are Part of the System Too
14. Common Mistakes
15. Worked Practice Problems
16. Summary and What's Next

---

A Plan You've Never Tested Isn't a Plan#

This exact principle has already been established, twice, elsewhere in this course — for etcd backups (Kubernetes Deep Dive series) and for database backups (Databases & Storage Reliability series). Here, it becomes the entire organizing theme of this Part: a DR strategy that exists only as a document, never actually exercised, is a hypothesis about what would happen in a disaster — not a verified, trustworthy capability.

Diagram

---

The DR Testing Maturity Ladder#

A genuinely useful, practical framework — testing doesn't have to jump straight to "actually fail over production" on day one; it's a deliberate progression, each level building real confidence and catching different classes of gaps.

Diagram

A strong, senior-level interview framing worth stating explicitly: "I wouldn't recommend jumping straight to unannounced production failover drills for a team that's never tested DR before — that's how you turn a learning exercise into an actual incident. I'd climb this ladder deliberately, building real confidence and fixing the gaps each level surfaces, before attempting the next, higher-risk level."

---

Level 1: Tabletop Exercises#

The cheapest, lowest-risk, and — genuinely worth stating explicitly — most commonly skipped level of testing, despite being extremely valuable.

Diagram

Why a tabletop exercise is so genuinely valuable despite touching zero real systems, worth stating explicitly: it surfaces PROCESS and OWNERSHIP gaps — "who has authority to make this call," "does everyone actually know where the runbook lives," "does the runbook reference a tool nobody uses anymore" — cheaply and quickly, often revealing serious gaps in minutes that would otherwise only surface during a real, high-stakes crisis. This directly extends the exact same value proposition as the GameDay exercises from the Incident Management series, at an even lower-cost, more frequent cadence.

---

Level 2: Walkthrough / Simulation Tests#

One step more rigorous: actually execute real steps of the DR plan, but in a non-production, isolated environment — genuinely doing the work, not just talking about it, but without any real production risk.

Diagram

---

Level 3: Partial (Component) Failover Tests#

Failing over one real, specific, bounded piece of the actual production system — during a controlled, planned window — rather than the whole thing at once.

Diagram

Why this level is such a valuable, practical middle ground, worth stating explicitly: it validates the ACTUAL, real mechanics (not a simulation) of a specific, bounded piece of the DR plan, without the cost, coordination overhead, and real risk of a full production failover — a genuinely common, practical choice for teams building toward full DR test maturity.

---

Level 4: Full Failover Tests#

The real thing: actually failing the entire production system over to the DR environment, genuinely serving real traffic from it, for a real period of time.

Diagram

A genuinely important, often-overlooked detail worth stating explicitly: testing the failover is only half the job — testing the FAILBACK (returning to the original primary once it's healthy again) is a real, separate, and sometimes even more complex operation (especially reconciling any data written to the DR environment while it was live) that deserves its own explicit test, not just an assumption that "reversing the process" will obviously work.

---

Level 5: Chaos Engineering and Unannounced Drills#

The most advanced, most realistic level — directly extending the chaos engineering discussion from the Incident Management series, applied specifically at the DR/regional-failure scale (this is essentially Google's DiRT program, already referenced in that earlier tutorial, now fully explained in its DR-specific context).

Diagram

A genuinely important safety caveat worth stating explicitly, and a real, practical constraint: this level requires a mature organization with strong safeguards (a clear abort mechanism, senior oversight, tight scoping) — attempting unannounced production drills before an organization has built real confidence through the earlier levels risks turning a learning exercise into an actual, uncontrolled incident, exactly the same caution already emphasized for chaos engineering generally in the Incident Management series.

---

How Often to Actually Test#

A genuinely practical, often-asked question — there's no single universal answer, but a reasonable, defensible framework worth having ready.

Diagram

A genuinely important point worth stating explicitly: a DR plan's validity has a real shelf life — systems change, dependencies get added, people leave. A full failover test that passed cleanly a year ago provides very little real confidence today if the architecture has meaningfully changed since then, which is exactly why re-testing after significant changes matters as much as a fixed calendar cadence.

---

Writing a Real DR Runbook#

Directly extending the runbook discussion from the Incident Management series (Part 2) — here's what makes a DR runbook specifically effective, as opposed to a generic incident runbook.

Diagram

---

A Full Worked DR Runbook#

A concrete, realistic example — genuinely worth having a shape like this ready to sketch in an interview.

DR Runbook: Primary Region Failure — Checkout Platform

Trigger Criteria

• Primary region (us-east-1) health checks failing for >5 consecutive minutes across multiple independent monitors
• OR: explicit declaration by the on-call Incident Commander (Incident Management series, Part 2) after manual confirmation

Declaration Authority

• Any Incident Commander may DECLARE the incident and begin Steps 1-3 (assessment) unilaterally
• Full production failover (Step 4+) requires sign-off from the Engineering Director OR a second Incident Commander, logged in the incident channel

Step 1: Confirm This Is Real

Copy
# Check from MULTIPLE independent vantage points — rule out
# a monitoring-system-specific false positive
curl -o /dev/null -s -w "%{http_code}\n" https://api.example.com/health
dig api.example.com  # confirm DNS itself is resolving correctly

Step 2: Assess Data Currency in the DR Region

Copy
# Check replication lag BEFORE failing over — know exactly
# how much data (if any) would be lost (Databases series, Part 1)
psql -h dr-region-db -c "SELECT now() - pg_last_xact_replay_timestamp();"

Step 3: Notify (Communication Plan, below)

• Post to #incident-checkout-platform
• Page: Engineering Director, Checkout Team Lead
• Update the public status page: "Investigating elevated errors"

Step 4: Execute Failover (requires sign-off, per above)

Copy
# Promote the DR region's database replica
./scripts/promote-dr-database.sh

# Update DNS/global load balancer to route to DR region
./scripts/failover-dns-to-dr.sh

Step 5: Verify

Copy
# Confirm real traffic is flowing to the DR region and succeeding
curl -o /dev/null -s -w "%{http_code}\n" https://api.example.com/health
# Check RED dashboard (Monitoring Methodologies series) for
# checkout-service specifically in the DR region

Step 6: Update Status Page and Stakeholders — "Service restored, operating from backup region"

Known Dependencies (verify these ALSO failed over correctly)

• Auth service (must be healthy BEFORE checkout can authenticate users)
• Payment gateway webhook endpoint (external providers may have the OLD region's IP cached — verify this separately)

Failback Procedure — See failback-runbook.md — NOT simply "reverse these steps."

Why this exact shape — trigger criteria, explicit declaration authority, numbered steps with real commands, explicit verification, known dependencies, and a SEPARATE failback procedure — is worth reproducing: it demonstrates genuine, practical DR runbook authorship, not just an abstract understanding that "you should have a runbook."

---

The Communication Plan — Often Forgotten#

A genuinely important, frequently under-planned piece — directly extending the communication discussion from the Incident Management series (Part 2), specifically for the DR/disaster scale.

Diagram

A genuinely sharp, real-world gotcha worth stating explicitly: a status page or incident communication tool that's HOSTED IN THE SAME REGION/INFRASTRUCTURE that just failed is a real, embarrassing, and surprisingly common gap — exactly the same "your monitoring shouldn't depend on the thing it's monitoring" principle that applies to health checks, applied here to the communication tooling itself.

---

The Dependency Mapping Problem#

A genuinely important, often-underestimated part of real DR planning — worth its own dedicated callout.

Diagram

Why this matters practically, worth stating explicitly: a DR plan that only considers the ONE service being failed over, without mapping its full dependency chain, risks a technically "successful" failover that STILL doesn't actually work — the checkout service might come up perfectly in the DR region, but if it depends on an internal Auth service that WASN'T also failed over, or an external identity provider that has its own, separate regional issue, the overall system is still broken, even though the specific runbook that was tested "succeeded." A genuinely thorough DR plan maps and tests the FULL dependency chain, not just the headline service.

---

People Are Part of the System Too#

A final, genuinely important, human-centered point worth closing this Part with — directly connecting to the on-call sustainability discussion from the Incident Management series.

Diagram

A strong, senior-level interview line, tying together the whole series so far: "A DR plan's real robustness isn't just about redundant infrastructure — it's also about not having a single point of failure in the PEOPLE who know how to execute it. If only one engineer truly understands the failover runbook, that's exactly as fragile as having only one server with no redundancy — the fix is the same principle applied to knowledge instead of infrastructure: cross-train multiple people, and validate that through the same DR testing this whole Part covers."

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A team has a well-written DR runbook that's never actually been executed, technically or through a tabletop exercise. A new engineering director asks "how confident are we in our DR readiness?" What would you actually say, and what would you recommend as the very next step?

Answer: I'd be honest that current confidence should be LOW, regardless of how well-written the document itself is — an untested plan is a hypothesis, not a verified capability, and real, documented failures happen where a runbook step references a decommissioned tool, an unreachable person, or a dependency nobody remembered, none of which show up just from reading the document. My recommended next step would be starting at the cheapest, lowest-risk level of the testing maturity ladder — a tabletop exercise — specifically because it's fast, cheap, and reliably surfaces real process/ownership gaps (like unclear declaration authority) before investing in the more expensive, higher-risk levels of testing.

Problem 2: During a full failover test, the checkout service comes up successfully in the DR region, but customers still can't complete purchases. Investigation reveals the internal Auth service was never included in the failover plan and is still only running in the (now-simulated-as-down) primary region. What does this reveal about the DR plan's real gap, and how would you fix it?

Answer: This reveals the dependency mapping problem directly — the DR plan only considered the headline service (checkout) in isolation, without mapping and testing its full dependency chain, including internal services it depends on (Auth) and any of ITS OWN dependencies in turn. A technically "successful" failover of the one service tested doesn't mean the overall system actually works if a critical dependency wasn't also failed over. The fix: explicitly map every dependency (direct and transitive) for each critical service as part of DR planning, and ensure the failover runbook and testing scope covers the FULL chain, not just the one service with the most visible customer impact.

Problem 3: A company's status page, used to communicate incident updates to customers, is hosted on the same cloud infrastructure and in the same region as their primary production systems. During a genuine regional outage, the team discovers they can't update the status page at all. What's the lesson, and how does it connect to a principle already established elsewhere in this course?

Answer: This is exactly the same "your monitoring/communication tooling shouldn't depend on the thing it's monitoring/communicating about" principle already established for health checks and monitoring generally — if the tool meant to communicate about an outage is itself a casualty of that same outage, it fails at the exact moment it's needed most. The fix: host status pages and critical incident-communication tooling on genuinely separate, independent infrastructure (many teams deliberately use a third-party-hosted status page service specifically for this reason) — ensuring the ability to communicate survives even a complete failure of the company's own primary infrastructure.

---

Summary and What's Next#

• An untested DR plan is a hypothesis, not a verified capability — the same principle already established for etcd and database backups elsewhere in this course, now the central theme of this entire Part.
• The DR testing maturity ladder — tabletop exercises, walkthroughs, partial failover tests, full failover tests, and unannounced chaos-style drills — is a deliberate progression; don't jump to the highest-risk level without building real confidence first.
• Testing failback is just as important as testing failover — returning to the original primary is a genuinely separate, sometimes more complex operation, especially reconciling data written during the DR period.
• A strong DR runbook has explicit trigger criteria, explicit declaration authority, exact copy-pasteable commands, explicit verification steps, and is kept current through regular testing.
• The communication plan is a real, often-forgotten piece of DR — and hosting status/incident-communication tooling on the SAME infrastructure being tested for failure is a genuine, common, embarrassing gap.
• Dependency mapping matters — a "successful" failover of one service means nothing if a critical dependency (like an internal Auth service) wasn't also included in the plan.
• People are part of the system too — a DR plan relying on one specific person's tribal knowledge is exactly as fragile as an un-redundant server, and needs the same fix: cross-training, validated through real testing.

Continue to Part 3 (03-real-world-disasters-and-case-studies.md) to ground everything in this series in real, documented industry disasters — what actually happened, what the root causes were, and the concrete, lasting lessons the industry took from them.