# Disaster Recovery & Business Continuity — Part 2: DR Testing & Runbooks > **Series:** Disaster Recovery & Business Continuity (2 of 3) > **Part 1:** `01-dr-fundamentals-and-strategies.md` — DR Fundamentals & Strategies > **Part 2:** This file — DR Testing & Runbooks > **Part 3:** `03-real-world-disasters-and-case-studies.md` — Real-World Disasters & Case Studies > **Questions:** `questions.md` ## Table of Contents 1. [A Plan You've Never Tested Isn't a Plan](#a-plan-youve-never-tested-isnt-a-plan) 2. [The DR Testing Maturity Ladder](#the-dr-testing-maturity-ladder) 3. [Level 1: Tabletop Exercises](#level-1-tabletop-exercises) 4. [Level 2: Walkthrough / Simulation Tests](#level-2-walkthrough--simulation-tests) 5. [Level 3: Partial (Component) Failover Tests](#level-3-partial-component-failover-tests) 6. [Level 4: Full Failover Tests](#level-4-full-failover-tests) 7. [Level 5: Chaos Engineering and Unannounced Drills](#level-5-chaos-engineering-and-unannounced-drills) 8. [How Often to Actually Test](#how-often-to-actually-test) 9. [Writing a Real DR Runbook](#writing-a-real-dr-runbook) 10. [A Full Worked DR Runbook](#a-full-worked-dr-runbook) 11. [The Communication Plan — Often Forgotten](#the-communication-plan--often-forgotten) 12. [The Dependency Mapping Problem](#the-dependency-mapping-problem) 13. [People Are Part of the System Too](#people-are-part-of-the-system-too) 14. [Common Mistakes](#common-mistakes) 15. [Worked Practice Problems](#worked-practice-problems) 16. [Summary and What's Next](#summary-and-whats-next) --- ## A Plan You've Never Tested Isn't a Plan This exact principle has already been established, twice, elsewhere in this course — for etcd backups (Kubernetes Deep Dive series) and for database backups (Databases & Storage Reliability series). Here, it becomes the entire organizing theme of this Part: **a DR strategy that exists only as a document, never actually exercised, is a hypothesis about what would happen in a disaster — not a verified, trustworthy capability.** ```mermaid graph TD Untested["An UNTESTED DR plan"] --> Risk["Real, documented failures:
a runbook step referencing
a tool that was
decommissioned years ago,
a critical person who left
the company, a dependency
nobody remembered existed
— ALL discovered for the
FIRST time during an
actual disaster"] Tested["A REGULARLY TESTED DR
plan"] --> Confidence["A GENUINELY verified,
trustworthy capability —
gaps found and fixed
SAFELY, on a controlled
schedule, not during a
real crisis"] ``` --- ## The DR Testing Maturity Ladder A genuinely useful, practical framework — testing doesn't have to jump straight to "actually fail over production" on day one; it's a deliberate progression, each level building real confidence and catching different classes of gaps. ```mermaid graph TD L1["Level 1: Tabletop
Exercise (talk through
it, no systems touched)"] --> L2 L2["Level 2: Walkthrough/
Simulation (execute
SOME real steps, in a
non-production
environment)"] --> L3 L3["Level 3: Partial
(Component) Failover
Test (fail over ONE real
piece, in production,
during a controlled
window)"] --> L4 L4["Level 4: Full Failover
Test (fail over the
ENTIRE system to the
DR environment, for real)"] --> L5 L5["Level 5: Chaos
Engineering / Unannounced
Drills (inject REAL
failures with NO advance
warning to responders)"] ``` **A strong, senior-level interview framing worth stating explicitly: "I wouldn't recommend jumping straight to unannounced production failover drills for a team that's never tested DR before — that's how you turn a learning exercise into an actual incident. I'd climb this ladder deliberately, building real confidence and fixing the gaps each level surfaces, before attempting the next, higher-risk level."** --- ## Level 1: Tabletop Exercises The cheapest, lowest-risk, and — genuinely worth stating explicitly — most commonly skipped level of testing, despite being extremely valuable. ```mermaid sequenceDiagram participant Facilitator participant Team as Response Team Facilitator->>Team: "It's 3am. The primary
database region is
completely unreachable.
Walk me through EXACTLY
what you'd do, step by
step, right now." Team->>Team: Talks through the ACTUAL
runbook, out loud,
identifying gaps as they go Facilitator->>Team: "Who has the authority
to approve the failover?" Team->>Team: "...actually, we're not
sure — that's never
been explicitly defined" ``` **Why a tabletop exercise is so genuinely valuable despite touching zero real systems, worth stating explicitly: it surfaces PROCESS and OWNERSHIP gaps — "who has authority to make this call," "does everyone actually know where the runbook lives," "does the runbook reference a tool nobody uses anymore" — cheaply and quickly, often revealing serious gaps in minutes that would otherwise only surface during a real, high-stakes crisis.** This directly extends the exact same value proposition as the GameDay exercises from the Incident Management series, at an even lower-cost, more frequent cadence. --- ## Level 2: Walkthrough / Simulation Tests One step more rigorous: actually execute real steps of the DR plan, but in a non-production, isolated environment — genuinely doing the work, not just talking about it, but without any real production risk. ```mermaid graph TD Sim["In a STAGING/isolated
environment: actually
RUN the restore-from-
backup commands, actually
provision the DR
infrastructure via IaC
(Automation series)"] --> Value["Catches REAL, technical
gaps a tabletop's
conversation alone can't —
a broken script, an
outdated Terraform module,
a permission that was
never actually granted"] ``` --- ## Level 3: Partial (Component) Failover Tests Failing over one real, specific, bounded piece of the actual production system — during a controlled, planned window — rather than the whole thing at once. ```mermaid graph TD Example["Example: intentionally
fail over JUST the read-
replica database (Databases
series, Part 1) to confirm
the promotion process
ACTUALLY works, without
touching the primary or
any application traffic"] --> Value["Real, genuine production
validation, with a
DELIBERATELY bounded,
limited blast radius"] ``` **Why this level is such a valuable, practical middle ground, worth stating explicitly: it validates the ACTUAL, real mechanics (not a simulation) of a specific, bounded piece of the DR plan, without the cost, coordination overhead, and real risk of a full production failover — a genuinely common, practical choice for teams building toward full DR test maturity.** --- ## Level 4: Full Failover Tests The real thing: actually failing the entire production system over to the DR environment, genuinely serving real traffic from it, for a real period of time. ```mermaid sequenceDiagram participant Primary as Primary Region participant DR as DR Region participant Traffic as Real Production Traffic Note over Primary,DR: Planned, announced,
fully staffed window Traffic->>Primary: Normal operation Primary->>DR: Execute the ACTUAL
failover runbook, for real Traffic->>DR: Traffic is REDIRECTED
to the DR environment Note over Primary,DR: DR environment runs
REAL production traffic
for a defined period DR->>Primary: Fail BACK to primary
(also tested — failing
BACK is its own real,
separate operation) ``` **A genuinely important, often-overlooked detail worth stating explicitly: testing the failover is only half the job — testing the FAILBACK (returning to the original primary once it's healthy again) is a real, separate, and sometimes even more complex operation (especially reconciling any data written to the DR environment while it was live) that deserves its own explicit test, not just an assumption that "reversing the process" will obviously work.** --- ## Level 5: Chaos Engineering and Unannounced Drills The most advanced, most realistic level — directly extending the chaos engineering discussion from the Incident Management series, applied specifically at the DR/regional-failure scale (this is essentially Google's DiRT program, already referenced in that earlier tutorial, now fully explained in its DR-specific context). ```mermaid graph TD Unannounced["An UNANNOUNCED drill: the
response team gets NO
advance warning that
THIS specific incident is
actually a planned test"] --> Value["Tests something even a
full, ANNOUNCED failover
test can't: genuine,
REALISTIC human response
under real uncertainty —
not a rehearsed, prepared
performance"] ``` **A genuinely important safety caveat worth stating explicitly, and a real, practical constraint: this level requires a mature organization with strong safeguards (a clear abort mechanism, senior oversight, tight scoping) — attempting unannounced production drills before an organization has built real confidence through the earlier levels risks turning a learning exercise into an actual, uncontrolled incident, exactly the same caution already emphasized for chaos engineering generally in the Incident Management series.** --- ## How Often to Actually Test A genuinely practical, often-asked question — there's no single universal answer, but a reasonable, defensible framework worth having ready. ```mermaid graph TD Cadence["Testing Cadence,
by Level"] --> C1["Tabletop exercises:
quarterly — cheap enough
to do often"] Cadence --> C2["Partial failover tests:
quarterly to semi-annually"] Cadence --> C3["Full failover tests:
annually, at minimum —
more often for the MOST
critical systems"] Cadence --> C4["ALSO: re-test after ANY
significant architecture
change — a DR plan tested
against last year's
architecture may not
reflect THIS year's reality
at all"] ``` **A genuinely important point worth stating explicitly: a DR plan's validity has a real shelf life — systems change, dependencies get added, people leave. A full failover test that passed cleanly a year ago provides very little real confidence today if the architecture has meaningfully changed since then, which is exactly why re-testing after significant changes matters as much as a fixed calendar cadence.** --- ## Writing a Real DR Runbook Directly extending the runbook discussion from the Incident Management series (Part 2) — here's what makes a DR runbook specifically effective, as opposed to a generic incident runbook. ```mermaid graph TD Runbook["A GOOD DR Runbook"] --> R1["EXACT commands, not
vague descriptions —
copy-pasteable, tested"] Runbook --> R2["Explicit DECISION
authority — WHO can
declare a disaster and
trigger the plan"] Runbook --> R3["Explicit DEPENDENCY
order — what needs to
come back FIRST (e.g.
DNS/auth BEFORE anything
that depends on them)"] Runbook --> R4["Explicit VERIFICATION
steps — how do you
CONFIRM each stage
actually succeeded before
moving to the next"] Runbook --> R5["Kept CURRENT — reviewed
and updated as part of
EVERY DR test, not left
to rot"] ``` --- ## A Full Worked DR Runbook A concrete, realistic example — genuinely worth having a shape like this ready to sketch in an interview. > **DR Runbook: Primary Region Failure — Checkout Platform** > > **Trigger Criteria** > - Primary region (us-east-1) health checks failing for >5 consecutive > minutes across multiple independent monitors > - OR: explicit declaration by the on-call Incident Commander > (Incident Management series, Part 2) after manual confirmation > > **Declaration Authority** > - Any Incident Commander may DECLARE the incident and begin > Steps 1-3 (assessment) unilaterally > - Full production failover (Step 4+) requires sign-off from the > Engineering Director OR a second Incident Commander, logged > in the incident channel > > **Step 1: Confirm This Is Real** > ```bash > # Check from MULTIPLE independent vantage points — rule out > # a monitoring-system-specific false positive > curl -o /dev/null -s -w "%{http_code}\n" https://api.example.com/health > dig api.example.com # confirm DNS itself is resolving correctly > ``` > > **Step 2: Assess Data Currency in the DR Region** > ```bash > # Check replication lag BEFORE failing over — know exactly > # how much data (if any) would be lost (Databases series, Part 1) > psql -h dr-region-db -c "SELECT now() - pg_last_xact_replay_timestamp();" > ``` > > **Step 3: Notify (Communication Plan, below)** > - Post to #incident-checkout-platform > - Page: Engineering Director, Checkout Team Lead > - Update the public status page: "Investigating elevated errors" > > **Step 4: Execute Failover (requires sign-off, per above)** > ```bash > # Promote the DR region's database replica > ./scripts/promote-dr-database.sh > > # Update DNS/global load balancer to route to DR region > ./scripts/failover-dns-to-dr.sh > ``` > > **Step 5: Verify** > ```bash > # Confirm real traffic is flowing to the DR region and succeeding > curl -o /dev/null -s -w "%{http_code}\n" https://api.example.com/health > # Check RED dashboard (Monitoring Methodologies series) for > # checkout-service specifically in the DR region > ``` > > **Step 6: Update Status Page and Stakeholders** — "Service restored, > operating from backup region" > > **Known Dependencies** (verify these ALSO failed over correctly) > - Auth service (must be healthy BEFORE checkout can authenticate users) > - Payment gateway webhook endpoint (external providers may have > the OLD region's IP cached — verify this separately) > > **Failback Procedure** — See `failback-runbook.md` — NOT simply > "reverse these steps." **Why this exact shape — trigger criteria, explicit declaration authority, numbered steps with real commands, explicit verification, known dependencies, and a SEPARATE failback procedure — is worth reproducing: it demonstrates genuine, practical DR runbook authorship, not just an abstract understanding that "you should have a runbook."** --- ## The Communication Plan — Often Forgotten A genuinely important, frequently under-planned piece — directly extending the communication discussion from the Incident Management series (Part 2), specifically for the DR/disaster scale. ```mermaid graph TD Comms["DR-Specific Communication
Needs"] --> C1["INTERNAL: does the
on-call/response team
have a communication
channel that DOESN'T
depend on the failed
infrastructure itself?
(a genuinely common,
embarrassing gap — e.g.
a Slack integration that
depends on the SAME
region that just failed)"] Comms --> C2["EXTERNAL: customers,
via a status page HOSTED
SEPARATELY from the
affected infrastructure"] Comms --> C3["REGULATORY/LEGAL: does
this specific disaster
trigger any mandatory
disclosure requirements
(e.g. a data breach
notification law)?"] Comms --> C4["EXECUTIVE: leadership
needs a clear, appropriately-
scoped update — not the
full technical detail, but
real, honest status"] ``` **A genuinely sharp, real-world gotcha worth stating explicitly: a status page or incident communication tool that's HOSTED IN THE SAME REGION/INFRASTRUCTURE that just failed is a real, embarrassing, and surprisingly common gap — exactly the same "your monitoring shouldn't depend on the thing it's monitoring" principle that applies to health checks, applied here to the communication tooling itself.** --- ## The Dependency Mapping Problem A genuinely important, often-underestimated part of real DR planning — worth its own dedicated callout. ```mermaid graph TD Service["Checkout Service"] --> Dep1["Auth Service"] Service --> Dep2["Payment Gateway
(EXTERNAL, third-party)"] Service --> Dep3["Inventory Service"] Dep1 --> Dep4["Identity Provider
(ALSO external —
does IT have its OWN
region failover?)"] ``` **Why this matters practically, worth stating explicitly: a DR plan that only considers the ONE service being failed over, without mapping its full dependency chain, risks a technically "successful" failover that STILL doesn't actually work — the checkout service might come up perfectly in the DR region, but if it depends on an internal Auth service that WASN'T also failed over, or an external identity provider that has its own, separate regional issue, the overall system is still broken, even though the specific runbook that was tested "succeeded."** A genuinely thorough DR plan maps and tests the FULL dependency chain, not just the headline service. --- ## People Are Part of the System Too A final, genuinely important, human-centered point worth closing this Part with — directly connecting to the on-call sustainability discussion from the Incident Management series. ```mermaid graph TD Question["'Do we have a strong
technical DR plan?'"] --> RealQuestion["The REAL question:
'Do the SPECIFIC PEOPLE
who'd need to execute this
plan at 3am ACTUALLY know
how, and are they
REACHABLE?'"] RealQuestion --> Risk["A plan that depends on
ONE specific person's
tribal knowledge — who
might be asleep,
unreachable, or has since
LEFT the company — is a
SINGLE POINT OF FAILURE,
exactly like an
un-redundant server
(Reliability & Architecture
Patterns series)"] ``` **A strong, senior-level interview line, tying together the whole series so far: "A DR plan's real robustness isn't just about redundant infrastructure — it's also about not having a single point of failure in the PEOPLE who know how to execute it. If only one engineer truly understands the failover runbook, that's exactly as fragile as having only one server with no redundancy — the fix is the same principle applied to knowledge instead of infrastructure: cross-train multiple people, and validate that through the same DR testing this whole Part covers."** --- ## Common Mistakes | Mistake | Why It's Wrong | Fix | |---|---|---| | Never testing a DR plan beyond writing the initial document | Real gaps (broken scripts, outdated dependencies, unclear ownership) only surface for the first time during an actual disaster | Climb the DR testing maturity ladder deliberately, starting cheap (tabletop) and building toward higher-fidelity tests | | Jumping straight to unannounced production drills for a team with no prior DR testing experience | Risks turning a learning exercise into an actual, uncontrolled incident | Build real confidence through lower-risk levels first, before attempting the highest-risk, most realistic tests | | Testing failover but never testing failback | Failing back is a genuinely separate, sometimes more complex operation (especially reconciling data written during the DR period) that can fail even when failover worked perfectly | Explicitly test the failback procedure as its own, separate exercise | | Hosting the incident communication/status page tooling in the same region/infrastructure being tested for failure | The exact tool needed to communicate about the disaster can itself become a casualty of it | Host communication tooling on genuinely separate, independent infrastructure | | Failing over only the headline service, without mapping and testing its full dependency chain | A technically "successful" failover of one service can still leave the overall system broken if a dependency wasn't also failed over | Map and test the complete dependency chain, not just the primary service being tested | | Relying on one specific person's tribal knowledge to execute a DR plan | A single point of failure in PEOPLE, exactly as fragile as an un-redundant server | Cross-train multiple people, and validate that through real DR testing | --- ## Worked Practice Problems **Problem 1:** A team has a well-written DR runbook that's never actually been executed, technically or through a tabletop exercise. A new engineering director asks "how confident are we in our DR readiness?" What would you actually say, and what would you recommend as the very next step? *Answer:* I'd be honest that current confidence should be LOW, regardless of how well-written the document itself is — an untested plan is a hypothesis, not a verified capability, and real, documented failures happen where a runbook step references a decommissioned tool, an unreachable person, or a dependency nobody remembered, none of which show up just from reading the document. My recommended next step would be starting at the cheapest, lowest-risk level of the testing maturity ladder — a tabletop exercise — specifically because it's fast, cheap, and reliably surfaces real process/ownership gaps (like unclear declaration authority) before investing in the more expensive, higher-risk levels of testing. **Problem 2:** During a full failover test, the checkout service comes up successfully in the DR region, but customers still can't complete purchases. Investigation reveals the internal Auth service was never included in the failover plan and is still only running in the (now-simulated-as-down) primary region. What does this reveal about the DR plan's real gap, and how would you fix it? *Answer:* This reveals the dependency mapping problem directly — the DR plan only considered the headline service (checkout) in isolation, without mapping and testing its full dependency chain, including internal services it depends on (Auth) and any of ITS OWN dependencies in turn. A technically "successful" failover of the one service tested doesn't mean the overall system actually works if a critical dependency wasn't also failed over. The fix: explicitly map every dependency (direct and transitive) for each critical service as part of DR planning, and ensure the failover runbook and testing scope covers the FULL chain, not just the one service with the most visible customer impact. **Problem 3:** A company's status page, used to communicate incident updates to customers, is hosted on the same cloud infrastructure and in the same region as their primary production systems. During a genuine regional outage, the team discovers they can't update the status page at all. What's the lesson, and how does it connect to a principle already established elsewhere in this course? *Answer:* This is exactly the same "your monitoring/communication tooling shouldn't depend on the thing it's monitoring/communicating about" principle already established for health checks and monitoring generally — if the tool meant to communicate about an outage is itself a casualty of that same outage, it fails at the exact moment it's needed most. The fix: host status pages and critical incident-communication tooling on genuinely separate, independent infrastructure (many teams deliberately use a third-party-hosted status page service specifically for this reason) — ensuring the ability to communicate survives even a complete failure of the company's own primary infrastructure. --- ## Summary and What's Next - **An untested DR plan is a hypothesis, not a verified capability** — the same principle already established for etcd and database backups elsewhere in this course, now the central theme of this entire Part. - The **DR testing maturity ladder** — tabletop exercises, walkthroughs, partial failover tests, full failover tests, and unannounced chaos-style drills — is a deliberate progression; don't jump to the highest-risk level without building real confidence first. - **Testing failback is just as important as testing failover** — returning to the original primary is a genuinely separate, sometimes more complex operation, especially reconciling data written during the DR period. - A strong **DR runbook** has explicit trigger criteria, explicit declaration authority, exact copy-pasteable commands, explicit verification steps, and is kept current through regular testing. - The **communication plan** is a real, often-forgotten piece of DR — and hosting status/incident-communication tooling on the SAME infrastructure being tested for failure is a genuine, common, embarrassing gap. - **Dependency mapping** matters — a "successful" failover of one service means nothing if a critical dependency (like an internal Auth service) wasn't also included in the plan. - **People are part of the system too** — a DR plan relying on one specific person's tribal knowledge is exactly as fragile as an un-redundant server, and needs the same fix: cross-training, validated through real testing. **Continue to Part 3** (`03-real-world-disasters-and-case-studies.md`) to ground everything in this series in real, documented industry disasters — what actually happened, what the root causes were, and the concrete, lasting lessons the industry took from them.