# Disaster Recovery & Business Continuity — Part 1: DR Fundamentals & Strategies > **Series:** Disaster Recovery & Business Continuity (1 of 3) > **Part 1:** This file — DR Fundamentals & Strategies > **Part 2:** `02-dr-testing-and-runbooks.md` — DR Testing & Runbooks > **Part 3:** `03-real-world-disasters-and-case-studies.md` — Real-World Disasters & Case Studies > **Questions:** `questions.md` ## Table of Contents 1. [Why This Is the Capstone Topic of the Whole Course](#why-this-is-the-capstone-topic-of-the-whole-course) 2. [What Counts as a "Disaster"](#what-counts-as-a-disaster) 3. [Disaster Recovery vs Business Continuity vs High Availability](#disaster-recovery-vs-business-continuity-vs-high-availability) 4. [RPO and RTO, Fully Explained](#rpo-and-rto-fully-explained) 5. [The RPO/RTO Cost Curve](#the-rporto-cost-curve) 6. [The Four Classic DR Strategies](#the-four-classic-dr-strategies) 7. [Strategy 1: Backup and Restore](#strategy-1-backup-and-restore) 8. [Strategy 2: Pilot Light](#strategy-2-pilot-light) 9. [Strategy 3: Warm Standby](#strategy-3-warm-standby) 10. [Strategy 4: Multi-Site Active-Active](#strategy-4-multi-site-active-active) 11. [Choosing a Strategy — The Real Decision Framework](#choosing-a-strategy--the-real-decision-framework) 12. [A Worked Example Across All Four Strategies](#a-worked-example-across-all-four-strategies) 13. [Common Mistakes](#common-mistakes) 14. [Worked Practice Problems](#worked-practice-problems) 15. [Summary and What's Next](#summary-and-whats-next) --- ## Why This Is the Capstone Topic of the Whole Course Every other topic in this course has built toward one underlying question: **what happens when things go seriously wrong, and how do we make sure the business survives it?** Disaster Recovery pulls together the SRE Fundamentals series' error budgets, the Reliability & Architecture Patterns series' redundancy, the Incident Management series' response process, and the Databases & Storage Reliability series' backup discipline into one final, unifying discipline — planning, deliberately and in advance, for the worst realistic day. ```mermaid graph TD SRE["SRE Fundamentals:
SLOs, error budgets"] --> DR["Disaster Recovery
(THIS series)"] Reliability["Reliability & Architecture
Patterns: redundancy,
multi-region"] --> DR Incident["Incident Management:
response process, chaos
engineering"] --> DR Databases["Databases & Storage
Reliability: backups,
RPO/RTO"] --> DR ``` --- ## What Counts as a "Disaster" A genuinely important framing worth stating explicitly up front, since the word "disaster" conjures images of fires and floods, but the real, everyday scope is much broader. ```mermaid graph TD Disaster["What Counts as a
'Disaster' for DR Planning"] --> D1["Infrastructure failure:
a data center, region, or
cloud provider outage"] Disaster --> D2["Human error: an
accidental deletion, a bad
deploy, a misconfiguration
(the Part 3 scenario from
the Databases series)"] Disaster --> D3["Security incident: a
breach, ransomware,
data exfiltration"] Disaster --> D4["Software failure: a bug
corrupting data over
time, silently, before
anyone notices"] Disaster --> D5["Genuine physical disaster:
fire, flood, earthquake,
power grid failure"] ``` **A strong, senior-level interview line worth having ready: "In my experience, the disasters that actually happen most often aren't fires or earthquakes — they're a bad deploy, an accidental deletion, or a misconfiguration. A DR plan that only covers 'the data center burns down' but doesn't cover 'someone ran an unscoped DELETE in production' is planning for the rare, dramatic case while ignoring the common, everyday one."** --- ## Disaster Recovery vs Business Continuity vs High Availability Three genuinely related but distinct terms, frequently confused — worth being precise, since this exact distinction is a common interview opener. ```mermaid graph TD HA["High Availability (HA):
PREVENTING downtime in
the FIRST place — redundancy,
failover, no single point
of failure (Reliability &
Architecture Patterns series)"] --> HANote["Proactive — reduces the
CHANCE of an outage"] DR["Disaster Recovery (DR):
RECOVERING once something
HAS already failed badly
— restoring systems and
data to a working state"] --> DRNote["Reactive — reduces the
IMPACT once an outage
DOES happen"] BC["Business Continuity (BC):
the BROADEST scope — keeping
the ENTIRE BUSINESS
functioning during a
disaster, not just the
technical systems
(people, processes,
communication, even
physical office access)"] --> BCNote["DR is a SUBSET of BC,
specifically the technical/
IT recovery piece"] ``` **A clean, memorable interview line: "High availability tries to prevent the outage from happening at all. Disaster recovery is the plan for when it happens anyway. Business continuity is the broadest umbrella — making sure the whole BUSINESS keeps functioning, which includes DR but also covers things like 'can our support team still answer calls if the office building is inaccessible,' not just the technical systems."** --- ## RPO and RTO, Fully Explained Already introduced in the Databases & Storage Reliability series (Part 3) as the two numbers driving backup strategy — here they get their full, proper, general treatment as the foundation of this entire series. ```mermaid graph TD Timeline["Disaster strikes at
this moment"] --> RPO["RPO looks BACKWARD:
how much data, created
BEFORE the disaster, are
we willing to lose?"] Timeline --> RTO["RTO looks FORWARD: how
long after the disaster
can we take to be back
up and running?"] ``` ```mermaid gantt dateFormat X axisFormat %Mmin title RPO and RTO on a Timeline section Before Disaster Last backup taken :done, a1, 0, 15 Normal operation :done, a2, 15, 45 section Disaster DISASTER STRIKES :crit, milestone, a3, 60, 0 section Recovery RPO window (data lost) :crit, a4, 45, 15 RTO window (downtime) :active, a5, 60, 30 Back online :milestone, a6, 90, 0 ``` **In this worked example: the last backup was taken 45 minutes before the disaster, meaning the RPO (data loss window) is 15 minutes' worth of unbacked-up data (from the last backup at minute 45, until the disaster at minute 60). Recovery then takes 30 minutes, meaning the RTO (downtime) is 30 minutes.** **Simple analogy, worth having ready: RPO is "how far back would we have to rewind the tape" — RTO is "how long would the screen stay black while we set the rewound scene back up and press play again."** --- ## The RPO/RTO Cost Curve The single most important, most frequently tested economic principle in this entire series — worth understanding deeply, not just naming. ```mermaid graph TD Loose["LOOSE RPO/RTO
(hours of data loss/
downtime acceptable)"] --> LooseCost["💰 Cheap — periodic
backups, manual recovery
process is fine"] Tight["TIGHT RPO/RTO
(seconds/minutes of data
loss/downtime acceptable)"] --> TightCost["💰💰💰💰 Expensive —
requires continuous
replication, automated
failover, standing
duplicate infrastructure"] ``` ```mermaid graph LR A["RTO: 1 week"] -->|"tightening costs
MORE at each step"| B["RTO: 1 day"] B --> C["RTO: 1 hour"] C --> D["RTO: 1 minute"] D --> E["RTO: seconds
(near-zero downtime)"] ``` **The core economic principle worth stating explicitly, and it's a genuine, deliberate business tradeoff, not a purely technical one: tightening RPO/RTO doesn't cost linearly more — it costs DRAMATICALLY, non-linearly more as you approach near-zero. Going from "we can be down a day" to "we can be down an hour" might mean automating a manual process. Going from "an hour" to "a few seconds" might mean running fully duplicate, continuously-synchronized infrastructure in a second region, multiplying real infrastructure cost.** This is precisely why RPO/RTO targets should be set deliberately, per system, based on genuine business impact — not defaulted to "as tight as technically possible" everywhere, which the business likely can't actually afford, and doesn't actually need for every single system. --- ## The Four Classic DR Strategies AWS's own well-known DR whitepaper (a widely-cited, genuinely industry-standard reference) frames DR strategy as a spectrum across exactly these four points — worth knowing this framing by name, since it's extremely commonly referenced in interviews. ```mermaid graph LR A["Backup & Restore
(cheapest, SLOWEST)"] --> B["Pilot Light"] B --> C["Warm Standby"] C --> D["Multi-Site Active-Active
(most expensive, FASTEST)"] ``` Each point on this spectrum trades cost for recovery speed — moving right costs more, ongoing, but recovers faster when disaster actually strikes. --- ## Strategy 1: Backup and Restore The simplest, cheapest strategy: regularly back up data (and infrastructure-as-code definitions), and — only when disaster actually strikes — provision everything from scratch and restore. ```mermaid graph TD Normal["Normal operation:
ONLY the primary region
is running anything at
all"] --> Backup["Backups taken
periodically, stored
durably (ideally in a
DIFFERENT region — the
'1 off-site' from the
3-2-1 rule, Databases
series)"] Disaster["Disaster strikes"] --> Provision["Provision ALL new
infrastructure from
scratch (ideally via IaC,
Automation series) in a
NEW region"] Provision --> Restore["Restore data from the
most recent backup"] ``` | | Value | |---|---| | **Typical RPO** | Hours (matches backup frequency) | | **Typical RTO** | Hours to days (must provision everything from scratch) | | **Ongoing cost** | Lowest — pay only for backup storage, nothing standing by | | **Best for** | Non-critical systems, or systems where a business genuinely can tolerate real downtime | --- ## Strategy 2: Pilot Light A small, minimal core of critical infrastructure runs continuously in the DR region — just enough to be quickly "switched on" and scaled up when needed. ```mermaid graph TD Primary["PRIMARY Region:
full production, ALL
traffic"] --> Replicate["Core DATA continuously
replicated (Databases
series, Part 1) to..."] Replicate --> Pilot["...DR Region: a
'pilot light' — the
DATABASE is running and
kept in sync, but
APPLICATION servers are
NOT running (or running
at minimal scale)"] Disaster["Disaster strikes
PRIMARY"] --> Scale["QUICKLY provision/scale
UP application servers in
the DR region — the data
is ALREADY there and
current"] ``` **Simple analogy, worth having ready: a pilot light on a gas stove — a small flame burns continuously, using minimal fuel, but it's specifically what lets the stove ignite to FULL power almost instantly when actually needed, rather than needing to start a fire completely from scratch.** | | Value | |---|---| | **Typical RPO** | Minutes (data is continuously replicated) | | **Typical RTO** | Tens of minutes to a few hours (application layer still needs to scale up) | | **Ongoing cost** | Low-moderate — pay for a running database replica, but minimal/no standing application compute | | **Best for** | Systems where data currency matters more than instant application availability | --- ## Strategy 3: Warm Standby A scaled-down, but fully functional, complete copy of the production environment runs continuously in the DR region — ready to take over, just not yet handling real production traffic. ```mermaid graph TD Primary["PRIMARY Region:
full-scale production"] --> Sync["Continuous data
replication"] Sync --> Standby["DR Region: a SCALED-DOWN
but FULLY FUNCTIONAL
copy — app servers,
database, everything is
ALREADY RUNNING, just at
a smaller capacity, and
NOT receiving real
traffic"] Disaster["Disaster strikes
PRIMARY"] --> ScaleUp["Scale the ALREADY-RUNNING
DR environment UP to full
capacity, and SWITCH
traffic to it"] ``` | | Value | |---|---| | **Typical RPO** | Seconds to minutes | | **Typical RTO** | Minutes (mostly just scaling up and switching traffic — everything is already running) | | **Ongoing cost** | Moderate-high — paying for a running (if smaller) complete duplicate environment, continuously | | **Best for** | Business-critical systems where meaningful downtime has real, direct cost | --- ## Strategy 4: Multi-Site Active-Active The most expensive, fastest-recovering strategy: **both** (or all) regions run at full production scale, simultaneously, actively serving real traffic all the time. ```mermaid graph TD DNS["DNS/Global Load
Balancer"] --> Region1["Region 1: FULL production
scale, ACTIVELY serving
real traffic"] DNS --> Region2["Region 2: FULL production
scale, ALSO actively
serving real traffic"] Region1 <-->|"continuous, bidirectional
data synchronization"| Region2 Disaster["Region 1 fails
entirely"] --> Reroute["DNS/load balancer simply
stops routing to it —
Region 2 was ALREADY
handling real traffic and
simply absorbs the rest"] ``` **Why this is the fastest possible recovery, worth stating explicitly: there's no "switching over" at all in the traditional sense — the failover is simply routing away from the failed region, since the surviving region was already fully live and handling real traffic the entire time. RTO can be genuinely near-zero.** | | Value | |---|---| | **Typical RPO** | Near-zero (continuous, bidirectional sync) | | **Typical RTO** | Near-zero (no actual "switch," just routing away from the failed region) | | **Ongoing cost** | Highest — running full production scale in multiple regions simultaneously, continuously | | **Best for** | The most critical, revenue-impacting systems, where even minutes of downtime have severe, direct business cost | **The real, genuine complexity worth naming explicitly, directly connecting back to the CAP theorem discussion from the Reliability & Architecture Patterns series: bidirectional, multi-region active-active replication reintroduces exactly the same consistency challenges covered in that tutorial — write conflicts between two simultaneously-active regions need a real, deliberate conflict-resolution strategy, and this is genuinely one of the hardest architectural problems in all of distributed systems, not a simple "just run it twice" solution.** --- ## Choosing a Strategy — The Real Decision Framework ```mermaid flowchart TD Start{"What's the REAL,
business-driven RTO/RPO
requirement for THIS
specific system?"} --> Q1{"Hours of downtime/data
loss genuinely acceptable?"} Q1 -->|Yes| BR["Backup & Restore"] Q1 -->|No| Q2{"Tens of minutes to a
few hours acceptable?"} Q2 -->|Yes| PL["Pilot Light"] Q2 -->|No| Q3{"Minutes acceptable?"} Q3 -->|Yes| WS["Warm Standby"] Q3 -->|No| Q4{"Near-zero downtime/data
loss REQUIRED, and the
budget genuinely supports
it?"} Q4 -->|Yes| AA["Multi-Site Active-Active"] ``` **A strong, senior-level interview line, tying this directly to the SRE Fundamentals series' error budget philosophy: "I wouldn't default every system to the most expensive DR strategy 'just to be safe' — exactly like choosing an SLO, this is a deliberate cost-vs-risk tradeoff, made explicitly per system based on genuine business impact, not a one-size-fits-all decision. A company's marketing website and its payment processing system almost certainly warrant very different DR strategies."** --- ## A Worked Example Across All Four Strategies A single, consistent scenario — an e-commerce checkout system — walked through each strategy, to make the tradeoffs concrete. | Strategy | What's Actually Running in DR, Day to Day | If Primary Fails Right Now | |---|---|---| | Backup & Restore | Nothing — just backups sitting in storage | Provision everything from scratch, restore data: hours to a day of downtime | | Pilot Light | The database, kept continuously in sync; no app servers | Spin up app servers, point them at the already-current database: 30-60 minutes | | Warm Standby | A smaller-scale but fully working copy of everything, idle | Scale up capacity, redirect traffic: 5-15 minutes | | Multi-Site Active-Active | An identical, full-scale, ALREADY-LIVE copy, actively serving real traffic | Just stop routing to the failed region: near-instant | **This table is genuinely worth having memorized in this exact shape — it's an extremely common, direct interview request: "walk me through the DR strategy spectrum for a system like [X]."** --- ## Common Mistakes | Mistake | Why It's Wrong | Fix | |---|---|---| | Planning DR only for dramatic, rare events (fires, earthquakes) | The disasters that actually happen most often are human error, bad deploys, and misconfigurations, not physical catastrophes | Plan explicitly for both categories — infrastructure failure AND operational/human error | | Applying the same, most-expensive DR strategy to every system "to be safe" | Wastes real budget on systems that don't genuinely need near-zero RTO/RPO | Choose the DR strategy per system, based on actual, deliberate business impact analysis | | Treating RPO and RTO as purely technical decisions, made by engineering alone | These are fundamentally business decisions about acceptable risk and cost — engineering implements them, but shouldn't unilaterally set them | Involve business/product stakeholders explicitly in setting RPO/RTO targets | | Assuming multi-site active-active is a simple "just run it twice" solution | Reintroduces genuine CAP theorem consistency challenges — write conflicts between regions need real, deliberate resolution strategies | Treat active-active as a genuinely hard distributed-systems problem, not a straightforward scaling decision | | Confusing DR, HA, and BC as interchangeable terms | Leads to gaps — a strong HA setup can still fail against a true disaster if there's no DR plan, and a strong DR plan can still fail the business if BC (people, process) isn't also considered | Understand each term's distinct, specific scope, and ensure all three are actually addressed | --- ## Worked Practice Problems **Problem 1:** A company runs both a payment-processing service (where even 5 minutes of downtime costs real, direct revenue and reputational damage) and an internal analytics dashboard (used occasionally by a small internal team, where a day of downtime is a genuine inconvenience but not a crisis). How would you recommend different DR strategies for each, and why? *Answer:* The payment-processing service warrants Warm Standby or, if the budget genuinely supports it and the revenue impact justifies it, Multi-Site Active-Active — the direct, ongoing cost of a duplicate (or fully active) environment is justified by the severe, immediate business impact of even brief downtime. The internal analytics dashboard is a strong fit for Backup & Restore — its RTO/RPO tolerance (a day of downtime being a genuine but non-critical inconvenience) doesn't justify the ongoing cost of any standing DR infrastructure at all; provisioning from scratch and restoring from backup when genuinely needed is the cost-appropriate choice. **Problem 2:** A team proposes multi-site active-active for a system with a genuinely modest RTO/RPO requirement (a few hours of downtime is acceptable), reasoning "more reliability is always better." How would you respond? *Answer:* I'd push back on the framing — this isn't "more reliability is always better," it's a real, deliberate cost-vs-benefit tradeoff, exactly like choosing an SLO in the SRE Fundamentals series. Multi-site active-active costs dramatically more, on an ongoing basis, than the system's actual, stated business requirement justifies, and it introduces genuine additional complexity (cross-region write conflict resolution, from the CAP theorem discussion) that isn't actually needed here. A strategy matching the real RTO/RPO requirement — likely Pilot Light or even Backup & Restore — would deliver the same acceptable outcome at meaningfully lower ongoing cost and complexity. **Problem 3:** Explain the difference between High Availability, Disaster Recovery, and Business Continuity using a concrete example involving a company's primary office building losing power for a day. *Answer:* High Availability is about the technical systems continuing to run smoothly despite normal, expected failures (a single server dying, a disk failing) — it wouldn't directly address an entire office losing power at all, since that's not really about redundant infrastructure design. Disaster Recovery covers restoring the actual technical systems and data if the outage genuinely disrupted them (e.g., if critical servers were physically located in that office building itself). Business Continuity is the broadest scope, covering the whole business functioning through the disruption — can employees work remotely, can customer support still answer calls, are critical business processes (invoicing, payroll) still able to run — regardless of whether the underlying technical systems were even affected by the power outage at all. --- ## Summary and What's Next - **High Availability** prevents downtime proactively; **Disaster Recovery** reduces the impact once a disaster has already happened; **Business Continuity** is the broadest umbrella, covering the entire business (people, process, communication), not just technical systems. - Real disasters are far more often human error, bad deploys, and misconfigurations than dramatic physical events — a DR plan needs to cover both. - **RPO** (how much data can we afford to lose) and **RTO** (how long can we afford to be down) are deliberate business decisions, not purely technical ones, and they directly drive every downstream DR design choice. - Tightening RPO/RTO costs **dramatically, non-linearly more** as you approach near-zero — this economic reality is exactly why DR strategy should be chosen deliberately, per system, based on genuine business impact. - The **four classic DR strategies** — Backup & Restore, Pilot Light, Warm Standby, Multi-Site Active-Active — form a spectrum trading ongoing cost for recovery speed, from AWS's widely-cited DR framework. - **Multi-site active-active** reintroduces genuine CAP theorem consistency challenges (cross-region write conflicts) — it's a hard distributed-systems problem, not a simple "run it twice" solution. **Continue to Part 2** (`02-dr-testing-and-runbooks.md`) to cover how DR plans actually get validated and practiced before a real disaster forces the question — testing methodologies, runbooks, and the communication planning that turns a strategy on paper into a real, working capability.