# Interview Questions: Disaster Recovery & Business Continuity

Companion question bank for the 3-part tutorial series in this folder:
`01-dr-fundamentals-and-strategies.md`, `02-dr-testing-and-runbooks.md`, `03-real-world-disasters-and-case-studies.md`.

Answers are short and plain — expand out loud using the diagrams and worked examples in the tutorials.

---

# Part 1 Questions: DR Fundamentals & Strategies

### 1. What kinds of events should a DR plan actually cover, beyond physical disasters?
Infrastructure/region failure, human error (an accidental deletion, a bad deploy), security incidents, software bugs silently corrupting data, and genuine physical disasters. In practice, human error and process failures happen far more often than fires or earthquakes.

### 2. Distinguish High Availability, Disaster Recovery, and Business Continuity.
HA proactively prevents downtime (redundancy, failover). DR reactively recovers once something has already failed badly. Business Continuity is the broadest scope — keeping the entire business functioning (people, process, communication), not just technical systems. DR is a subset of BC.

### 3. Define RPO and RTO in plain terms.
RPO looks backward: how much data, created before a disaster, can we afford to lose? RTO looks forward: how long can we afford to be down while recovering? Both are deliberate business decisions, not purely technical ones.

### 4. Why does tightening RPO/RTO cost non-linearly more as you approach near-zero?
Going from hours to an hour might mean automating a manual process. Going from an hour to a few seconds might mean running fully duplicate, continuously-synchronized infrastructure in a second region — the cost curve gets dramatically steeper near the top, not linear.

### 5. Name the four classic DR strategies, from cheapest/slowest to most expensive/fastest.
Backup & Restore, Pilot Light, Warm Standby, Multi-Site Active-Active.

### 6. Describe the Pilot Light strategy with an analogy.
A small, continuously-running core (typically just the database, kept in sync) in the DR region — like a gas stove's pilot light: minimal fuel used continuously, but it lets the stove ignite to full power almost instantly when actually needed, instead of starting a fire from scratch.

### 7. What's the key difference between Pilot Light and Warm Standby?
Pilot Light has only the data layer running continuously (app servers are off or minimal). Warm Standby has a full, scaled-down but FUNCTIONAL copy of everything already running, just not receiving real traffic — recovery is mostly scaling up and switching traffic, not provisioning from scratch.

### 8. Why can Multi-Site Active-Active achieve near-zero RTO?
There's no real "switch" — both regions are already fully live and serving real traffic simultaneously, so failover is just routing away from the failed region, which the other region was already handling anyway.

### 9. What genuine complexity does Multi-Site Active-Active reintroduce?
The exact CAP theorem consistency challenges from the Reliability & Architecture Patterns series — bidirectional replication between two simultaneously-active regions creates real write-conflict scenarios needing a deliberate resolution strategy, not a simple "run it twice" solution.

### 10. How should a team actually choose a DR strategy for a given system?
Based on the real, business-driven RPO/RTO requirement for THAT specific system — not defaulting every system to the most expensive strategy "to be safe," exactly like choosing an SLO deliberately rather than always aiming for 100%.

---

# Part 2 Questions: DR Testing & Runbooks

### 11. Why is an untested DR plan described as "a hypothesis, not a verified capability"?
Real, documented failures happen where a runbook references a decommissioned tool, an unreachable person, or a forgotten dependency — none of which show up just from reading the document. Only actual testing surfaces these gaps, safely, before a real disaster does.

### 12. Walk through the DR testing maturity ladder.
Tabletop exercise (talk through it, no systems touched) → walkthrough/simulation (real steps, non-production environment) → partial/component failover test (one real piece, in production, controlled) → full failover test (the entire system, for real) → chaos engineering/unannounced drills (no advance warning).

### 13. Why is a tabletop exercise so valuable despite touching zero real systems?
It cheaply and quickly surfaces PROCESS and OWNERSHIP gaps — like unclear declaration authority or a runbook referencing a tool nobody uses anymore — often revealing serious gaps in minutes that would otherwise only surface during a real crisis.

### 14. Why is testing failback just as important as testing failover?
Returning to the original primary is a genuinely separate, sometimes more complex operation — especially reconciling data written to the DR environment while it was live — that can fail even when the failover itself worked perfectly.

### 15. What should a good DR runbook always include?
Exact, copy-pasteable commands (not vague descriptions), explicit declaration authority, explicit dependency ordering, explicit verification steps at each stage, and a separate failback procedure — kept current through regular testing.

### 16. Why is hosting a status page on the same infrastructure being tested for failure a real, common gap?
The exact tool needed to communicate about the disaster becomes a casualty of it — the same "your monitoring shouldn't depend on the thing it's monitoring" principle applied to communication tooling specifically.

### 17. What is the "dependency mapping problem" in DR planning?
A DR plan that only considers one headline service, without mapping its full dependency chain (internal services it depends on, external providers), can produce a technically "successful" failover of that one service that still doesn't actually work, because a dependency wasn't also failed over.

### 18. How does "people are part of the system too" connect to redundancy principles from earlier in the course?
A DR plan relying on one specific person's tribal knowledge is exactly as fragile as an un-redundant server — the fix is the same principle applied to knowledge instead of infrastructure: cross-train multiple people, validated through real testing.

### 19. How often should different levels of DR testing happen?
Tabletop exercises: quarterly (cheap enough to do often). Partial failover tests: quarterly to semi-annually. Full failover tests: at least annually, more often for the most critical systems. Also: re-test after any significant architecture change, since a plan tested against last year's architecture may not reflect this year's reality.

---

# Part 3 Questions: Real-World Disasters & Case Studies

### 20. Summarize GitLab's 2017 incident and its main lesson.
An engineer accidentally deleted production data while troubleshooting replication lag on the wrong server. Of five believed-redundant backup mechanisms, four were broken or not actually running — recovery relied on a lucky manual snapshot, with real permanent data loss for the gap since. Main lesson: untested backups provide false confidence, not real protection.

### 21. Summarize the 2017 AWS S3 us-east-1 outage and its main lesson.
A mistyped command removed far more servers than intended from a core S3 subsystem, taking it down entirely — an enormous number of other services and customer applications across the internet had hidden, unmapped dependencies on that exact subsystem and were affected. Main lesson: hidden dependencies are real and severely underestimated, and a single region CAN experience a genuinely severe, wide-ranging outage.

### 22. Summarize the 2021 Facebook/Meta BGP outage and its main lesson.
A maintenance command accidentally withdrew all of the company's BGP route announcements, making their infrastructure unreachable globally — including their own internal DNS and tools, and reportedly even physical badge-reader access, since those also depended on the now-unreachable infrastructure. Main lesson: recovery mechanisms must never depend entirely on the exact systems they're meant to recover — "break glass" independent access matters.

### 23. Why is Knight Capital relevant to a DR/business-continuity discussion specifically?
It shows a purely software/deployment-process failure — with zero infrastructure or hardware failure involved — can be just as catastrophic to business continuity as any physical disaster, causing ~$440M in losses in about 45 minutes and threatening the company's survival.

### 24. What's the common thread across all four case studies?
None were caused by a fire, flood, or earthquake — every one was human/process error or a hidden dependency. In every case, the severity was made dramatically worse by a genuine preparedness gap (untested backups, unmapped dependencies, no independent recovery path), not by the initial mistake alone.

### 25. What does a genuine DR culture look like, as opposed to a checkbox exercise?
DR planning is everyone's concern, not a document written once and filed away. Testing gaps are treated as valuable findings (like a chaos experiment disproving a hypothesis), not embarrassing failures to hide. DR readiness is tracked and reported like an SLO, with real, sustained leadership support for ongoing testing.

### 26. Name the sections of a complete DR plan template.
Business impact analysis (with explicit RPO/RTO), chosen DR strategy and why, full dependency map, runbook (with trigger criteria, declaration authority, exact steps, verification, and separate failback), communication plan, testing schedule, and clear ownership.

---

## Quick-Fire / Rapid Recall

| Q | A |
|---|---|
| HA vs DR vs BC? | Prevent downtime vs. recover from it vs. keep the whole business running |
| RPO vs RTO? | Data loss tolerance (backward) vs. downtime tolerance (forward) |
| Why does RPO/RTO cost curve get steep near zero? | Near-zero requires continuous replication + standing duplicate infra |
| Four DR strategies, cheap to expensive? | Backup & Restore, Pilot Light, Warm Standby, Multi-Site Active-Active |
| Fastest possible RTO strategy? | Multi-Site Active-Active (no real "switch" needed) |
| What does active-active reintroduce? | CAP theorem write-conflict challenges |
| Is an untested DR plan trustworthy? | No — treat it as unverified |
| DR testing maturity ladder order? | Tabletop → walkthrough → partial failover → full failover → unannounced drills |
| Why test failback separately from failover? | It's a genuinely separate, sometimes harder operation |
| Common status-page mistake? | Hosting it on the same infra being tested for failure |
| What's the "dependency mapping problem"? | Failing over one service without its dependencies still leaves things broken |
| GitLab 2017 — main lesson? | Untested backups ≠ real protection |
| AWS S3 2017 — main lesson? | Hidden dependencies + real region-wide outages are possible |
| Facebook/Meta 2021 — main lesson? | Recovery tools must not depend on the systems they're recovering |
| Knight Capital — main lesson? | Pure process/deploy failures can be business-ending too |
| Common thread across all 4 case studies? | Human/process error + a preparedness gap, not a physical disaster |
| What turns a mistake into a disaster? | The absence of a genuinely tested recovery plan |