# Disaster Recovery & Business Continuity — Part 3: Real-World Disasters & Case Studies > **Series:** Disaster Recovery & Business Continuity (3 of 3) > **Part 1:** `01-dr-fundamentals-and-strategies.md` — DR Fundamentals & Strategies > **Part 2:** `02-dr-testing-and-runbooks.md` — DR Testing & Runbooks > **Part 3:** This file — Real-World Disasters & Case Studies > **Questions:** `questions.md` ## Table of Contents 1. [Why Real Case Studies Matter More Than Theory Alone](#why-real-case-studies-matter-more-than-theory-alone) 2. [Case Study 1: GitLab's 2017 Database Incident](#case-study-1-gitlabs-2017-database-incident) 3. [Case Study 2: The 2017 AWS S3 us-east-1 Outage](#case-study-2-the-2017-aws-s3-us-east-1-outage) 4. [Case Study 3: The 2021 Facebook/Meta BGP Outage](#case-study-3-the-2021-facebookmeta-bgp-outage) 5. [Case Study 4: Knight Capital, Revisited](#case-study-4-knight-capital-revisited) 6. [The Common Thread Across All Four](#the-common-thread-across-all-four) 7. [Building a Genuine DR Culture](#building-a-genuine-dr-culture) 8. [The Complete DR Plan Template](#the-complete-dr-plan-template) 9. [A Final Pre-Disaster Checklist](#a-final-pre-disaster-checklist) 10. [How This Whole Course Fits Together](#how-this-whole-course-fits-together) 11. [Common Mistakes](#common-mistakes) 12. [Worked Practice Problems](#worked-practice-problems) 13. [Summary — The Complete Disaster Recovery & Business Continuity Series](#summary--the-complete-disaster-recovery--business-continuity-series) --- ## Why Real Case Studies Matter More Than Theory Alone Parts 1 and 2 of this series covered the theory and practice of DR planning and testing. This final Part grounds all of it in real, publicly documented industry incidents — being able to cite a real case study by name, with its actual root cause and lasting lesson, is one of the strongest possible signals of genuine depth in an SRE interview, far stronger than reciting definitions alone. --- ## Case Study 1: GitLab's 2017 Database Incident Widely considered one of the most instructive, most transparently documented incidents in the entire industry — GitLab published a genuinely detailed, public postmortem, and it remains one of the best real-world teaching examples of nearly every concept from the Databases & Storage Reliability series colliding at once. ```mermaid flowchart TD A["An engineer, troubleshooting
replication lag, ran a command
intending to clear a
REPLICA's data directory"] --> B["🚨 Ran it against the WRONG
server — the PRIMARY,
production database"] B --> C["~300GB of production data
DELETED within seconds"] C --> D["Team then discovered: of
FIVE distinct backup/
replication mechanisms
they believed they had,
FOUR were broken or not
actually running as
expected"] D --> E["Recovery relied on a
SNAPSHOT that happened to
have been taken manually,
roughly 6 hours before —
real, permanent data loss
of everything created in
that 6-hour gap"] ``` **The concrete, lasting lessons worth citing by name, each directly connecting to a tutorial elsewhere in this course:** - **Untested backups are a hypothesis, not a safety net** (directly echoing the exact principle from the Databases series, Part 3, and Part 2 of THIS series) — having five theoretically-redundant backup mechanisms provided almost zero real protection, because none of them had been genuinely, regularly verified through an actual restore test. - **Human error against production, with no safeguard, is a real, ever-present risk** — not a rare, exotic edge case (directly echoing the "what actually counts as a disaster" discussion from Part 1 of this series) — a single wrong command, run against the wrong server, caused the entire incident. - **GitLab's own response — publishing a fully transparent, detailed public postmortem — is itself a strong, positive example of blameless postmortem culture** (SRE Fundamentals series) applied at real, public-facing scale, turning a genuinely damaging incident into a widely-cited industry teaching moment rather than a hidden, repeated failure. --- ## Case Study 2: The 2017 AWS S3 us-east-1 Outage A genuinely instructive case study specifically about the risks of hidden dependencies and cascading failure at massive scale. ```mermaid flowchart TD A["An engineer, debugging a
billing system issue,
intended to remove a
SMALL number of servers
from one S3 subsystem"] --> B["A command's INPUT was
entered incorrectly,
removing a MUCH LARGER
set of servers than
intended"] --> C["This took down a core
S3 subsystem in the
us-east-1 region entirely"] C --> D["🚨 CASCADING impact: an
enormous number of OTHER
AWS services, and
countless customer
applications ACROSS THE
INTERNET, had a HIDDEN
dependency on this exact
S3 subsystem — many had
NO IDEA they depended on
it until it went down"] ``` **The concrete, lasting lessons worth citing:** - **Hidden, undocumented dependencies are a real, common, and severely underestimated risk** — directly connecting to the dependency mapping problem from Part 2 of this series: countless services that had never explicitly, consciously chosen to depend on this specific S3 subsystem were nonetheless deeply affected, because the dependency existed but was never mapped or considered. - **A single region CAN, in fact, experience a genuinely severe, wide-ranging outage** — directly validating the entire premise of multi-region DR strategies from Part 1 of this series; some organizations affected by this specific outage had never seriously planned for a "the entire AWS region is degraded" scenario at all. - **Operational tooling itself needs real safeguards** — a genuinely important, concrete lesson: the specific tool used to remove servers didn't have sufficient guardrails (like a hard cap on how many servers could be removed by one command) to catch an oversized, likely-mistaken input before executing it — directly connecting to the "operational tooling needs guardrails, not just correct intentions" theme from the DevSecOps series. --- ## Case Study 3: The 2021 Facebook/Meta BGP Outage A genuinely remarkable, widely-discussed case study specifically because of how the company's OWN internal systems — including the tools needed to actually FIX the problem — were themselves victims of the outage. ```mermaid flowchart TD A["A routine, intended-to-be-
safe maintenance command
accidentally withdrew ALL
of the company's own BGP
route announcements
(Linux & Networking
Fundamentals series, Part 2
— the internet-scale
routing protocol)"] --> B["🚨 This effectively removed
the company's ENTIRE
infrastructure from the
internet's routing tables
— their services became
UNREACHABLE globally"] B --> C["Their OWN internal DNS
(Linux & Networking
Fundamentals series, Part 2)
depended on the SAME
infrastructure that had
just disappeared — engineers
couldn't even reach their
OWN internal tools to
diagnose or fix it"] C --> D["🚨 Even physical access to
data centers was reportedly
complicated, since BADGE
READERS depended on
internal systems that were
ALSO now unreachable"] ``` **The concrete, lasting lessons worth citing:** - **A recovery plan must not depend entirely on the exact systems it's trying to recover** — a genuinely profound, often-underestimated principle, directly extending the communication-plan lesson from Part 2 of this series (hosting your status page on the same infrastructure being tested for failure) all the way to its most extreme, real-world conclusion: even PHYSICAL access and internal tooling had a hidden dependency on the very systems that had just failed. - **"Break glass" / out-of-band access matters** — mature organizations maintain a genuinely separate, independent way to regain access and control during a catastrophic failure of primary systems, precisely to avoid this exact trap. - **BGP and DNS sit at a foundational, easy-to-overlook layer** — directly reinforcing why the Linux & Networking Fundamentals series treats these topics as genuine, deep foundations rather than a superficial afterthought — a mistake at this layer can have consequences far more severe and far-reaching than a typical application-level bug. --- ## Case Study 4: Knight Capital, Revisited Already introduced in the DevSecOps series (Part 5) as a supply-chain/deployment-process case study — worth revisiting here specifically through the DR/business-continuity lens. ```mermaid graph TD A["A botched deployment
activated old, dead test
code in production"] --> B["The bug caused the firm to
execute a massive volume of
unintended, erroneous stock
trades within approximately
45 MINUTES"] --> C["~$440 million in losses —
a financial disaster
SEVERE enough that it
threatened the company's
continued existence, and
ultimately led to its
acquisition shortly after"] ``` **The concrete, lasting lesson worth citing, specifically through this series' lens: this wasn't a "disaster" in the traditional sense of a data center fire or a hardware failure — it was a genuinely severe BUSINESS CONTINUITY event caused entirely by a software deployment process gap, directly reinforcing the Part 1 principle that real disasters are far more often operational/process failures than dramatic physical events.** A DR/business-continuity plan that only ever considers infrastructure-level failures would have had nothing to say about this specific, genuinely business-ending scenario at all. --- ## The Common Thread Across All Four ```mermaid graph TD Thread["The Common Thread"] --> T1["NONE of these were caused
by a fire, flood, or
earthquake — every single
one was HUMAN/PROCESS
error, or a HIDDEN
dependency nobody had
mapped"] Thread --> T2["In EVERY case, the actual
SEVERITY was made
dramatically worse by a gap
in PREPAREDNESS (untested
backups, unmapped
dependencies, no
independent recovery path)
— NOT by the initial
triggering mistake alone"] ``` **The single strongest, most memorable synthesis worth having ready for an interview: "Every one of these famous, industry-defining incidents was ultimately a relatively small, human mistake — a wrong command, a bad deploy — that became catastrophic specifically because of a PREPAREDNESS gap: untested backups, unmapped dependencies, or no independent way to recover. The mistake itself is almost always survivable. It's the absence of a genuinely tested, well-understood recovery plan that turns a mistake into a disaster."** --- ## Building a Genuine DR Culture A closing, culture-level synthesis, directly extending the blameless postmortem culture from the SRE Fundamentals series to this entire series' subject matter. ```mermaid graph TD Culture["A Genuine DR Culture"] --> C1["DR planning is EVERYONE's
concern, not a document
written once by one person
and filed away"] Culture --> C2["Testing gaps are treated
as VALUABLE FINDINGS
(exactly like a chaos
experiment 'disproving' a
hypothesis, Incident
Management series) — NOT
as embarrassing failures
to hide"] Culture --> C3["DR readiness is TRACKED
and REPORTED, the same way
error budgets and SLOs are
(SRE Fundamentals series) —
not a one-time checkbox"] Culture --> C4["Leadership genuinely
SUPPORTS the real cost
(engineering time, some
ongoing infrastructure
spend) of REGULAR testing —
not just the cost of
writing the initial plan"] ``` --- ## The Complete DR Plan Template A genuinely practical, consolidated template — worth having this exact shape ready to sketch from memory, since "walk me through how you'd build a DR plan for a new system" is a very common, direct senior-level interview request. ```markdown # Disaster Recovery Plan: [System Name] ## 1. Business Impact Analysis - What does this system do, and who depends on it? - What's the REAL cost of downtime? (revenue, reputation, regulatory/compliance exposure) - Explicitly stated RPO: ___ - Explicitly stated RTO: ___ ## 2. Chosen DR Strategy (Part 1) - Backup & Restore / Pilot Light / Warm Standby / Multi-Site Active-Active — and WHY this one, given the RPO/RTO above ## 3. Dependency Map (Part 2) - Direct dependencies: ___ - Transitive dependencies: ___ - External/third-party dependencies: ___ - For EACH: does it ALSO have its own DR plan/failover? ## 4. Runbook (Part 2) - Trigger criteria - Declaration authority - Step-by-step recovery procedure, with EXACT commands - Verification steps - SEPARATE failback procedure ## 5. Communication Plan (Part 2) - Internal: channel INDEPENDENT of the systems being recovered - External: status page, hosted INDEPENDENTLY - Regulatory/legal: any mandatory disclosure triggers? ## 6. Testing Schedule (Part 2) - Tabletop: quarterly - Partial failover: semi-annually - Full failover: annually (or more often, for critical systems) - Re-test triggers: after any significant architecture change ## 7. Ownership - Plan owner: ___ - Last reviewed: ___ - Last ACTUALLY tested: ___ (not just reviewed — tested) ``` --- ## A Final Pre-Disaster Checklist A last, practical, memorable checklist — genuinely useful both as real guidance and as a strong closing answer to "what would you check before signing off that a system is DR-ready." ```mermaid graph TD Checklist["Pre-Disaster Readiness
Checklist"] --> Q1["✅ Is RPO/RTO EXPLICITLY
defined, and was it a
BUSINESS decision, not just
engineering's guess?"] Checklist --> Q2["✅ Has the backup/failover
mechanism been ACTUALLY
tested, not just assumed
to work?"] Checklist --> Q3["✅ Is the FULL dependency
chain mapped, not just the
headline service?"] Checklist --> Q4["✅ Does communication
tooling survive the SAME
failure it's meant to
communicate about?"] Checklist --> Q5["✅ Do MULTIPLE people know
how to execute this, not
just one?"] Checklist --> Q6["✅ Has FAILBACK, not just
failover, been tested?"] ``` --- ## How This Whole Course Fits Together A final, closing synthesis — genuinely worth internalizing as the single overarching narrative connecting every topic in this entire course. ```mermaid graph TD SLO["SRE Fundamentals:
define what 'reliable
enough' means (SLO)"] --> Monitor["Monitoring & Observability:
MEASURE whether you're
meeting it"] Monitor --> Security["DevSecOps: prevent
security-driven disasters
at every stage"] Monitor --> Reliability["Reliability Patterns:
BUILD systems that resist
failure in the first place"] Reliability --> Capacity["Capacity Planning:
ensure there's ENOUGH
capacity to actually meet
demand"] Capacity --> Linux["Linux & Networking,
Kubernetes, Databases:
the DEEP technical
foundations everything
else runs on"] Linux --> Incident["Incident Management:
RESPOND well when
something breaks anyway"] Incident --> Automation["Automation & GitOps:
make changes SAFELY and
REPEATABLY"] Automation --> DR["Disaster Recovery
(THIS series): the final
safety net for when
EVERYTHING else still
somehow fails"] ``` **The single sentence worth closing this entire course on: every topic covered has been building toward the same underlying goal — keeping a promise to users about how reliable a system will be, and Disaster Recovery is simply that same promise extended to cover the worst, most extreme case, planned for deliberately and in advance, rather than improvised under real, uncontrolled pressure.** --- ## Common Mistakes | Mistake | Why It's Wrong | Fix | |---|---|---| | Assuming a well-known cloud provider's infrastructure can never have a severe, region-wide outage | Real, well-documented incidents (like the 2017 S3 outage) prove otherwise, at massive scale | Explicitly plan for genuine regional/provider-level failure, not just individual server failure | | Believing "five different backup mechanisms" automatically means real redundancy | GitLab's incident shows redundant-on-paper mechanisms can ALL be broken or non-functional simultaneously if none are actually tested | Regularly, genuinely test every backup/recovery mechanism you believe you have | | Not considering whether your OWN recovery tooling depends on the exact systems you're trying to recover | The 2021 Facebook/Meta outage shows this can escalate to a genuinely severe, self-inflicted lockout | Maintain a genuinely independent, "break glass" path to regain access/control during a catastrophic failure | | Treating DR planning as purely an infrastructure/hardware-failure concern | Knight Capital's case shows a pure software/process failure can be just as catastrophic to business continuity as any physical disaster | Plan explicitly for human error and deployment-process failures, not only infrastructure failures | | Treating DR readiness as a one-time project with a defined "done" state | Real systems and dependencies change continuously — a plan tested a year ago against a different architecture provides limited real confidence today | Track DR readiness as an ongoing practice, re-tested regularly and after significant architecture changes | --- ## Worked Practice Problems **Problem 1:** An interviewer asks you to name a real-world disaster/incident and explain what your organization would do differently as a result. Walk through a strong answer using GitLab's 2017 incident. *Answer:* I'd reference GitLab's 2017 incident specifically: an engineer accidentally deleted production data while troubleshooting replication, and the team discovered that of five theoretically redundant backup mechanisms, four weren't actually working — recovery relied on a lucky manual snapshot, with real, permanent data loss for the gap since that snapshot. What I'd do differently: treat "we have multiple backup mechanisms" as meaningless until each one has been individually, regularly verified through an actual restore test — exactly the "untested backup is a hypothesis" principle from this series — and I'd add explicit safeguards (confirmation prompts, environment-name verification) to any command capable of deleting production data, since the root trigger was ultimately a human running a legitimate command against the wrong target. **Problem 2:** How does the 2021 Facebook/Meta BGP outage change how you'd think about designing an internal "break glass" emergency access procedure? *Answer:* That incident showed that even a company with immense engineering resources can end up locked out of their own systems — including internal tools and even physical building access — when a catastrophic failure takes down the exact infrastructure that recovery normally depends on. I'd design break-glass access to be genuinely, deliberately independent of primary production infrastructure — for example, out-of-band access credentials that don't rely on internal SSO tied to the same systems, physical access mechanisms that don't depend purely on networked badge readers, and internal communication channels (Part 2 of this series) hosted on completely separate infrastructure — specifically so a catastrophic failure of the primary environment can never also take down the team's ability to actually respond to and fix it. **Problem 3:** A junior engineer argues "our DR plan only needs to cover infrastructure failures like a data center outage, since that's what 'disaster recovery' really means." Using real case studies, how would you respond? *Answer:* I'd point to Knight Capital as a direct counterexample — a purely software/deployment-process failure, with zero infrastructure or hardware failure involved at all, caused a genuinely business-ending financial disaster in about 45 minutes. I'd also note that GitLab's 2017 incident and the AWS S3 outage were both ultimately triggered by human/operational error, not physical infrastructure failure, even though the S3 outage's IMPACT was infrastructure-shaped. The lesson across essentially every well-known, real-world case study: disasters are far more often caused by human error, deployment mistakes, and hidden dependencies than by data centers physically failing — a DR plan scoped only to hardware/infrastructure failure misses the majority of what actually causes real, severe incidents in practice. --- ## Summary — The Complete Disaster Recovery & Business Continuity Series - **GitLab (2017)**: an accidental production database deletion revealed that five theoretically redundant backup mechanisms were mostly non-functional — the defining, industry-wide lesson that untested backups provide false confidence, not real protection. - **AWS S3 us-east-1 (2017)**: a mistyped command's cascading impact revealed how deeply and invisibly the broader internet depended on one region's infrastructure — reinforcing that hidden, unmapped dependencies and genuine region-wide failures are both real, not theoretical risks. - **Facebook/Meta (2021)**: a BGP misconfiguration locked the company out of its own internet-facing infrastructure — and, remarkably, out of much of its own internal tooling and even physical access — the definitive lesson that recovery mechanisms must never depend entirely on the exact systems they're meant to recover. - **Knight Capital**: a deployment process failure with zero infrastructure involvement caused a genuinely business-ending financial disaster in under an hour — proof that business continuity planning must cover human/process failures, not only physical infrastructure ones. - **The common thread across all four**: none were caused by a fire or flood — every one was human/process error made catastrophic by a genuine preparedness gap, not by the initial mistake alone. - A **genuine DR culture** treats testing gaps as valuable findings, tracks DR readiness as an ongoing practice (like an SLO), and requires real, sustained leadership support for regular testing — not a document written once and filed away. - The complete **DR plan template** — business impact analysis, chosen strategy, dependency map, runbook, communication plan, testing schedule, and clear ownership — ties every concept from this entire three-part series into one practical, reusable structure. This completes the **Disaster Recovery & Business Continuity** series — and with it, the full course. See `questions.md` in this folder for the full interview question bank covering all three parts.