Real-World Disasters & Case Studies

Table of Contents#

1. Why Real Case Studies Matter More Than Theory Alone
2. Case Study 1: GitLab's 2017 Database Incident
3. Case Study 2: The 2017 AWS S3 us-east-1 Outage
4. Case Study 3: The 2021 Facebook/Meta BGP Outage
5. Case Study 4: Knight Capital, Revisited
6. The Common Thread Across All Four
7. Building a Genuine DR Culture
8. The Complete DR Plan Template
9. A Final Pre-Disaster Checklist
10. How This Whole Course Fits Together
11. Common Mistakes
12. Worked Practice Problems
13. Summary — The Complete Disaster Recovery & Business Continuity Series

---

Why Real Case Studies Matter More Than Theory Alone#

Parts 1 and 2 of this series covered the theory and practice of DR planning and testing. This final Part grounds all of it in real, publicly documented industry incidents — being able to cite a real case study by name, with its actual root cause and lasting lesson, is one of the strongest possible signals of genuine depth in an SRE interview, far stronger than reciting definitions alone.

---

Case Study 1: GitLab's 2017 Database Incident#

Widely considered one of the most instructive, most transparently documented incidents in the entire industry — GitLab published a genuinely detailed, public postmortem, and it remains one of the best real-world teaching examples of nearly every concept from the Databases & Storage Reliability series colliding at once.

Diagram

The concrete, lasting lessons worth citing by name, each directly connecting to a tutorial elsewhere in this course:

• Untested backups are a hypothesis, not a safety net (directly echoing the exact principle from the Databases series, Part 3, and Part 2 of THIS series) — having five theoretically-redundant backup mechanisms provided almost zero real protection, because none of them had been genuinely, regularly verified through an actual restore test.
• Human error against production, with no safeguard, is a real, ever-present risk — not a rare, exotic edge case (directly echoing the "what actually counts as a disaster" discussion from Part 1 of this series) — a single wrong command, run against the wrong server, caused the entire incident.
• GitLab's own response — publishing a fully transparent, detailed public postmortem — is itself a strong, positive example of blameless postmortem culture (SRE Fundamentals series) applied at real, public-facing scale, turning a genuinely damaging incident into a widely-cited industry teaching moment rather than a hidden, repeated failure.

---

Case Study 2: The 2017 AWS S3 us-east-1 Outage#

A genuinely instructive case study specifically about the risks of hidden dependencies and cascading failure at massive scale.

Diagram

The concrete, lasting lessons worth citing:

• Hidden, undocumented dependencies are a real, common, and severely underestimated risk — directly connecting to the dependency mapping problem from Part 2 of this series: countless services that had never explicitly, consciously chosen to depend on this specific S3 subsystem were nonetheless deeply affected, because the dependency existed but was never mapped or considered.
• A single region CAN, in fact, experience a genuinely severe, wide-ranging outage — directly validating the entire premise of multi-region DR strategies from Part 1 of this series; some organizations affected by this specific outage had never seriously planned for a "the entire AWS region is degraded" scenario at all.
• Operational tooling itself needs real safeguards — a genuinely important, concrete lesson: the specific tool used to remove servers didn't have sufficient guardrails (like a hard cap on how many servers could be removed by one command) to catch an oversized, likely-mistaken input before executing it — directly connecting to the "operational tooling needs guardrails, not just correct intentions" theme from the DevSecOps series.

---

Case Study 3: The 2021 Facebook/Meta BGP Outage#

A genuinely remarkable, widely-discussed case study specifically because of how the company's OWN internal systems — including the tools needed to actually FIX the problem — were themselves victims of the outage.

Diagram

The concrete, lasting lessons worth citing:

• A recovery plan must not depend entirely on the exact systems it's trying to recover — a genuinely profound, often-underestimated principle, directly extending the communication-plan lesson from Part 2 of this series (hosting your status page on the same infrastructure being tested for failure) all the way to its most extreme, real-world conclusion: even PHYSICAL access and internal tooling had a hidden dependency on the very systems that had just failed.
• "Break glass" / out-of-band access matters — mature organizations maintain a genuinely separate, independent way to regain access and control during a catastrophic failure of primary systems, precisely to avoid this exact trap.
• BGP and DNS sit at a foundational, easy-to-overlook layer — directly reinforcing why the Linux & Networking Fundamentals series treats these topics as genuine, deep foundations rather than a superficial afterthought — a mistake at this layer can have consequences far more severe and far-reaching than a typical application-level bug.

---

Case Study 4: Knight Capital, Revisited#

Already introduced in the DevSecOps series (Part 5) as a supply-chain/deployment-process case study — worth revisiting here specifically through the DR/business-continuity lens.

Diagram

The concrete, lasting lesson worth citing, specifically through this series' lens: this wasn't a "disaster" in the traditional sense of a data center fire or a hardware failure — it was a genuinely severe BUSINESS CONTINUITY event caused entirely by a software deployment process gap, directly reinforcing the Part 1 principle that real disasters are far more often operational/process failures than dramatic physical events. A DR/business-continuity plan that only ever considers infrastructure-level failures would have had nothing to say about this specific, genuinely business-ending scenario at all.

---

The Common Thread Across All Four#

Diagram

The single strongest, most memorable synthesis worth having ready for an interview: "Every one of these famous, industry-defining incidents was ultimately a relatively small, human mistake — a wrong command, a bad deploy — that became catastrophic specifically because of a PREPAREDNESS gap: untested backups, unmapped dependencies, or no independent way to recover. The mistake itself is almost always survivable. It's the absence of a genuinely tested, well-understood recovery plan that turns a mistake into a disaster."

---

Building a Genuine DR Culture#

A closing, culture-level synthesis, directly extending the blameless postmortem culture from the SRE Fundamentals series to this entire series' subject matter.

Diagram

---

The Complete DR Plan Template#

A genuinely practical, consolidated template — worth having this exact shape ready to sketch from memory, since "walk me through how you'd build a DR plan for a new system" is a very common, direct senior-level interview request.

Copy
# Disaster Recovery Plan: [System Name]

## 1. Business Impact Analysis
- What does this system do, and who depends on it?
- What's the REAL cost of downtime? (revenue, reputation,
  regulatory/compliance exposure)
- Explicitly stated RPO: ___
- Explicitly stated RTO: ___

## 2. Chosen DR Strategy (Part 1)
- Backup & Restore / Pilot Light / Warm Standby / Multi-Site
  Active-Active — and WHY this one, given the RPO/RTO above

## 3. Dependency Map (Part 2)
- Direct dependencies: ___
- Transitive dependencies: ___
- External/third-party dependencies: ___
- For EACH: does it ALSO have its own DR plan/failover?

## 4. Runbook (Part 2)
- Trigger criteria
- Declaration authority
- Step-by-step recovery procedure, with EXACT commands
- Verification steps
- SEPARATE failback procedure

## 5. Communication Plan (Part 2)
- Internal: channel INDEPENDENT of the systems being recovered
- External: status page, hosted INDEPENDENTLY
- Regulatory/legal: any mandatory disclosure triggers?

## 6. Testing Schedule (Part 2)
- Tabletop: quarterly
- Partial failover: semi-annually
- Full failover: annually (or more often, for critical systems)
- Re-test triggers: after any significant architecture change

## 7. Ownership
- Plan owner: ___
- Last reviewed: ___
- Last ACTUALLY tested: ___ (not just reviewed — tested)

---

A Final Pre-Disaster Checklist#

A last, practical, memorable checklist — genuinely useful both as real guidance and as a strong closing answer to "what would you check before signing off that a system is DR-ready."

Diagram

---

How This Whole Course Fits Together#

A final, closing synthesis — genuinely worth internalizing as the single overarching narrative connecting every topic in this entire course.

Diagram

The single sentence worth closing this entire course on: every topic covered has been building toward the same underlying goal — keeping a promise to users about how reliable a system will be, and Disaster Recovery is simply that same promise extended to cover the worst, most extreme case, planned for deliberately and in advance, rather than improvised under real, uncontrolled pressure.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: An interviewer asks you to name a real-world disaster/incident and explain what your organization would do differently as a result. Walk through a strong answer using GitLab's 2017 incident.

Answer: I'd reference GitLab's 2017 incident specifically: an engineer accidentally deleted production data while troubleshooting replication, and the team discovered that of five theoretically redundant backup mechanisms, four weren't actually working — recovery relied on a lucky manual snapshot, with real, permanent data loss for the gap since that snapshot. What I'd do differently: treat "we have multiple backup mechanisms" as meaningless until each one has been individually, regularly verified through an actual restore test — exactly the "untested backup is a hypothesis" principle from this series — and I'd add explicit safeguards (confirmation prompts, environment-name verification) to any command capable of deleting production data, since the root trigger was ultimately a human running a legitimate command against the wrong target.

Problem 2: How does the 2021 Facebook/Meta BGP outage change how you'd think about designing an internal "break glass" emergency access procedure?

Answer: That incident showed that even a company with immense engineering resources can end up locked out of their own systems — including internal tools and even physical building access — when a catastrophic failure takes down the exact infrastructure that recovery normally depends on. I'd design break-glass access to be genuinely, deliberately independent of primary production infrastructure — for example, out-of-band access credentials that don't rely on internal SSO tied to the same systems, physical access mechanisms that don't depend purely on networked badge readers, and internal communication channels (Part 2 of this series) hosted on completely separate infrastructure — specifically so a catastrophic failure of the primary environment can never also take down the team's ability to actually respond to and fix it.

Problem 3: A junior engineer argues "our DR plan only needs to cover infrastructure failures like a data center outage, since that's what 'disaster recovery' really means." Using real case studies, how would you respond?

Answer: I'd point to Knight Capital as a direct counterexample — a purely software/deployment-process failure, with zero infrastructure or hardware failure involved at all, caused a genuinely business-ending financial disaster in about 45 minutes. I'd also note that GitLab's 2017 incident and the AWS S3 outage were both ultimately triggered by human/operational error, not physical infrastructure failure, even though the S3 outage's IMPACT was infrastructure-shaped. The lesson across essentially every well-known, real-world case study: disasters are far more often caused by human error, deployment mistakes, and hidden dependencies than by data centers physically failing — a DR plan scoped only to hardware/infrastructure failure misses the majority of what actually causes real, severe incidents in practice.

---

Summary — The Complete Disaster Recovery & Business Continuity Series#

• GitLab (2017): an accidental production database deletion revealed that five theoretically redundant backup mechanisms were mostly non-functional — the defining, industry-wide lesson that untested backups provide false confidence, not real protection.
• AWS S3 us-east-1 (2017): a mistyped command's cascading impact revealed how deeply and invisibly the broader internet depended on one region's infrastructure — reinforcing that hidden, unmapped dependencies and genuine region-wide failures are both real, not theoretical risks.
• Facebook/Meta (2021): a BGP misconfiguration locked the company out of its own internet-facing infrastructure — and, remarkably, out of much of its own internal tooling and even physical access — the definitive lesson that recovery mechanisms must never depend entirely on the exact systems they're meant to recover.
• Knight Capital: a deployment process failure with zero infrastructure involvement caused a genuinely business-ending financial disaster in under an hour — proof that business continuity planning must cover human/process failures, not only physical infrastructure ones.
• The common thread across all four: none were caused by a fire or flood — every one was human/process error made catastrophic by a genuine preparedness gap, not by the initial mistake alone.
• A genuine DR culture treats testing gaps as valuable findings, tracks DR readiness as an ongoing practice (like an SLO), and requires real, sustained leadership support for regular testing — not a document written once and filed away.
• The complete DR plan template — business impact analysis, chosen strategy, dependency map, runbook, communication plan, testing schedule, and clear ownership — ties every concept from this entire three-part series into one practical, reusable structure.

This completes the Disaster Recovery & Business Continuity series — and with it, the full course. See questions.md in this folder for the full interview question bank covering all three parts.