# Incident Management — Part 3: MTTR/MTTD & Chaos Engineering

> **Series:** Incident Management (3 of 3)
> **Part 1:** `01-oncall-and-severity.md` — On-Call Design & Severity Levels
> **Part 2:** `02-incident-response-process.md` — Incident Response Process & Roles
> **Part 3:** This file — MTTR/MTTD & Chaos Engineering
> **Questions:** `questions.md`

## Table of Contents

1. [Why Measure Incident Response At All](#why-measure-incident-response-at-all)
2. [The Core Metrics, Defined Simply](#the-core-metrics-defined-simply)
3. [Breaking Down MTTR Into Its Real Phases](#breaking-down-mttr-into-its-real-phases)
4. [Why "Average" Can Be a Misleading Word Here Too](#why-average-can-be-a-misleading-word-here-too)
5. [What Actually Moves Each Metric](#what-actually-moves-each-metric)
6. [A Worked Example: Improving MTTR Over a Year](#a-worked-example-improving-mttr-over-a-year)
7. [Why Practicing for Incidents Matters](#why-practicing-for-incidents-matters)
8. [Chaos Engineering — The Core Idea](#chaos-engineering--the-core-idea)
9. [The History: Chaos Monkey and the Simian Army](#the-history-chaos-monkey-and-the-simian-army)
10. [The Principles of Chaos Engineering](#the-principles-of-chaos-engineering)
11. [Running a Chaos Experiment, Step by Step](#running-a-chaos-experiment-step-by-step)
12. [Chaos Engineering Tools in Practice](#chaos-engineering-tools-in-practice)
13. [GameDays — Chaos Engineering for Humans, Not Just Systems](#gamedays--chaos-engineering-for-humans-not-just-systems)
14. [DiRT — Google's Disaster Recovery Testing](#dirt--googles-disaster-recovery-testing)
15. [Common Mistakes](#common-mistakes)
16. [Worked Practice Problems](#worked-practice-problems)
17. [Summary — The Complete Incident Management Series](#summary--the-complete-incident-management-series)

---

## Why Measure Incident Response At All

Parts 1 and 2 covered *how* to run a good incident response. This part asks: **how do you actually know if it's working, and getting better over time — not just based on gut feeling?**

```mermaid
graph LR
    Gut["'I FEEL like we're<br/>getting better at<br/>handling incidents'"] --> GutProb["❌ Not verifiable, not<br/>trackable, no way to<br/>prove improvement to<br/>leadership or catch<br/>a regression"]

    Metrics["'Our MTTR dropped from<br/>45 minutes to 18 minutes<br/>over the last two<br/>quarters'"] --> MetricsGood["✅ Concrete, trackable,<br/>and tells you WHETHER<br/>your process changes<br/>actually worked"]
```

---

## The Core Metrics, Defined Simply

```mermaid
graph TD
    Metrics[Core Incident Metrics] --> MTTD["MTTD: Mean Time To Detect —<br/>how long from when<br/>something ACTUALLY broke<br/>to when someone/something<br/>NOTICED"]
    Metrics --> MTTA["MTTA: Mean Time To<br/>Acknowledge — how long<br/>from the ALERT firing to a<br/>human confirming 'I've got<br/>this'"]
    Metrics --> MTTR["MTTR: Mean Time To Resolve<br/>(sometimes 'Repair' or<br/>'Recover') — how long from<br/>detection to the issue<br/>actually being fixed"]
    Metrics --> MTBF["MTBF: Mean Time Between<br/>Failures — how often<br/>incidents happen AT ALL"]
```

**A simple analogy that ties them together:** imagine a smoke alarm in your house. MTTD is how long the fire burned before the alarm went off. MTTA is how long between the alarm going off and you actually getting out of bed to check. MTTR is how long from getting up to actually putting the fire out. MTBF is simply how often fires start in your house in the first place.

| Metric | What It Measures | What Improving It Signals |
|---|---|---|
| **MTTD** | Detection speed | Better monitoring/alerting (Observability series) |
| **MTTA** | Response speed | Better on-call design, escalation policy (Part 1) |
| **MTTR** | Fix speed | Better runbooks, tooling, process (Part 2) |
| **MTBF** | How often things break | Better underlying reliability engineering (Reliability & Architecture Patterns series) |

---

## Breaking Down MTTR Into Its Real Phases

A genuinely important, frequently-tested nuance: **"MTTR" as a single number hides a lot of useful detail.** A sharp interview answer breaks it down into its real, distinct phases — because each phase is improved by a completely different kind of work.

```mermaid
graph LR
    Detect["Detection<br/>(MTTD)"] --> Ack["Acknowledgment<br/>(MTTA)"]
    Ack --> Diagnose["Diagnosis:<br/>figuring out<br/>WHAT'S wrong"]
    Diagnose --> Fix["Fix:<br/>actually APPLYING<br/>the mitigation"]
    Fix --> Verify["Verification:<br/>confirming it's<br/>ACTUALLY resolved"]
```

**Why breaking it down matters practically:** if your overall MTTR is high, "improve MTTR" is too vague to act on. But if you can see that **Diagnosis** specifically is consistently the slowest phase (e.g., engineers spend 30 of your 40-minute average incident just figuring out *what's* broken), that tells you exactly where to invest — better dashboards, better tracing (Observability series), more complete runbooks — rather than, say, focusing on faster deploy/rollback tooling, which would speed up the **Fix** phase but do nothing for the actual bottleneck.

```mermaid
graph TD
    A["Total MTTR = 40 minutes"] --> B["Detection: 2 min"]
    A --> C["Acknowledgment: 3 min"]
    A --> D["Diagnosis: 30 min ⚠️<br/>(the REAL bottleneck)"]
    A --> E["Fix + Verify: 5 min"]
```

---

## Why "Average" Can Be a Misleading Word Here Too

This directly reuses the exact "averages hide outliers" lesson from the Monitoring Methodologies series — worth explicitly connecting the two topics in an interview, since it's the same underlying statistical trap.

```mermaid
graph TD
    A["10 incidents: 9 resolved<br/>in 10 minutes, 1 took<br/>6 hours (360 minutes)"] --> B["Average (Mean) MTTR:<br/>≈ 45 minutes — looks<br/>'not too bad'"]
    A --> C["MEDIAN: 10 minutes<br/>(the typical case)<br/>+ the one 6-hour outlier<br/>is a SEPARATE, important<br/>story worth investigating<br/>on its own"]
```

**A strong interview line:** "Just like with latency percentiles, I'd rather look at the median MTTR (the typical case) alongside the worst outliers specifically, rather than a single blended average — a handful of very long, painful incidents can make the average look moderately bad while hiding the fact that most incidents actually resolve quickly, and vice versa, a few very fast trivial incidents can make the average look great while masking a real, recurring slow-resolution problem."

---

## What Actually Moves Each Metric

A practical, concrete table — genuinely useful both for real work and for structuring interview answers about "how would you improve incident response."

| Metric | Concrete Levers | Ties Back To |
|---|---|---|
| **MTTD** | Better symptom-based alerting, lower alert latency, synthetic monitoring | Observability series, Alerting Design |
| **MTTA** | Reliable escalation policies, on-call sustainability (fewer burned-out people ignoring pages) | Part 1 of this series |
| **MTTR — Diagnosis phase** | Better dashboards, distributed tracing, complete runbooks, practiced familiarity (chaos engineering, below) | Observability series, Monitoring Methodologies series |
| **MTTR — Fix phase** | Fast, safe rollback tooling, feature flags, well-tested automation | Automation/CI-CD tutorial |
| **MTBF** | Actual reliability engineering — redundancy, circuit breakers, better testing | Reliability & Architecture Patterns series |

---

## A Worked Example: Improving MTTR Over a Year

A realistic, narrated case, exactly the kind of concrete story worth having ready for a behavioral interview question like "tell me about a time you improved an operational metric."

**Q1 baseline:** MTTR averages 52 minutes. A phase breakdown shows Diagnosis alone averages 38 minutes — the clear bottleneck.

**Q2 investment:** the team builds standardized RED dashboards (Monitoring Methodologies series) for every service, so on-call no longer has to manually piece together ad hoc queries during an incident. Diagnosis phase drops to 22 minutes; overall MTTR drops to 34 minutes.

**Q3 investment:** the team adds distributed tracing (Observability series) for their top five services, letting on-call jump straight from "error rate is up" to "here's exactly which downstream call is slow" in one step instead of manually correlating logs. Diagnosis drops further to 12 minutes; overall MTTR drops to 22 minutes.

**Q4 investment:** the team writes and links runbooks (Part 2) for the ten most common alert types, cutting the "what do I even try first" hesitation at the very start of diagnosis. Overall MTTR drops to 15 minutes.

```mermaid
graph LR
    Q1["Q1: MTTR 52 min<br/>(Diagnosis: 38 min)"] --> Q2["Q2: MTTR 34 min<br/>(better dashboards)"]
    Q2 --> Q3["Q3: MTTR 22 min<br/>(distributed tracing)"]
    Q3 --> Q4["Q4: MTTR 15 min<br/>(runbooks)"]
```

**Why this story works so well in an interview:** it demonstrates the exact "break the metric into phases, find the real bottleneck, invest specifically there, remeasure" discipline — not just "we tried to be faster" in the abstract.

---

## Why Practicing for Incidents Matters

Here's an uncomfortable truth: **the first time your team ever actually exercises your incident response process shouldn't be during a real, high-stakes production outage.** Just like a fire drill, practicing the process when nothing is actually on fire is how you find the gaps safely.

```mermaid
graph TD
    NoPractice["NEVER practiced —<br/>first real test IS a<br/>genuine production<br/>outage"] --> NoPracticeRisk["❌ Discover process gaps<br/>(missing runbook, unclear<br/>escalation, confused roles)<br/>DURING the worst possible<br/>moment to discover them"]

    Practice["Regularly practiced<br/>(via chaos engineering<br/>and GameDays, below)"] --> PracticeGood["✅ Discover and fix process<br/>gaps in a SAFE, controlled<br/>setting, so the real<br/>incident goes smoothly"]
```

This is exactly the bridge into **chaos engineering** — deliberately, safely breaking things on purpose, specifically to find and fix these gaps before a real, uncontrolled failure finds them for you.

---

## Chaos Engineering — The Core Idea

**Chaos engineering** is the practice of deliberately injecting controlled failures into a system — killing a server, cutting network access, adding artificial latency — to verify the system (and the *people* responding to it) actually behave the way you *believe* they will, before a real, unplanned failure forces the question.

```mermaid
graph TD
    Belief["'We believe our system<br/>can survive one database<br/>replica dying, thanks to<br/>our redundancy setup'"] --> Q{"Have we ever ACTUALLY<br/>tested this belief?"}
    Q -->|No| Risk["❌ It's just an assumption —<br/>untested redundancy has<br/>failed to actually work<br/>in MANY real, documented<br/>incidents"]
    Q -->|"Yes — via a controlled<br/>chaos experiment"| Confidence["✅ It's now a VERIFIED fact,<br/>with real evidence"]
```

**A simple analogy:** it's the difference between *assuming* your car's spare tire is inflated and ready to go, versus actually checking it in your driveway before you need it on the side of a highway during a storm.

---

## The History: Chaos Monkey and the Simian Army

**Chaos Monkey**, created at Netflix around 2011, is the tool that popularized this entire practice, and knowing this origin story is genuinely useful, high-signal interview context.

```mermaid
graph TD
    Netflix["Netflix's problem: moving<br/>to AWS cloud infrastructure,<br/>where individual instances<br/>fail CONSTANTLY and<br/>unpredictably"] --> Insight["Insight: rather than hoping<br/>instance failures don't<br/>happen, deliberately CAUSE<br/>them, randomly, DURING<br/>business hours, when<br/>engineers are alert and<br/>ready to respond"]
    Insight --> ChaosMonkey["Chaos Monkey: randomly<br/>terminates production<br/>instances, on a schedule,<br/>as standard practice"]
```

**Why "during business hours" is such an important, specific detail:** Netflix deliberately ran Chaos Monkey during normal working hours precisely *so that* engineers were alert, present, and could observe and fix any real gaps it uncovered immediately — running it at 3 AM unattended would defeat the entire learning purpose of the exercise.

The **Simian Army** extended this idea into a whole family of tools, each testing a different kind of failure — worth knowing a couple of the names by heart:

| Tool | What It Tests |
|---|---|
| **Chaos Monkey** | Random instance termination |
| **Latency Monkey** | Injects artificial delays into service calls |
| **Chaos Kong** | Simulates an entire AWS *region* failing (the most extreme version) |

---

## The Principles of Chaos Engineering

A more formalized version of this practice (published by Netflix engineers as "The Principles of Chaos Engineering") gives a repeatable, structured method — genuinely worth citing by name in an interview.

```mermaid
flowchart TD
    A["1. Define 'steady state' —<br/>what does NORMAL,<br/>healthy behavior<br/>actually look like?<br/>(usually your SLI/RED<br/>metrics)"] --> B["2. Hypothesize that steady<br/>state will CONTINUE, even<br/>through the specific<br/>failure you're about<br/>to inject"]
    B --> C["3. Inject a REAL-WORLD<br/>failure (kill a server,<br/>add latency, drop network<br/>traffic)"]
    C --> D["4. Try to DISPROVE the<br/>hypothesis by comparing<br/>steady state BEFORE vs<br/>DURING the experiment"]
    D --> E["5. Learn — either the<br/>hypothesis held (confidence<br/>gained) or it didn't<br/>(a real gap found, safely,<br/>before a real outage<br/>found it for you)"]
```

**Why "steady state" is defined using your existing SLI/RED metrics, specifically, and not a new metric invented just for this:** it directly ties chaos engineering back to the SRE Fundamentals and Monitoring Methodologies series — you're not measuring something new and unfamiliar, you're checking whether the metrics you already trust and watch every day hold steady through the injected failure.

---

## Running a Chaos Experiment, Step by Step

A concrete, worked example — testing whether the checkout service actually survives losing one database replica, exactly as claimed in the redundancy design from the Reliability & Architecture Patterns series.

```mermaid
sequenceDiagram
    participant Team as Engineering Team
    participant Chaos as Chaos Tool
    participant DB as Database Replica
    participant Dash as SLI Dashboard

    Team->>Team: Define steady state:<br/>checkout error rate < 0.1%,<br/>p99 latency < 400ms
    Team->>Team: Hypothesis: killing ONE<br/>replica should NOT<br/>affect steady state
    Team->>Chaos: Start SMALL — target<br/>1% of traffic first,<br/>during business hours,<br/>with a rollback plan ready
    Chaos->>DB: Terminate one<br/>database replica
    Team->>Dash: Monitor error rate<br/>and latency CLOSELY
    alt Steady state holds
        Dash-->>Team: ✅ Error rate/latency<br/>unchanged — hypothesis<br/>CONFIRMED, real confidence gained
    else Steady state breaks
        Dash-->>Team: ❌ Error rate spiked!<br/>Hypothesis DISPROVED —<br/>a REAL gap found safely
        Team->>Chaos: ABORT experiment<br/>immediately (rollback plan)
    end
```

---

## Chaos Engineering Tools in Practice

```bash
# Chaos Mesh — a popular open-source, Kubernetes-native
# chaos engineering platform

# Example: kill a random pod matching a label, to test
# whether the service actually recovers automatically
cat <<EOF | kubectl apply -f -
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: checkout-pod-kill
  namespace: checkout
spec:
  action: pod-kill
  mode: one
  selector:
    labelSelectors:
      app: checkout-service
  scheduler:
    cron: "@every 10m"
EOF
```

```bash
# Example: inject 500ms of network latency into calls
# from checkout-service to the payment-service
cat <<EOF | kubectl apply -f -
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-latency-test
  namespace: checkout
spec:
  action: delay
  mode: all
  selector:
    labelSelectors:
      app: checkout-service
  delay:
    latency: "500ms"
  duration: "5m"
  target:
    selector:
      labelSelectors:
        app: payment-service
EOF
```

**Other widely-used tools worth knowing by name:** **Gremlin** (a polished, commercial chaos engineering platform, often used for org-wide GameDays), **Litmus** (another popular Kubernetes-native, open-source option), and AWS's own **Fault Injection Simulator (FIS)** for cloud-native fault injection without needing a separate tool.

---

## GameDays — Chaos Engineering for Humans, Not Just Systems

Everything above tests whether **systems** behave as expected under failure. A **GameDay** extends the same idea to test whether **people and process** hold up too — a scheduled, deliberate exercise where a team responds to a simulated incident as if it were completely real.

```mermaid
graph TD
    GameDay["GameDay Exercise"] --> G1["A facilitator secretly<br/>injects a realistic failure<br/>(or simulates one)"]
    GameDay --> G2["The on-call team responds<br/>EXACTLY as they would for<br/>a real incident — declares<br/>severity, assigns IC, follows<br/>runbooks, communicates"]
    GameDay --> G3["Afterward: a retrospective —<br/>what worked, what didn't,<br/>EXACTLY like a blameless<br/>postmortem"]
```

**Why GameDays matter specifically, beyond pure technical chaos experiments:** a technical chaos experiment might prove "the system recovers automatically in 30 seconds." A GameDay tests something different and equally important: "does the team actually know the escalation policy? Does the Incident Commander role get filled correctly? Is the runbook actually findable and accurate under pressure?" **This directly exercises everything from Parts 1 and 2 of this series — on-call, severity classification, roles, communication — not just the underlying infrastructure.**

---

## DiRT — Google's Disaster Recovery Testing

Worth knowing by name as a well-documented, real-world example of this practice taken to an even larger scale: Google's internal **DiRT (Disaster Recovery Testing)** program runs company-wide, coordinated exercises simulating major failures — sometimes including deliberately taking down real internal tools and services (with appropriate safeguards) to test organization-wide resilience, not just a single team's runbook.

**Why this is worth citing specifically in an interview:** it demonstrates that this practice scales beyond a single team's chaos experiments into genuine organizational muscle-memory — the same underlying principle (test your assumptions safely, before reality tests them for you) applied at the scale of an entire company's operational readiness.

---

## Common Mistakes

| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Tracking only a single, blended MTTR average | Hides whether most incidents are fast with a few painful outliers, or a consistent slow problem — very different situations needing very different fixes | Break down by phase (detect/acknowledge/diagnose/fix), and look at median plus worst outliers, not just the mean |
| "Improve MTTR" as a vague, unbroken goal | No clear, actionable lever to actually invest in | Identify which specific phase (usually Diagnosis) is the real bottleneck, and invest there specifically |
| Running chaos experiments unannounced, at random, with no rollback plan | Can turn a controlled learning exercise into an actual, uncontrolled outage | Start small (a tiny % of traffic), during business hours, with a clear abort/rollback plan ready |
| Only testing systems, never practicing the human process | A perfectly resilient system can still suffer a slow, chaotic response if the team has never actually practiced roles, escalation, and communication | Run GameDays specifically to exercise Parts 1 and 2 of this series — on-call, roles, communication — not just infrastructure |
| Treating a chaos experiment's failure ("the hypothesis was disproved") as a bad outcome | This is precisely the valuable, intended result — a real gap found safely | Treat a disproved hypothesis as a genuine win: exactly the same blameless, learning-focused framing as a postmortem |
| Never practicing at all, and discovering process gaps for the first time during a real, high-stakes incident | The worst possible moment to discover a missing runbook or unclear escalation path | Regularly scheduled chaos experiments and GameDays, treated as standard operational practice, not an occasional afterthought |

---

## Worked Practice Problems

**Problem 1:** Your team's MTTR has held steady at 35 minutes for the last three quarters, despite investing in faster deploy/rollback tooling each quarter. What would you investigate, and why might the investment not have helped?

*Answer:* I'd break MTTR down by phase before concluding the tooling investment failed — if the real bottleneck is actually the Diagnosis phase (figuring out *what's* wrong), not the Fix phase (applying the fix once known), then faster rollback tooling wouldn't move the needle at all, since it only speeds up a phase that was never the bottleneck in the first place. I'd measure the phase breakdown directly, and if Diagnosis is indeed the real culprit, redirect investment toward better dashboards, tracing, and runbooks instead.

**Problem 2:** A team wants to run their first-ever chaos experiment: killing a production database replica, to test their claimed redundancy. What would you recommend for how to run this safely, and why?

*Answer:* Start small and controlled: run it during business hours (so engineers are alert and ready to respond, exactly as Netflix originally designed Chaos Monkey), target a small percentage of traffic or a non-critical replica first rather than going straight for the primary database's full failover path, have a clear, pre-agreed abort/rollback plan ready before starting, and define the "steady state" metrics (error rate, latency) to watch beforehand so there's an objective, agreed-upon signal for whether the hypothesis held or the experiment needs to be aborted immediately.

**Problem 3:** After a real production incident, the postmortem reveals the team's actual escalation policy was different from what was documented, and the Incident Commander role sat unfilled for the first 15 minutes because nobody was sure whose job it was. The system itself recovered fine once engineers engaged. What kind of practice would have caught this gap beforehand, and why wouldn't a pure technical chaos experiment have found it?

*Answer:* A GameDay exercise specifically, not a pure technical chaos experiment — a technical experiment (like killing a server) tests whether the *system* recovers, but this gap was entirely about the *human process*: an out-of-date escalation policy and unclear IC ownership. Only an exercise that requires the team to actually go through declaring severity, filling roles, and following the real (or realistically simulated) escalation process — exactly what a GameDay is designed to do — would have surfaced that the documented process didn't match reality, safely, before a real incident exposed the same gap under actual pressure.

---

## Summary — The Complete Incident Management Series

- **MTTD, MTTA, MTTR, and MTBF** each measure a different phase of incident response, and each is improved by a different kind of investment — detection by better alerting, acknowledgment by better on-call design, resolution by better tooling/process, and frequency by genuine reliability engineering.
- **Break MTTR down into its real phases** (detect, acknowledge, diagnose, fix, verify) rather than treating it as one blended number — this is usually the difference between a vague "let's be faster" goal and a concrete, effective investment.
- Just like latency percentiles, **watch the median and worst outliers, not just the average** — a single blended MTTR can hide either a few very painful incidents or a consistent slow problem, and the fix for each is completely different.
- **Chaos engineering** deliberately injects controlled failures to verify a system actually behaves the way you believe it does, following a repeatable method: define steady state, hypothesize it holds, inject a real failure, try to disprove the hypothesis, and learn from the result either way.
- **Chaos Monkey** (Netflix, ~2011) popularized this practice, deliberately run during business hours so engineers could observe and respond to real gaps safely.
- **GameDays** extend the same principle to test the **human process** (on-call, roles, communication, from Parts 1 and 2 of this series) — not just infrastructure — and Google's **DiRT** program shows this practice scaled to an entire organization.
- The unifying theme across this whole series: **it's far better to discover a gap — in your systems, your process, or your people — during a safe, controlled, deliberately-scheduled exercise than during the worst possible moment: a real, unplanned outage.**

This completes the **Incident Management** series. See `questions.md` in this folder for the full interview question bank covering all three parts.