<style> body, p, li { text-align: justify; } </style>

Interview Questions: Incident Management

Companion question bank for the 3-part tutorial series in this folder: 01-oncall-and-severity.md, 02-incident-response-process.md, 03-metrics-and-chaos-engineering.md.

Answers are short and plain — expand out loud using the diagrams and worked examples in the tutorials.

---

Part 1 Questions: On-Call & Severity

1. What's the point of an on-call rotation?#

It guarantees a real, accountable human is always reachable when something breaks, and spreads that disruption fairly across a team over time instead of resting on one person permanently.

2. What's the most common on-call rotation length, and why?#

About one week — long enough to build some continuity with what's going on, short enough that no one person carries the disruption for too long. Shorter rotations reduce individual burden but add more handoffs; much longer rotations raise real burnout risk.

3. What is a follow-the-sun rotation?#

On-call responsibility hands off between teams in different time zones so it always lands on whoever is in their normal waking hours — nobody gets paged at 3 AM their own local time. It requires a genuinely distributed team and disciplined handoffs.

4. Why have both a primary and a secondary on-call?#

So a single unreachable person (dead phone, dead zone, deep sleep) can't leave a page unanswered indefinitely — an automatic, timed escalation moves it to the secondary if the primary doesn't acknowledge in time.

5. Walk through a typical severity framework (SEV1-SEV4).#

SEV1: complete outage, major data loss, or active security breach — all-hands, immediate. SEV2: significant but partial impact — major feature down or a subset of users affected. SEV3: minor impact with a workaround, handled at standard priority. SEV4: cosmetic or internal-only, fixed whenever convenient.

6. Is it a problem if an incident's severity changes partway through?#

No — it's normal and expected as facts become clearer. Getting the initial classification slightly wrong isn't a failure; refusing to update it as new information arrives is.

7. How does severity typically relate to error budget impact?#

A SEV1 usually burns a large, alarming chunk of the error budget quickly — exactly what a critical burn-rate alert is designed to catch. A SEV4 often has close to zero measurable SLO impact at all.

8. What should a healthy on-call program track about sustainability?#

Paging frequency and load per individual person, not just per team — a consistently overloaded person is a concrete signal something needs to change (real reliability work, or better-tuned alerting), not something to just "push through."

---

Part 2 Questions: Incident Response Process & Roles

9. Why do incidents often go worse when one skilled person tries to do everything alone?#

Constant context-switching between diagnosing, fixing, and answering worried questions is slower and more error-prone than each task getting focused attention. Structured roles let each person stay in one lane.

10. What does an Incident Commander actually do, and why shouldn't it usually be the most senior technical expert?#

They coordinate the response and make calls under uncertainty — not necessarily fix anything themselves. If the most knowledgeable engineer also runs IC, their attention splits between deep technical work and coordination, and both suffer. A separate person handling IC lets the expert stay fully focused on the fix.

11. Name the other common incident roles besides Incident Commander.#

Operations/Technical Lead (drives the actual fix), Communications Lead (writes and sends status updates), and Scribe (keeps a timestamped log of what happened — the raw material for the later postmortem).

12. Why is the Scribe role easy to skip, and why is skipping it costly?#

"Write everything down" feels like the least urgent task mid-incident. But without it, the postmortem gets written later from fuzzy memory, producing a shallow root-cause analysis that never reaches the real systemic cause.

13. Why use one dedicated incident channel instead of letting people DM the responder directly?#

Individual DMs fragment the responder's attention across repeated, duplicate "any update?" questions. One channel with scheduled updates lets stakeholders stay informed without interrupting.

14. What's the difference between mitigating and resolving an incident?#

Mitigating stops the user-facing pain right now (a rollback, a failover) — it doesn't mean the root cause is actually fixed. Resolving means confirming the user-facing metric has genuinely, durably recovered. Declaring resolution the instant a fix is applied, without confirming recovery, is a common risky shortcut.

15. How should external, customer-facing status updates differ from internal ones?#

Internal updates can be technical and detailed ("a connection pool leak from the 14:02 deploy"). External updates need calm, plain language focused on impact and expected resolution ("we've identified the cause and are deploying a fix") — most customers don't know what a connection pool is.

16. What's the difference between a runbook and a playbook?#

A runbook is specific, step-by-step instructions for one known problem (exact commands, exact thresholds). A playbook is broader guidance for a category of situation too varied to fully script (like a data breach response), requiring more judgment to adapt.

17. What should a good shift-change handoff include?#

Current status/severity, what's already been tried (and specifically ruled out), the current leading hypothesis, what's actively in progress, and who else is involved and reachable — skipping this means the next responder wastes time re-investigating already-ruled-out leads.

---

Part 3 Questions: MTTR/MTTD & Chaos Engineering

18. Define MTTD, MTTA, MTTR, and MTBF in plain terms.#

MTTD: how long from something actually breaking to someone noticing. MTTA: how long from the alert firing to a human acknowledging it. MTTR: how long from detection to actually fixing it. MTBF: how often incidents happen at all.

19. Why is breaking MTTR into phases more useful than tracking one number?#

"Improve MTTR" alone gives no clear lever to pull. Breaking it into detect/acknowledge/diagnose/fix/verify shows exactly which phase is the real bottleneck (often Diagnosis) — so you invest in the right fix (better dashboards/tracing) instead of the wrong one (faster rollback tooling that doesn't touch the actual bottleneck).

20. Why should you look at median and worst-case MTTR, not just the average?#

Same reason as latency percentiles — a few very long, painful incidents can make the average look moderately bad while hiding that most incidents actually resolve fast, and vice versa. The median shows the typical case; the outliers deserve their own separate investigation.

21. What is chaos engineering, in one sentence?#

Deliberately injecting controlled failures into a system to verify it (and the people responding to it) actually behaves the way you believe it will, before a real, unplanned failure forces the question.

22. What was Chaos Monkey, and why did Netflix run it during business hours?#

A tool that randomly terminates production instances, created at Netflix as they moved to cloud infrastructure where individual failures are constant and unpredictable. It ran during business hours specifically so engineers were alert and present to observe and fix any real gaps it uncovered immediately — running it unattended overnight would defeat the learning purpose.

23. Walk through the five steps of a formal chaos experiment.#

Define steady state (using existing SLI/RED metrics), hypothesize it will hold through a specific injected failure, inject a real-world failure, try to disprove the hypothesis by comparing steady state before vs. during, and learn from the result either way.

24. Why does chaos engineering use your existing SLI/RED metrics for "steady state" instead of a new metric invented just for the experiment?#

It ties the experiment directly to the metrics you already trust and watch every day, rather than measuring something unfamiliar — you're checking whether normal, everyday health holds through the injected failure.

25. What's a GameDay, and how is it different from a pure technical chaos experiment?#

A scheduled exercise where a team responds to a simulated incident as if it were real — testing whether people and process (on-call, roles, communication, runbooks) hold up, not just whether the system technically recovers. A technical chaos experiment can prove the system self-heals in 30 seconds while completely missing that nobody actually knew who was supposed to be Incident Commander.

26. What is Google's DiRT program?#

A company-wide disaster recovery testing program that runs coordinated exercises simulating major failures across the organization, sometimes deliberately taking down real internal tools (with safeguards) — the same core principle (test assumptions safely before reality tests them) scaled to an entire company.

---

Quick-Fire / Rapid Recall#