<style> body, p, li { text-align: justify; } </style>

Incident Management — Part 2: Incident Response Process & Roles

Series: Incident Management (2 of 3) Part 1: 01-oncall-and-severity.md — On-Call Design & Severity Levels Part 2: This file — Incident Response Process & Roles Part 3: 03-metrics-and-chaos-engineering.md — MTTR/MTTD & Chaos Engineering Questions: questions.md

Table of Contents#

1. Why Structure Matters More Than Heroics
2. The Incident Lifecycle
3. The Incident Commander Role
4. Other Key Roles
5. A Worked Example: Roles in Action
6. Communication During an Incident
7. The Incident Channel
8. Status Updates — A Simple Template
9. Status Pages — Talking to Customers
10. Runbooks vs Playbooks
11. A Real Worked Runbook
12. Declaring an Incident Resolved
13. The Handoff Problem
14. Common Mistakes
15. Worked Practice Problems
16. Summary and What's Next

---

Why Structure Matters More Than Heroics#

A common myth about incident response: the best on-call engineer is the one who single-handedly, brilliantly fixes everything under pressure. In reality, the incidents that go worst are almost always the ones where a talented person tries to do everything alone — diagnosing, fixing, and communicating with worried stakeholders, all at once, with nobody else coordinating.

Diagram

Simple analogy: think of a fire. You don't want the one firefighter closest to the flames also answering phone calls from worried neighbors and directing traffic — that's why a fire scene has a designated incident commander who coordinates, while the firefighter who's actually holding the hose just focuses on putting out the fire.

---

The Incident Lifecycle#

Diagram

A genuinely important distinction worth calling out explicitly: Mitigate is not the same as Resolve. Mitigating means stopping the user-facing pain right now (e.g., a rollback, a manual failover, disabling a broken feature flag) — it doesn't mean the underlying root cause is actually fixed yet. Declaring an incident "resolved" the moment mitigation happens, without confirming impact is truly gone and without following up on the real root cause, is a common and risky shortcut.

---

The Incident Commander Role#

The Incident Commander (IC) is the single most important structural role in incident response — and one of the most commonly asked-about topics in SRE interviews.

Diagram

The single most counter-intuitive, most important fact about this role, worth stating explicitly in an interview: the Incident Commander does not need to be the most senior engineer, or even the person who understands the broken system best. Their job is coordination and decision-making under uncertainty, not technical execution — in fact, having your most senior systems expert also try to run the IC role means they're distracted from the one thing only they can do (deep technical diagnosis), while coordination (which many people can be trained to do well) goes unstaffed.

Diagram

---

Other Key Roles#

Beyond the Incident Commander, mature incident response processes (this exact role structure is closely modeled on Google's and PagerDuty's published incident command frameworks, both worth knowing by name) typically staff a few more distinct roles for anything beyond a small, quick incident:

Diagram

Why the Scribe role is so easy to skip, and so costly when skipped: in the middle of a stressful incident, "write everything down" feels like the least urgent task. But without it, the postmortem (SRE Fundamentals series) is written from fuzzy, incomplete memory hours or days later — exactly the kind of gap that produces a shallow "Five Whys" analysis that never reaches the real systemic cause.

---

A Worked Example: Roles in Action#

A short, realistic transcript-style walkthrough of a SEV1 incident with roles properly staffed:

Diagram

Notice what Jamal (Ops Lead) never had to do: stop and write a status update, or personally answer a worried Slack message from a VP. That's the entire value of having distinct roles — each person stays in exactly one lane, and the incident moves faster as a direct result.

---

Communication During an Incident#

Poor communication during an incident is one of the most common, most avoidable sources of frustration — both internally (engineers getting pinged individually by five different people asking "any update?") and externally (customers left completely in the dark).

Diagram

---

The Incident Channel#

A single, dedicated communication channel (a Slack/Teams channel created specifically for this incident, often auto-created by tooling like PagerDuty's incident workflows) is the standard, practical fix.

Copy
# Many orgs automate this — a simple example using a Slack CLI/API
# to spin up a fresh, dedicated channel the moment a SEV1/SEV2 fires
curl -X POST https://slack.com/api/conversations.create \
  -H "Authorization: Bearer $SLACK_BOT_TOKEN" \
  -d "name=incident-2026-06-01-checkout-outage"

Why a fresh, dedicated channel per incident (rather than reusing a general team channel) matters practically: it keeps the incident's timeline and discussion cleanly separated from normal day-to-day chatter, makes it trivial to find and reference later (for the postmortem), and gives the Comms Lead one obvious, unambiguous place to post updates that everyone knows to watch.

---

Status Updates — A Simple Template#

A consistent, repeatable format for status updates saves real time and cognitive load under pressure — nobody has to think about what to write, just fill in the blanks.

Copy
**[SEV1] Checkout Service Outage — Update #3, 14:22 UTC**

**Status:** Mitigating
**Impact:** ~40% of checkout attempts failing since 14:02 UTC
**What we know:** Root cause identified — a database connection
pool leak from the 14:02 deploy
**What we're doing:** Rolling back the deploy now, ETA 5 minutes
**Next update:** by 14:30 UTC, or sooner if status changes

Why "Next update: by [specific time]" is a small but genuinely important detail: it removes the anxious "should I ask for an update yet, or is that annoying" uncertainty from everyone watching — they know exactly when to expect the next one, and won't feel the need to interrupt before then.

---

Status Pages — Talking to Customers#

For customer-facing incidents, an external status page (tools like Statuspage.io, or a self-hosted equivalent) is the standard way to communicate outward, separate from internal incident channels.

Diagram

A key practical distinction worth naming: the internal update ("a database connection pool leak from the 14:02 deploy") is far too technical and detailed for customers — the external status page update should be something like "We're investigating reports of checkout errors and have identified the cause. A fix is being deployed." Translating internal technical detail into calm, appropriately-scoped external language is genuinely a skill, and it's exactly why a dedicated Communications Lead role (rather than leaving it to whoever's free) matters for anything beyond a minor incident.

---

Runbooks vs Playbooks#

These two terms are often used interchangeably, but there's a genuinely useful distinction worth knowing, and it connects directly back to the "alerts need runbooks" discussion in the Observability series.

Diagram

Interview-ready line: "A runbook is a recipe — follow these exact steps for this exact known problem. A playbook is more like a strategy guide — general principles and key decision points for a category of situation too varied to script exactly. Most day-to-day paging alerts should link to a runbook; things like security incidents or major data-loss events are better served by a playbook, since no two are ever quite identical."

---

A Real Worked Runbook#

Runbook: Checkout Service — High Error Rate

Trigger: Fires when the checkout-service critical burn-rate alert (see Observability series, Part 3) exceeds 14.4x over both the 5-minute and 1-hour windows.

Step 1: Confirm the alert is real — Open the checkout-service RED dashboard (Monitoring Methodologies series). Confirm error rate is actually elevated, not a dashboard/metric pipeline glitch.

Step 2: Check for a recent deploy

Copy
kubectl rollout history deployment/checkout-service -n checkout

If a deploy happened in the last 30 minutes, this is the PRIME suspect — go to Step 3.

Step 3: Roll back if a recent deploy is implicated

Copy
kubectl rollout undo deployment/checkout-service -n checkout

Monitor the RED dashboard for 3-5 minutes to confirm recovery.

Step 4: If no recent deploy, check dependency health — Open the USE dashboard for checkout-service's database and payment gateway dependencies (Monitoring Methodologies series). Look specifically for connection pool saturation.

Step 5: Escalate if not resolved within 15 minutes — Page the secondary on-call (Part 1) and the checkout-service team lead.

Known Related Incidents: 2026-03-14 — Connection pool exhaustion from a config typo (see postmortem link)

Why this exact shape is worth reproducing in an interview if asked to sketch a runbook: it's specific (exact commands, not vague advice), it's ordered by likelihood/ease-of-check first (recent deploy is checked before deep dependency investigation, since it's usually faster to rule in/out), and it explicitly links related past incidents — directly reusing the postmortem discipline from the SRE Fundamentals series as a live, practical resource during a new incident, not just an after-the-fact document.

---

Declaring an Incident Resolved#

A commonly overlooked, practically important step: resolution should be a deliberate, confirmed decision, not just "things look quiet now."

Diagram

A real, common trap worth naming: declaring resolution the instant a fix is applied, rather than waiting to confirm the fix actually worked and the underlying metric has genuinely, durably recovered — a rollback that looks like it fixed things in the first 30 seconds can sometimes be premature (e.g., a cache still serving stale-but-working responses, briefly masking a problem that returns once the cache expires).

---

The Handoff Problem#

For incidents that run long (hours, or span a shift change), transferring context to a new responder without losing critical information is a genuine, practical skill.

Diagram

This is exactly the same discipline as the follow-the-sun on-call handoff from Part 1 — a dropped or rushed handoff means the incoming responder re-investigates things already ruled out, wasting exactly the time a fresh, alert responder was supposed to save.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: During a SEV1, the most senior backend engineer on the team is asked to be Incident Commander, since "they know the system best." Two hours in, the incident still isn't resolved, and the team seems disorganized despite having a very knowledgeable person nominally in charge. What likely went wrong?

Answer: The senior engineer is almost certainly spending their time split between deep technical diagnosis (the thing only they can uniquely do) and coordination/communication duties (writing updates, fielding questions, tracking who's doing what) — doing both worse than if those were separated. The fix: have someone else, even a less senior person trained in incident command, take the IC role specifically so the senior engineer can stay fully focused on the technical fix, while the IC handles coordination and keeps everyone else unblocked and informed.

Problem 2: An incident's Comms Lead sends an external customer-facing status update that says: "We identified a connection pool exhaustion issue in our checkout-service caused by a leaked connection from a recent deployment, and are executing a rollback via kubectl." What's wrong with this message, and how would you rewrite it?

Answer: This is far too technical for an external, customer-facing audience — most customers have no idea what a "connection pool," "kubectl," or even "deployment" means, and this level of detail can come across as either confusing or alarming rather than reassuring. A better version: "We've identified the cause of checkout errors and are deploying a fix now. We expect this to be resolved within the next few minutes." Internal, technical detail belongs in the internal incident channel; external updates should be short, plain-language, and focused on impact and expected resolution.

Problem 3: A long-running SEV2 incident hits a shift change. The outgoing on-call engineer sends a single Slack message saying "still investigating, gtg" before logging off. The incoming engineer spends the first 45 minutes re-discovering things already ruled out. What should the handoff have included instead?

Answer: A proper handoff should have covered: current status/severity, exactly what's already been tried (and specifically what was ruled out, so it isn't re-investigated), the current leading hypothesis for root cause, anything actively in progress, and who else is involved and reachable. Without this, the incoming engineer has no way to know what ground has already been covered, effectively restarting the investigation from zero and wasting the exact time a fresh, alert responder should have saved.

---

Summary and What's Next#

• Structure beats heroics — the incidents that go worst are usually the ones where one person tries to diagnose, fix, and communicate all at once, alone.
• The Incident Commander coordinates and makes calls under uncertainty — critically, this does not need to be (and often shouldn't be) the most technically knowledgeable person, so that expert can stay fully focused on the actual fix.
• Other key roles — Operations/Technical Lead, Communications Lead, Scribe — each protect one person's focus from being fragmented by the others' responsibilities.
• A single, dedicated incident channel with regular, scheduled status updates (using a consistent template, including "next update by [time]") prevents responders from being individually pinged by every worried stakeholder.
• Status pages require translating internal technical detail into calm, plain-language external communication — a distinct skill from writing internal updates.
• Runbooks are specific, step-by-step recipes for known problems; playbooks are broader, judgment-requiring guidance for a category of situation too varied to fully script.
• Declaring an incident resolved should be a deliberate confirmation that the user-facing metric has genuinely, durably recovered — not just "the fix was applied."
• Handoffs (whether shift-change or follow-the-sun) need to explicitly transfer status, what's been tried, current hypothesis, and in-progress work — a dropped handoff wastes exactly the time a fresh responder should save.

Continue to Part 3 (03-metrics-and-chaos-engineering.md) for how to measure whether all of this is actually working (MTTR, MTTD, MTBF) and how to proactively practice incident response before a real incident forces you to, through chaos engineering.