# Incident Management — Part 1: On-Call & Severity Levels > **Series:** Incident Management (1 of 3) > **Part 1:** This file — On-Call Design & Severity Levels > **Part 2:** `02-incident-response-process.md` — Incident Response Process & Roles > **Part 3:** `03-metrics-and-chaos-engineering.md` — MTTR/MTTD & Chaos Engineering > **Questions:** `questions.md` ## Table of Contents 1. [Why This Topic Sits Downstream of Everything Else](#why-this-topic-sits-downstream-of-everything-else) 2. [What On-Call Actually Is](#what-on-call-actually-is) 3. [Designing an On-Call Rotation](#designing-an-on-call-rotation) 4. [Rotation Length — The Real Tradeoff](#rotation-length--the-real-tradeoff) 5. [Follow-the-Sun Rotations](#follow-the-sun-rotations) 6. [Primary and Secondary On-Call](#primary-and-secondary-on-call) 7. [Escalation Policies](#escalation-policies) 8. [A Worked PagerDuty-Style Escalation Policy](#a-worked-pagerduty-style-escalation-policy) 9. [Severity Levels — The Common Framework](#severity-levels--the-common-framework) 10. [Choosing a Severity Level in the Moment](#choosing-a-severity-level-in-the-moment) 11. [On-Call Compensation and Sustainability](#on-call-compensation-and-sustainability) 12. [Protecting On-Call From Burnout](#protecting-on-call-from-burnout) 13. [Common Mistakes](#common-mistakes) 14. [Worked Practice Problems](#worked-practice-problems) 15. [Summary and What's Next](#summary-and-whats-next) --- ## Why This Topic Sits Downstream of Everything Else Every other tutorial in this course builds toward one moment: a person's phone buzzing at 2 AM. The SLOs and error budgets (SRE Fundamentals) decide *what counts as bad enough to page*. The alerting design (Observability, Part 3) decides *when* the page actually fires. This tutorial is about what happens the instant it does — who gets paged, how they're organized, and how everyone agrees on how serious the situation actually is. ```mermaid graph LR SLO["SLO & Error Budget
(SRE Fundamentals)"] --> Alert["Alerting Design
(Observability)"] Alert --> OnCall["On-Call & Severity
(THIS tutorial)"] OnCall --> Response["Incident Response
Process (Part 2)"] Response --> Learn["Postmortems
(SRE Fundamentals)"] ``` --- ## What On-Call Actually Is **On-call** simply means: one specific person is designated as "the one who gets woken up" if something breaks during a specific window of time, so that a page always reaches a real, accountable human, instead of silently going nowhere or waking up an entire team at once. **A simple analogy:** think of a hospital's on-call doctor. Not every doctor in the building has to be reachable every night — one specific doctor is designated the on-call doctor for tonight, and everyone (nurses, the front desk) knows exactly who to call. That's the whole idea. ```mermaid graph TD Without["WITHOUT on-call:
an alert fires at 2 AM —
who's supposed to see it?
Nobody, until someone
happens to check email
the next morning"] --> WithoutBad["❌ Slow response,
unclear ownership"] With["WITH on-call: ONE specific
person is designated —
the alert pages THEM
directly, immediately"] --> WithGood["✅ Fast, clear,
accountable response"] ``` --- ## Designing an On-Call Rotation An on-call **rotation** spreads this responsibility across a team over time, so no single person is "always" on-call. ```mermaid gantt dateFormat YYYY-MM-DD title A Simple Weekly On-Call Rotation (4-person team) section Week 1 Alice on-call :a1, 2026-06-01, 7d section Week 2 Bob on-call :a2, 2026-06-08, 7d section Week 3 Carol on-call :a3, 2026-06-15, 7d section Week 4 Dave on-call :a4, 2026-06-22, 7d ``` **Why a rotation, not just "one designated on-call person forever"?** Because being on-call is genuinely disruptive — interrupted sleep, restricted freedom of movement, mental load even when nothing fires. Spreading it fairly across a team is both a sustainability requirement and, in many places, close to a legal/contractual expectation. --- ## Rotation Length — The Real Tradeoff ```mermaid graph TD Short["SHORT rotation
(e.g. 1 day)"] --> ShortPro["✅ Less disruption per person
per shift"] Short --> ShortCon["❌ Less time to build deep
familiarity with what's
currently happening in
the system; more
handoffs = more chances
to drop context"] Long["LONG rotation
(e.g. 2+ weeks)"] --> LongPro["✅ More continuity, fewer
handoffs, more time to
actually learn the
system deeply"] Long --> LongCon["❌ Much more disruptive
and tiring for the
individual — real burnout
risk over that stretch"] ``` **The most common real-world answer, worth citing:** **one week** is the single most widely used rotation length across the industry — long enough to build some continuity, short enough that no one person carries the load for too long. Shorter rotations (a few days) are more common at smaller teams or for very high-severity/high-frequency-paging services; longer rotations (2+ weeks) are increasingly seen as a burnout risk and are less common in mature orgs. --- ## Follow-the-Sun Rotations For a genuinely global company (or a team distributed across multiple time zones), a clever alternative avoids waking anyone up at 2 AM at all. ```mermaid graph LR US["US Team
(on-call 9am-5pm US time)"] -->|"handoff"| APAC["APAC Team
(on-call during THEIR
daytime, which is US
nighttime)"] APAC -->|"handoff"| EU["EU Team
(on-call during THEIR
daytime)"] EU -->|"handoff"| US ``` **The idea, in plain terms:** as the sun "follows" around the globe, on-call responsibility hands off to whichever region is currently in normal waking/working hours — nobody is ever paged at 3 AM their own local time, because someone else, somewhere, is always awake and covering. **The real cost worth naming:** this requires a genuinely distributed team across multiple regions (not every company has this), plus disciplined, well-documented handoffs between regions (exactly the kind of information transfer problem Part 2 of this tutorial covers) — a dropped handoff means the *next* region inherits an incident with no context. --- ## Primary and Secondary On-Call A common, simple safety net: designate **two** people on-call at once — a **primary** (gets paged first) and a **secondary** (gets paged if primary doesn't respond in time, or can be pulled in to help on a bigger incident). ```mermaid sequenceDiagram participant Alert participant Primary as Primary On-Call participant Secondary as Secondary On-Call Alert->>Primary: Page fires Note over Primary: 5 minutes pass,
no acknowledgment Alert->>Secondary: Escalate automatically —
primary might be asleep,
unreachable, or already
overwhelmed Secondary->>Secondary: Acknowledges and responds ``` **Why this matters practically:** a single point of failure in your *incident response process itself* is a genuine risk — if the one and only on-call person is unreachable (phone died, in a dead zone, deep sleep), a page with no secondary escalation path can simply go unanswered indefinitely. Primary/secondary is the on-call equivalent of the redundancy principle from the Reliability & Architecture Patterns series, applied to *people* instead of servers. --- ## Escalation Policies An **escalation policy** is the automated, pre-defined chain of "who gets paged next, and when" if the current person doesn't respond — directly extending the alert routing concepts from the Observability series' alerting tutorial. ```mermaid flowchart TD A["Alert fires"] --> B["Page Primary On-Call"] B --> C{"Acknowledged within
5 minutes?"} C -->|Yes| D["✅ Primary handles it"] C -->|No| E["Page Secondary On-Call"] E --> F{"Acknowledged within
5 more minutes?"} F -->|Yes| G["✅ Secondary handles it"] F -->|No| H["Escalate to Engineering
Manager / Team Lead"] ``` --- ## A Worked PagerDuty-Style Escalation Policy A realistic, concrete configuration (the exact shape used by tools like PagerDuty, Opsgenie, or Grafana OnCall): ```yaml escalation_policy: name: checkout-service-escalation rules: - escalation_delay_minutes: 5 targets: - type: schedule id: checkout-primary-oncall - escalation_delay_minutes: 5 targets: - type: schedule id: checkout-secondary-oncall - escalation_delay_minutes: 10 targets: - type: user id: engineering-manager - escalation_delay_minutes: 15 targets: - type: user id: director-of-engineering ``` **Worth naming specifically:** most real tools also support **acknowledging** a page (stops it from escalating further, "I've got this") separately from **resolving** it (the incident is actually fixed) — a very common, practical distinction that's easy to overlook if you've never actually used one of these systems. --- ## Severity Levels — The Common Framework The moment an incident is confirmed, the very next question is: **how bad is this, really?** Severity levels give the whole organization a shared, consistent vocabulary for answering that instantly, without a debate. ```mermaid graph TD SEV1["SEV1 (Critical):
Complete outage or major
data loss/security breach.
ALL hands, immediate
response, executive visibility"] --> SEV2 SEV2["SEV2 (High):
Significant, but PARTIAL
impact — a major feature
down, or a subset of
users affected"] --> SEV3 SEV3["SEV3 (Medium):
Minor impact, workaround
exists, doesn't need
immediate all-hands"] --> SEV4 SEV4["SEV4 (Low):
Cosmetic issue, or an
internal-only tool —
fix during business hours"] ``` | Severity | Example | Typical Response | |---|---|---| | **SEV1** | The entire site is down; a payment system is charging customers incorrectly; a data breach is confirmed | Immediate, all-hands, Incident Commander assigned (Part 2), executive/customer communication | | **SEV2** | Checkout is broken for one payment method, but others still work; one region is down but others are fine | Urgent, dedicated response, may not need full exec visibility | | **SEV3** | A non-critical internal dashboard is slow; a minor UI bug affecting a small % of users | Standard priority, fixed within normal work hours | | **SEV4** | A typo in an error message; a cosmetic styling issue | Backlog, fixed whenever convenient | **The direct tie-in to the SRE Fundamentals series, worth stating explicitly:** severity often maps closely to **error budget impact** — a SEV1 typically burns a large, alarming chunk of the error budget very quickly (which is exactly what the "Critical" burn-rate alert tier from the Observability series' alerting tutorial was designed to catch), while a SEV4 might have essentially zero measurable SLO impact at all. --- ## Choosing a Severity Level in the Moment A practical decision framework, since "how bad is this" can genuinely be hard to judge in the first confusing minutes of an incident. ```mermaid flowchart TD Start{"Is there a complete
outage, data loss, or
active security breach?"} -->|Yes| SEV1["SEV1"] Start -->|No| Q2{"Is a MAJOR feature down,
or a significant subset
of users affected?"} Q2 -->|Yes| SEV2["SEV2"] Q2 -->|No| Q3{"Is there real user impact,
but a workaround exists
or it's a small % affected?"} Q3 -->|Yes| SEV3["SEV3"] Q3 -->|No| SEV4["SEV4"] ``` **A genuinely important, often-tested practical point: it's completely normal and expected to re-classify severity as more information comes in.** An incident might start as a SEV2 ("checkout is slow") and get upgraded to SEV1 once it becomes clear payments are actually failing silently, or downgraded from an initial panicked SEV1 once it turns out only an internal admin tool (not customer-facing) was affected. **Getting the initial severity slightly wrong isn't a failure — refusing to update it as facts change is.** --- ## On-Call Compensation and Sustainability A real, practical, and frequently underestimated topic — genuinely worth being able to speak to in an interview about how you'd run a healthy on-call program. ```mermaid graph TD Comp["On-Call Sustainability
Practices"] --> C1["Fair compensation
(stipend, extra time off,
or both — being on-call
has a real cost)"] Comp --> C2["A genuine cap on paging
frequency — if someone is
paged every single night,
that's a system problem,
not a 'toughen up' problem"] Comp --> C3["Time off / a lighter
workload the day AFTER a
rough on-call night"] Comp --> C4["Track and review paging
volume per person, not
just per team"] ``` **A directly reused concept, worth calling out:** this connects straight back to the **toil** and **alert fatigue** discussions from the SRE Fundamentals and Observability series — if a specific person is being paged constantly, that's the same underlying signal (excessive, low-value interrupts) driving two different but related fixes: reducing toil through automation, and reducing raw paging *volume* through better alert design. --- ## Protecting On-Call From Burnout ```mermaid flowchart TD A["Track pages-per-person-
per-week over time"] --> B{"Trending up, or
consistently high?"} B -->|Yes| C["🚨 Investigate: is this a
real, worsening reliability
problem, or an alerting
tuning problem (too
sensitive/noisy)?"] B -->|No, stable/low| D["✅ Healthy on-call load"] ``` **A strong, senior-level interview line:** "I treat on-call load itself as a metric worth tracking, the same way we'd track any other operational health signal — if one person is consistently getting paged far more than their teammates, or the whole team's paging volume is climbing, that's a concrete signal something needs to change: either genuine reliability work, or better-tuned alerting, not just 'push through it.'" --- ## Common Mistakes | Mistake | Why It's Wrong | Fix | |---|---|---| | No secondary on-call / no escalation path | A single unreachable person means a page can go unanswered indefinitely | Always configure primary + secondary, with a timed automatic escalation | | Very long rotations (a month+) with no compensation | Real burnout risk, high turnover, resentment | Prefer shorter rotations (commonly ~1 week), with fair compensation | | Treating the initial severity assessment as fixed/final | Facts often become clearer minutes into an incident; sticking to a wrong initial call wastes time or under-reacts | Explicitly re-evaluate and re-classify severity as new information arrives | | No shared, written severity definitions | Different people/teams calling the "same" situation different severities, causing confusion about how urgently to respond | Publish and socialize a clear, concrete severity rubric (with examples) organization-wide | | Ignoring paging-volume trends per individual | A quietly overloaded on-call engineer burns out silently, often leaving before anyone notices a pattern | Track and review paging load per person, not just in aggregate | | Follow-the-sun without disciplined handoffs | A dropped handoff means the next region inherits an incident with zero context, wasting critical time | Require a structured handoff (what's happening, what's been tried, current status) at every region transition | --- ## Worked Practice Problems **Problem 1:** A 3-person team runs a 1-week on-call rotation with no secondary. Last month, one engineer's phone died during their shift, and a SEV1 alert went unanswered for 40 minutes before someone happened to notice in Slack. What's the systemic fix? *Answer:* This is exactly the single-point-of-failure risk that a secondary on-call role and a timed escalation policy exist to prevent — the fix isn't "tell that engineer to keep their phone charged," it's adding a secondary on-call (even rotating among the same 3 people) and configuring an automatic escalation (e.g., page secondary after 5 unacknowledged minutes) so a single unreachable person can never again mean a completely unanswered SEV1. **Problem 2:** An incident starts as a reported SEV3 ("some users seeing a slow checkout page"), but 20 minutes in, an engineer discovers payments are actually being silently double-charged for a subset of transactions. What should happen to the severity classification, and why is this not a sign that the initial call was a mistake? *Answer:* This should immediately be re-classified to SEV1 — active, silent financial harm to real customers is a textbook critical incident, regardless of how the situation was initially described. This isn't a failure of the initial SEV3 call; it's the normal, expected process of incident severity evolving as more facts become available. The mistake would be *not* updating the severity once the double-charging was discovered, sticking rigidly to the original classification out of a reluctance to "escalate late." **Problem 3:** Reviewing paging data, you notice one engineer on a 6-person team has been paged 3x more often than anyone else over the last quarter, despite an evenly-split rotation schedule. What would you investigate? *Answer:* Since the rotation is evenly split, the disparity likely isn't about whose "turn" it is — I'd check whether this engineer's shifts happen to fall during genuinely higher-traffic/higher-incident windows (e.g., always covering weekends, or a particular time zone gap), or whether specific alert rules are miscalibrated and disproportionately noisy during their specific shifts. I'd also directly ask them whether the load feels sustainable — a 3x paging disparity is a concrete, trackable signal worth investigating before it turns into burnout or attrition. --- ## Summary and What's Next - **On-call** ensures a real, accountable human is always reachable when something breaks — organized into a fair **rotation** across the team, most commonly around one week per shift. - **Follow-the-sun** rotations (for genuinely global teams) avoid ever waking anyone at 3 AM, at the cost of requiring disciplined, well-documented cross-region handoffs. - **Primary + secondary** on-call, tied to a timed **escalation policy**, prevents a single unreachable person from leaving a page unanswered indefinitely. - **Severity levels** (commonly SEV1-SEV4) give the whole org a shared, fast vocabulary for "how bad is this" — and it's normal and expected to **re-classify severity** as new facts emerge during an incident. - **On-call sustainability** — fair compensation, capped paging frequency, tracked per-person paging load — is a real, ongoing operational responsibility, directly connected to the toil and alert-fatigue concepts from earlier tutorials in this course. **Continue to Part 2** (`02-incident-response-process.md`) for what actually happens once an incident is declared: roles like the Incident Commander, structured communication, runbooks, and status pages.