<style> body, p, li { text-align: justify; } </style>

Incident Management — Part 1: On-Call & Severity Levels

Series: Incident Management (1 of 3) Part 1: This file — On-Call Design & Severity Levels Part 2: 02-incident-response-process.md — Incident Response Process & Roles Part 3: 03-metrics-and-chaos-engineering.md — MTTR/MTTD & Chaos Engineering Questions: questions.md

Table of Contents#

1. Why This Topic Sits Downstream of Everything Else
2. What On-Call Actually Is
3. Designing an On-Call Rotation
4. Rotation Length — The Real Tradeoff
5. Follow-the-Sun Rotations
6. Primary and Secondary On-Call
7. Escalation Policies
8. A Worked PagerDuty-Style Escalation Policy
9. Severity Levels — The Common Framework
10. Choosing a Severity Level in the Moment
11. On-Call Compensation and Sustainability
12. Protecting On-Call From Burnout
13. Common Mistakes
14. Worked Practice Problems
15. Summary and What's Next

---

Why This Topic Sits Downstream of Everything Else#

Every other tutorial in this course builds toward one moment: a person's phone buzzing at 2 AM. The SLOs and error budgets (SRE Fundamentals) decide what counts as bad enough to page. The alerting design (Observability, Part 3) decides when the page actually fires. This tutorial is about what happens the instant it does — who gets paged, how they're organized, and how everyone agrees on how serious the situation actually is.

Diagram

---

What On-Call Actually Is#

On-call simply means: one specific person is designated as "the one who gets woken up" if something breaks during a specific window of time, so that a page always reaches a real, accountable human, instead of silently going nowhere or waking up an entire team at once.

A simple analogy: think of a hospital's on-call doctor. Not every doctor in the building has to be reachable every night — one specific doctor is designated the on-call doctor for tonight, and everyone (nurses, the front desk) knows exactly who to call. That's the whole idea.

Diagram

---

Designing an On-Call Rotation#

An on-call rotation spreads this responsibility across a team over time, so no single person is "always" on-call.

Diagram

Why a rotation, not just "one designated on-call person forever"? Because being on-call is genuinely disruptive — interrupted sleep, restricted freedom of movement, mental load even when nothing fires. Spreading it fairly across a team is both a sustainability requirement and, in many places, close to a legal/contractual expectation.

---

Rotation Length — The Real Tradeoff#

Diagram

The most common real-world answer, worth citing: one week is the single most widely used rotation length across the industry — long enough to build some continuity, short enough that no one person carries the load for too long. Shorter rotations (a few days) are more common at smaller teams or for very high-severity/high-frequency-paging services; longer rotations (2+ weeks) are increasingly seen as a burnout risk and are less common in mature orgs.

---

Follow-the-Sun Rotations#

For a genuinely global company (or a team distributed across multiple time zones), a clever alternative avoids waking anyone up at 2 AM at all.

Diagram

The idea, in plain terms: as the sun "follows" around the globe, on-call responsibility hands off to whichever region is currently in normal waking/working hours — nobody is ever paged at 3 AM their own local time, because someone else, somewhere, is always awake and covering.

The real cost worth naming: this requires a genuinely distributed team across multiple regions (not every company has this), plus disciplined, well-documented handoffs between regions (exactly the kind of information transfer problem Part 2 of this tutorial covers) — a dropped handoff means the next region inherits an incident with no context.

---

Primary and Secondary On-Call#

A common, simple safety net: designate two people on-call at once — a primary (gets paged first) and a secondary (gets paged if primary doesn't respond in time, or can be pulled in to help on a bigger incident).

Diagram

Why this matters practically: a single point of failure in your incident response process itself is a genuine risk — if the one and only on-call person is unreachable (phone died, in a dead zone, deep sleep), a page with no secondary escalation path can simply go unanswered indefinitely. Primary/secondary is the on-call equivalent of the redundancy principle from the Reliability & Architecture Patterns series, applied to people instead of servers.

---

Escalation Policies#

An escalation policy is the automated, pre-defined chain of "who gets paged next, and when" if the current person doesn't respond — directly extending the alert routing concepts from the Observability series' alerting tutorial.

Diagram

---

A Worked PagerDuty-Style Escalation Policy#

A realistic, concrete configuration (the exact shape used by tools like PagerDuty, Opsgenie, or Grafana OnCall):

Copy
escalation_policy:
  name: checkout-service-escalation
  rules:
    - escalation_delay_minutes: 5
      targets:
        - type: schedule
          id: checkout-primary-oncall
    - escalation_delay_minutes: 5
      targets:
        - type: schedule
          id: checkout-secondary-oncall
    - escalation_delay_minutes: 10
      targets:
        - type: user
          id: engineering-manager
    - escalation_delay_minutes: 15
      targets:
        - type: user
          id: director-of-engineering

Worth naming specifically: most real tools also support acknowledging a page (stops it from escalating further, "I've got this") separately from resolving it (the incident is actually fixed) — a very common, practical distinction that's easy to overlook if you've never actually used one of these systems.

---

Severity Levels — The Common Framework#

The moment an incident is confirmed, the very next question is: how bad is this, really? Severity levels give the whole organization a shared, consistent vocabulary for answering that instantly, without a debate.

Diagram

The direct tie-in to the SRE Fundamentals series, worth stating explicitly: severity often maps closely to error budget impact — a SEV1 typically burns a large, alarming chunk of the error budget very quickly (which is exactly what the "Critical" burn-rate alert tier from the Observability series' alerting tutorial was designed to catch), while a SEV4 might have essentially zero measurable SLO impact at all.

---

Choosing a Severity Level in the Moment#

A practical decision framework, since "how bad is this" can genuinely be hard to judge in the first confusing minutes of an incident.

Diagram

A genuinely important, often-tested practical point: it's completely normal and expected to re-classify severity as more information comes in. An incident might start as a SEV2 ("checkout is slow") and get upgraded to SEV1 once it becomes clear payments are actually failing silently, or downgraded from an initial panicked SEV1 once it turns out only an internal admin tool (not customer-facing) was affected. Getting the initial severity slightly wrong isn't a failure — refusing to update it as facts change is.

---

On-Call Compensation and Sustainability#

A real, practical, and frequently underestimated topic — genuinely worth being able to speak to in an interview about how you'd run a healthy on-call program.

Diagram

A directly reused concept, worth calling out: this connects straight back to the toil and alert fatigue discussions from the SRE Fundamentals and Observability series — if a specific person is being paged constantly, that's the same underlying signal (excessive, low-value interrupts) driving two different but related fixes: reducing toil through automation, and reducing raw paging volume through better alert design.

---

Protecting On-Call From Burnout#

Diagram

A strong, senior-level interview line: "I treat on-call load itself as a metric worth tracking, the same way we'd track any other operational health signal — if one person is consistently getting paged far more than their teammates, or the whole team's paging volume is climbing, that's a concrete signal something needs to change: either genuine reliability work, or better-tuned alerting, not just 'push through it.'"

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A 3-person team runs a 1-week on-call rotation with no secondary. Last month, one engineer's phone died during their shift, and a SEV1 alert went unanswered for 40 minutes before someone happened to notice in Slack. What's the systemic fix?

Answer: This is exactly the single-point-of-failure risk that a secondary on-call role and a timed escalation policy exist to prevent — the fix isn't "tell that engineer to keep their phone charged," it's adding a secondary on-call (even rotating among the same 3 people) and configuring an automatic escalation (e.g., page secondary after 5 unacknowledged minutes) so a single unreachable person can never again mean a completely unanswered SEV1.

Problem 2: An incident starts as a reported SEV3 ("some users seeing a slow checkout page"), but 20 minutes in, an engineer discovers payments are actually being silently double-charged for a subset of transactions. What should happen to the severity classification, and why is this not a sign that the initial call was a mistake?

Answer: This should immediately be re-classified to SEV1 — active, silent financial harm to real customers is a textbook critical incident, regardless of how the situation was initially described. This isn't a failure of the initial SEV3 call; it's the normal, expected process of incident severity evolving as more facts become available. The mistake would be not updating the severity once the double-charging was discovered, sticking rigidly to the original classification out of a reluctance to "escalate late."

Problem 3: Reviewing paging data, you notice one engineer on a 6-person team has been paged 3x more often than anyone else over the last quarter, despite an evenly-split rotation schedule. What would you investigate?

Answer: Since the rotation is evenly split, the disparity likely isn't about whose "turn" it is — I'd check whether this engineer's shifts happen to fall during genuinely higher-traffic/higher-incident windows (e.g., always covering weekends, or a particular time zone gap), or whether specific alert rules are miscalibrated and disproportionately noisy during their specific shifts. I'd also directly ask them whether the load feels sustainable — a 3x paging disparity is a concrete, trackable signal worth investigating before it turns into burnout or attrition.

---

Summary and What's Next#

• On-call ensures a real, accountable human is always reachable when something breaks — organized into a fair rotation across the team, most commonly around one week per shift.
• Follow-the-sun rotations (for genuinely global teams) avoid ever waking anyone at 3 AM, at the cost of requiring disciplined, well-documented cross-region handoffs.
• Primary + secondary on-call, tied to a timed escalation policy, prevents a single unreachable person from leaving a page unanswered indefinitely.
• Severity levels (commonly SEV1-SEV4) give the whole org a shared, fast vocabulary for "how bad is this" — and it's normal and expected to re-classify severity as new facts emerge during an incident.
• On-call sustainability — fair compensation, capped paging frequency, tracked per-person paging load — is a real, ongoing operational responsibility, directly connected to the toil and alert-fatigue concepts from earlier tutorials in this course.

Continue to Part 2 (02-incident-response-process.md) for what actually happens once an incident is declared: roles like the Incident Commander, structured communication, runbooks, and status pages.