Alerting Design & Burn-Rate Alerts

Table of Contents#

1. The Actual Goal of Alerting
2. Symptom-Based vs Cause-Based Alerting
3. The Alert Fatigue Problem
4. What Makes a Good Alert — Four Properties
5. Recap: SLOs and Error Budgets
6. Why a Simple Threshold Alert Isn't Good Enough
7. Burn Rate — Rebuilt From Scratch
8. The Two-Window Trick
9. Google's Standard Multi-Window Burn-Rate Table
10. Building a Burn-Rate Alert in PromQL
11. Alert Severity and Routing
12. Alerts Need Runbooks
13. Deduplication, Grouping, and Silencing
14. Alert Review — Treating Alerting as a Living System
15. A Complete Worked Alert Design
16. Common Mistakes
17. Worked Practice Problems
18. Summary — The Complete Observability Series

---

The Actual Goal of Alerting#

Here's a simple test for whether an alert is any good: every single page should require a human to actually do something. If an alert fires and the response is "yeah, I saw it, nothing to do," that alert shouldn't have paged anyone in the first place.

Diagram

The single most important interview line on this topic: "The goal of alerting isn't to catch every anomaly — it's to page a human only when a human is genuinely needed, and to say exactly what's wrong when they are. Too many alerts is just as much a failure as too few."

---

Symptom-Based vs Cause-Based Alerting#

This is one of the most important distinctions in the entire alerting discipline, and a direct extension of the "Golden Signals" idea from the Monitoring Methodologies series.

Diagram

The core principle, worth memorizing: alert on symptoms (what the user actually experiences), not on causes (internal system metrics that might or might not actually matter). A cause-based alert like "CPU > 85%" fires whether or not anyone is actually affected — maybe that CPU usage is completely healthy and expected. A symptom-based alert like "error rate > 5%" or "p99 latency > SLO threshold" only fires when something a real user would actually notice is happening.

Where cause-based metrics still matter: they're extremely useful as diagnostic information after a symptom-based alert has already fired — e.g., once "error rate is high" pages someone, checking "is CPU also high?" helps them figure out why. This ties directly back to the USE method from the Monitoring Methodologies series: USE metrics (cause-level) are for diagnosis; RED/Golden-Signal metrics (symptom-level) are for alerting.

Diagram

---

The Alert Fatigue Problem#

Alert fatigue is what happens when a team receives so many low-value alerts that they start ignoring — or worse, sleeping through — pages, including the genuinely important ones.

Diagram

Why this is such a serious, well-documented failure mode: alert fatigue isn't just an annoyance — it's a genuine, real cause of extended outages, because it degrades the exact signal (a page) that's supposed to guarantee fast human attention. This is precisely why SRE teams treat "alert volume" and "% of alerts that were actually actionable" as real, tracked health metrics for their alerting system, not just an afterthought.

Analogy: it's the boy who cried wolf, but for pagers — cry wolf enough times over trivial things, and nobody comes running when the real wolf shows up.

---

What Makes a Good Alert — Four Properties#

A practical, memorable checklist worth having ready in an interview:

Diagram

If an alert fails any of these four tests, it's a candidate for being deleted, downgraded to a non-paging notification (e.g., a Slack message or a ticket instead of a page), or fixed to be clearer/less noisy.

---

Recap: SLOs and Error Budgets#

The rest of this tutorial builds directly on the SLI/SLO/error budget concepts from the SRE Fundamentals series — a quick recap of just the pieces needed here:

Diagram

If you haven't already read the SRE Fundamentals series, the short version needed here: an error budget is the amount of "badness" (failed requests, downtime) you're allowed before you've broken your own reliability promise. Burn rate measures how fast you're spending that budget.

---

Why a Simple Threshold Alert Isn't Good Enough#

A naive first attempt at alerting: "page me if the error rate goes above 1%." This has two real, opposite failure modes.

Diagram

The deeper issue: a fixed threshold has no relationship at all to your actual SLO or error budget — it's just a number someone picked. A genuinely well-designed alert should be tied directly to how much of your error budget is actually being burned, which is exactly what burn-rate alerting fixes.

---

Burn Rate — Rebuilt From Scratch#

(A fuller treatment lives in the SRE Fundamentals series — this is a self-contained recap specifically for building the alert.)

Burn rate answers: "at the current error rate, how much faster or slower than sustainable are we spending our error budget?"

Burn Rate = (Actual Error Rate) / (Allowed Error Rate implied by the SLO)

Diagram

Why burn rate is a better basis for alerting than a raw threshold: it's automatically calibrated against your own service's actual SLO, rather than an arbitrary fixed number, and it naturally distinguishes "a brief, small blip" from "a sustained problem that will actually exhaust the budget" — which is exactly the two failure modes the naive threshold alert couldn't tell apart.

---

The Two-Window Trick#

A single-window burn-rate check still has a real problem: pick a short window (like 5 minutes) and you get fast detection but lots of noise from tiny blips; pick a long window (like 6 hours) and you filter noise well but detection becomes painfully slow for a genuinely severe, fast-moving outage.

The fix, used in every real production burn-rate alerting setup: require BOTH a short window AND a long window to simultaneously show a high burn rate before paging.

Diagram

Why this specifically works, explained simply: the long window acts as a "is this actually sustained, or just a blip" filter — a 30-second spike vanishes quickly from a 1-hour average, so it won't trip the long-window check even if it briefly trips the short-window check. The short window acts as the "how fast do we find out" mechanism — without it, you'd have to wait for the full long window to elapse before ever detecting anything, which is far too slow for a genuinely severe, fast-developing outage. Requiring both together gets you fast detection with built-in noise filtering — the best of both individually-flawed approaches.

---

Google's Standard Multi-Window Burn-Rate Table#

This exact table (or a close variant of it) comes directly from Google's SRE Workbook, and citing these specific numbers is one of the strongest, most concrete signals of real depth you can bring to an SRE interview.

Diagram

Why the specific number 14.4 isn't arbitrary: it's derived directly from wanting a 1-hour window to represent 2% of a 28-day error budget being consumed — 28 days × 24 hours = 672 hours in the full window; consuming 2% of the budget in 1 out of those 672 hours implies a burn rate of 0.02 × 672 ≈ 13.44, and Google's published version rounds this (with some adjustment for their exact chosen percentages) to the commonly-cited 14.4x figure. You don't need to derive this exact number live in an interview — but being able to explain that it's derived from "what % of budget would this consume over what time period," rather than being a magic number, is exactly the kind of understanding that separates real knowledge from memorization.

---

Building a Burn-Rate Alert in PromQL#

A concrete, realistic example, connecting directly back to the PromQL syntax from Part 1.

Copy
# Error rate over the SHORT window (5 minutes)
(
  sum(rate(http_requests_total{service="checkout", status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total{service="checkout"}[5m]))
) > (14.4 * 0.001)   # 14.4x the SLO's allowed error rate (SLO = 99.9%, so allowed rate = 0.001)

  AND

# Error rate over the LONG window (1 hour) — must ALSO be high
(
  sum(rate(http_requests_total{service="checkout", status=~"5.."}[1h]))
  /
  sum(rate(http_requests_total{service="checkout"}[1h]))
) > (14.4 * 0.001)

Reading this in plain English: "Page only if the error rate over the last 5 minutes is at least 14.4 times the allowed rate from our SLO, AND the error rate over the last hour is also at least 14.4 times the allowed rate." Both conditions must be true at once — exactly the two-window trick from above, expressed as an actual, runnable query.

---

Alert Severity and Routing#

Not every alert should page a human at 3 AM. A mature alerting setup routes different severities to different channels.

Diagram

A useful mental model: think of severity level as answering "how much am I willing to inconvenience a human, right now, for this?" A critical, budget-threatening issue is worth waking someone up at 3 AM for. A slow-burning, 3-day-window issue is worth a ticket someone looks at during their next work day — waking them up for it would be actively harmful (contributing to alert fatigue) without providing any real benefit, since a few hours' delay doesn't meaningfully change the outcome for something burning that slowly.

---

Alerts Need Runbooks#

An alert that just says "checkout-service: high error rate" and nothing else forces the responder to start every single incident completely from scratch, even for a problem the team has already seen and solved before.

Diagram

Why this matters, concretely: this ties directly back to the toil discussion from the SRE Fundamentals series — if the first three diagnostic steps are always the same for a given alert, that's a textbook case of toil that should be automated or, at minimum, codified into a runbook the alert links to directly, so the responder isn't reinventing the same investigation from memory (or worse, from scratch) every single time, especially at 3 AM when memory and judgment are both impaired.

A genuinely strong practice worth naming: some mature teams go further and build self-diagnosing alerts — the alert payload itself includes a link to a pre-built dashboard already filtered/scoped to exactly the affected service and time window, or even runs a few automatic diagnostic checks and includes their results directly in the alert notification, cutting the very first few minutes of manual investigation out entirely.

---

Deduplication, Grouping, and Silencing#

A single real incident can trigger many raw, low-level alert conditions simultaneously — without careful handling, this floods on-call with dozens of separate pages for what is genuinely just one underlying problem.

Diagram

• Deduplication: the same alert condition firing repeatedly (e.g., re-evaluated every minute while still true) shouldn't generate a brand-new page each time — it should be recognized as "still the same ongoing issue."
• Grouping: multiple different alerts that are actually symptoms of the same underlying root cause should be bundled into one notification, not sent as separate, disconnected pages.
• Silencing: deliberately, temporarily suppressing specific alerts — e.g., during a planned maintenance window, so expected, intentional disruption doesn't generate unnecessary pages.

This is exactly the job of Prometheus's Alertmanager component (from Part 1's architecture diagram) — it sits between "an alert condition became true" and "a human actually gets notified," specifically to prevent the kind of alert flood shown above.

---

Alert Review — Treating Alerting as a Living System#

A mature team doesn't just set up alerts once and leave them alone forever — alerting rules need periodic review, exactly like code.

Diagram

A concrete, practical metric worth naming in an interview: tracking the percentage of pages that were genuinely actionable (as opposed to false alarms, or things that resolved on their own before anyone even looked) over time. A declining actionable-percentage is a direct, quantified signal that specific alert rules need tuning — exactly the same discipline the SRE Fundamentals series describes for postmortem action items (measure the process itself, not just individual incidents).

---

A Complete Worked Alert Design#

Bringing everything in this tutorial together into one realistic, full example for a single service:

Copy
# Alerting Design: checkout-service

## SLO
99.9% availability, rolling 28-day window (allowed error rate: 0.1%)

## Alert 1: Critical Burn Rate (PAGE)
- Condition: error rate ≥ 14.4x allowed rate, over BOTH 5-minute
  AND 1-hour windows simultaneously
- Severity: Critical → immediate page (call + SMS)
- Runbook link: included directly in the alert payload
- Rationale: at this burn rate, the 28-day budget would be
  exhausted in ~2 days if sustained — genuinely urgent

## Alert 2: High Burn Rate (PAGE, lower urgency)
- Condition: error rate ≥ 6x allowed rate, over BOTH 30-minute
  AND 6-hour windows simultaneously
- Severity: High → page notification, not an urgent call
- Rationale: developing more slowly, still needs same-day attention

## Alert 3: Low Burn Rate (TICKET ONLY)
- Condition: error rate ≥ 1x allowed rate, over BOTH 6-hour
  AND 3-day windows simultaneously
- Severity: Low → creates a ticket, reviewed next business day
- Rationale: sustainable-pace concern, not urgent

## Diagnostic Dashboards (linked from every alert)
- RED dashboard for checkout-service (from the Monitoring
  Methodologies series)
- USE dashboard for its direct dependencies (database,
  payment gateway)
- Recent deploys timeline overlay

## Explicitly NOT Alerted On Directly (used for diagnosis only)
- Raw CPU/memory utilization (cause-based, not symptom-based —
  see USE dashboard instead, only checked AFTER a symptom alert fires)
- Individual pod restarts (too noisy on their own; only
  meaningful in aggregate, already reflected in the error-rate SLI)

This kind of complete, structured worked example — not just an isolated PromQL snippet — is exactly the depth of answer that distinguishes a strong response to "design an alerting strategy for a service" in an interview.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: Your on-call rotation reports being paged 15 times last week, but only 2 of those pages required any real action — the rest resolved on their own within a minute or two. What's happening, and what would you investigate first?

Answer: This is alert fatigue in the making — a low "actionable %" (2/15 ≈ 13%) is a direct, measurable signal that the alerting rules are miscalibrated. I'd first check whether these are single-window, threshold-based alerts (likely too sensitive to brief blips) rather than proper multi-window burn-rate alerts — the two-window trick specifically exists to filter out exactly this kind of short-lived noise, requiring a long window to also confirm the problem is sustained before paging.

Problem 2: A service has an SLO of 99.95% over 28 days. Design the "Critical" tier burn-rate alert conceptually — what would trigger it, and roughly how urgent should it be?

Answer: Using Google's standard critical-tier pattern: alert if the error rate is at least ~14.4x the allowed rate (which for a 99.95% SLO is an allowed error rate of 0.05%, so the critical threshold would be roughly 14.4 × 0.05% ≈ 0.72% actual error rate), sustained over both a short window (e.g., 5 minutes) and a longer window (e.g., 1 hour) simultaneously. This should page immediately with high urgency, since at that burn rate, the entire 28-day budget would be consumed in roughly 2 days if the condition continued unaddressed.

Problem 3: A database goes down, and five different microservices each fire their own "high error rate" alert within the same minute, generating five separate pages to the same on-call engineer. How would you fix this specific problem?

Answer: This is exactly what Alertmanager's grouping feature is for — configure grouping so that alerts sharing a common cause (e.g., a shared "depends_on: database" label, or simply alerts firing within the same short time window across related services) get bundled into a single notification listing all five affected services together, rather than generating five separate, disorienting pages for what is genuinely one underlying incident.

---

Summary — The Complete Observability Series#

• The goal of alerting: page a human only when a human genuinely needs to act — every unnecessary page is a cost, not a free safety margin.
• Alert on symptoms (what users experience — RED-style metrics), not causes (internal system metrics like CPU/memory — USE-style) — cause-based metrics are for diagnosis after a symptom-based alert fires, not for triggering the page itself.
• Alert fatigue is a real, measurable cause of extended outages, because it degrades the reliability of the exact signal (a page) meant to guarantee fast attention.
• A simple fixed threshold is either too noisy for blips or too blind to slow leaks — burn-rate alerting, tied directly to your actual SLO and error budget, fixes both.
• The two-window trick (require both a short AND a long window to simultaneously show a high burn rate) gives fast detection with built-in noise filtering — this is the industry-standard pattern, and Google's published 14.4x/6x/1x tiered table is worth knowing by name and rough derivation.
• Every alert should link a runbook (or a pre-scoped dashboard), directly connecting to the toil discussion from the SRE Fundamentals series — repeated, identical first-response steps should be automated or codified, not reinvented from memory during every incident.
• Deduplication, grouping, and silencing (handled by tools like Prometheus's Alertmanager) prevent one real incident from flooding on-call with many disconnected pages.
• Treat alerting as a living system: track the percentage of pages that were genuinely actionable, and use a declining trend as a concrete signal to tune specific rules.

This completes the Observability series (Metrics/Logs/Traces, Prometheus, Distributed Tracing/OpenTelemetry, and Alerting Design). See questions.md in this folder for the full interview question bank covering all three parts, and see the Incident Management tutorial (topic 5) for what happens once a well-designed alert actually pages someone.