Error Budgets

Table of Contents#

1. Recap: Where the Error Budget Comes From
2. What Is an Error Budget, Precisely
3. Why Error Budgets Are the Single Best Idea in SRE
4. Calculating an Error Budget — Worked Examples
5. What Consumes the Error Budget
6. Burn Rate — The Core Metric
7. Multi-Window, Multi-Burn-Rate Alerting (Preview)
8. The Error Budget Policy
9. Designing an Error Budget Policy — A Worked Template
10. Error Budgets as a Negotiation Tool
11. Error Budgets and Release Velocity
12. Error Budgets Across Multiple Teams and Dependencies
13. Common Anti-Patterns With Error Budgets
14. Case Study: Google's Error Budget Policy
15. Worked Practice Problems
16. Summary and What's Next

---

Recap: Where the Error Budget Comes From#

In Part 2 we established: SLO = the target reliability level (e.g., 99.9% over 28 days). The error budget is simply the complement of that target — the amount of unreliability you're allowed before you've broken your promise to yourself.

Diagram

---

What Is an Error Budget, Precisely#

Error Budget = 100% − SLO

This is usually expressed two equivalent ways:

1. As a percentage of requests: e.g., "0.1% of requests may fail."
2. As an absolute amount of time: e.g., "40.32 minutes of full downtime allowed per 28 days" (assuming a pure-availability SLO; for latency SLOs it's "0.1% of requests may be slow," not literally minutes of downtime).

Critical nuance for latency-based SLOs: if your SLO is about latency ("99% of requests under 400ms"), the error budget isn't "minutes of downtime" — it's "how many individual requests are allowed to be slow." You can't cleanly convert that into a single downtime duration the way you can for a pure availability SLO. Interviewers sometimes test whether you blindly apply the "minutes of downtime" framing to a latency SLO where it doesn't actually apply.

Diagram

---

Why Error Budgets Are the Single Best Idea in SRE#

If an interviewer asks "what's the single most important SRE concept," a well-argued case for error budgets is a strong, memorable answer. Here's why:

1. It Converts a Political Argument Into a Data-Driven One#

Without an error budget, "should we slow down and stabilize, or keep shipping features" is a subjective argument — whoever is louder, more senior, or more recently burned by an incident tends to win. With an error budget, the question becomes objectively answerable: "Do we have budget left? Yes → ship. No → the pre-agreed policy kicks in."

Diagram

2. It Aligns Incentives Between Dev and Ops#

Both product/dev teams and SRE/ops teams share the same number. Devs want the budget to have headroom (so they can ship). Ops wants the budget to have headroom (so the system is stable). For the first time, both groups are optimizing for the same metric instead of pulling in opposite directions.

3. It Makes "How Reliable Is Reliable Enough" Concrete#

Instead of an unbounded, anxiety-inducing goal ("never go down"), the team has a finite, spendable resource. This is psychologically different — spending down a budget you're allowed to spend feels fundamentally different from "breaking a promise every single time anything goes wrong."

4. It Naturally Prioritizes Reliability Work When It Matters Most#

When the budget is healthy, the team doesn't need to obsess over minor reliability improvements — that time is better spent on features. When the budget is being burned quickly, the signal automatically redirects attention to reliability work. The system self-regulates, rather than relying on someone remembering to periodically "check in" on reliability.

---

Calculating an Error Budget — Worked Examples#

Example 1: Simple Availability SLO#

• SLO: 99.95% availability, rolling 30-day window.
• Total minutes in 30 days: 30 × 24 × 60 = 43,200 minutes.
• Error budget = (1 − 0.9995) × 43,200 = 0.0005 × 43,200 = 21.6 minutes.

Example 2: Request-Based SLO (More Common in Practice)#

Most real SLOs are actually defined in terms of request counts, not wall-clock time, because a service can have zero traffic during a "down" period and it wouldn't actually impact any users.

• SLO: 99.9% of requests succeed, rolling 28-day window.
• Total requests in the window: 50,000,000.
• Error budget = (1 − 0.999) × 50,000,000 = 0.001 × 50,000,000 = 50,000 failed requests allowed.

This framing is important: two services with identical percentage-based SLOs can have very different "danger" profiles in absolute terms depending on traffic volume. A low-traffic internal tool with 99.9% SLO might have an error budget of "50 failed requests," while a high-traffic public API with the same 99.9% SLO might have "50,000 failed requests" — the latter can absorb a much larger single outage before running out.

Example 3: Combining Both Views#

Diagram

Example 4: A Full Month of Budget Tracking (Illustrated)#

Diagram

By day 20, the team has consumed 5 + 10 + 22 = 37 minutes of their 40.32-minute budget — 91.8% consumed with 8 days still left in the window. This should have already triggered the "freeze" tier of the error budget policy well before the third incident, ideally causing the team to be more cautious about risky changes during that final week.

---

What Consumes the Error Budget#

A common interview question: "list the things that consume an error budget." A comprehensive answer covers more than just "outages":

Diagram

Interview nuance: whether planned maintenance counts against the error budget is a real design decision teams make explicitly, and it's worth naming as a decision rather than assuming one answer. Some teams exclude pre-announced maintenance windows from the SLI calculation entirely (since users were notified); others deliberately include it, reasoning that "the user doesn't care why the service was down, only that it was" — which is closer to Google's actual philosophy in the SRE book.

---

Burn Rate — The Core Metric#

Burn rate measures how fast the error budget is being consumed relative to a sustainable pace — the pace that would exhaust the budget exactly when the measurement window closes, no sooner, no later.

Burn Rate = (Actual Error Rate) / (Allowed Error Rate implied by the SLO)

Interpreting Burn Rate Values#

Diagram

Why Burn Rate Beats a Simple Threshold Alert#

A naive approach — "alert if error rate > 1%" — has two failure modes:

1. Too sensitive for a brief blip: a 30-second error spike during a deploy might cross 1% instantly but consume a trivial amount of actual budget.
2. Too insensitive for a slow leak: an error rate of 0.5% sustained for days might never cross a fixed 1% threshold, yet it can still fully exhaust the budget over the SLO window if left unaddressed.

Burn rate solves both: it's normalized against the SLO itself, and (as covered next) it's evaluated over multiple time windows simultaneously to catch both fast, severe issues and slow, sustained leaks.

---

Multi-Window, Multi-Burn-Rate Alerting (Preview)#

This topic is covered in full depth in the Observability tutorial (topic 4) since it's fundamentally an alerting design pattern — but the concept originates directly from error budget math, so a preview belongs here.

The core idea: evaluate burn rate over two windows at once — a short window (fast detection) and a long window (confirms it's not just noise).

Diagram

Why two windows, not one: using only a long window (e.g., 6 hours) is slow to detect a genuinely severe outage — you'd wait hours before paging. Using only a short window (e.g., 5 minutes) is noisy — any tiny blip triggers a page. Requiring both the short AND long window to simultaneously exceed the threshold gets you fast detection and confirmation that it's not just transient noise, before waking someone up at 3 AM.

This exact 14.4x/1-hour + 5-minute pattern (and the 6x/6-hour, 1x/3-day tiers) come directly from Google's own published SRE workbook alerting recipes — citing these specific numbers in an interview is a strong signal of real depth.

---

The Error Budget Policy#

An error budget without a policy is just a dashboard nobody acts on. The policy is the pre-agreed set of consequences triggered at different budget levels — written down before a crisis, not improvised during one.

Diagram

Why the Policy Must Be Pre-Agreed#

If the "what happens when budget runs out" decision is made during an active incident or a heated planning meeting, it inevitably becomes political again — exactly what the error budget was supposed to prevent. The policy needs buy-in from engineering, product, and leadership before it's needed, precisely so that when it's needed, it's simply executed rather than re-litigated.

---

Designing an Error Budget Policy — A Worked Template#

A complete, realistic policy document (the kind you might actually write at a company):

Copy
# Error Budget Policy — Checkout Service

## SLO Reference
- Availability: 99.95%, rolling 28-day window
- Latency (p99 < 400ms): 99%, rolling 28-day window

## Policy Tiers

### Tier 1: Healthy (>50% budget remaining)
- Normal release cadence.
- Standard canary process (5% → 25% → 100% over 2 hours).

### Tier 2: Watch (25-50% budget remaining)
- Release approvers must review recent burn trend before approving
  high-risk changes (schema migrations, infra changes).
- No change required to canary process.

### Tier 3: Constrained (10-25% budget remaining)
- Canary duration doubled (5% → 25% → 100% over 4+ hours).
- Non-critical/experimental feature flags default to OFF for new rollout.
- Daily budget review in team standup.

### Tier 4: Frozen (<10% budget remaining, or budget exhausted)
- All non-critical releases frozen.
- Only P0/P1 bug fixes and reliability improvements may ship.
- Override requires sign-off from Eng Director + Product Director jointly,
  and must be logged as a documented risk acceptance.
- Team's sprint priorities shift to reliability work until budget recovers
  above Tier 3 threshold.

### Tier 5: Repeated Breach (exhausted budget in 2+ consecutive windows)
- Escalated to VP Engineering.
- Mandatory postmortem-of-postmortems: is the SLO unrealistic for the
  current architecture, or is there a systemic reliability gap?
- Roadmap review: reliability investment may be reprioritized above
  planned feature work for the following quarter.

## Exclusions
- Pre-announced maintenance windows (>72h notice) are excluded from SLI
  calculation, capped at 4 hours/quarter.
- Load-testing-induced degradation in a designated non-prod environment
  does not count.

## Review Cadence
- Policy reviewed quarterly by SRE + Product leadership.
- SLO targets themselves reviewed quarterly based on trailing performance
  and business needs.

Interview signal: being able to sketch a policy like this — with graduated tiers rather than a single binary "frozen/not frozen" — demonstrates real operational maturity. A binary policy is brittle (either it's ignored because it's too strict, or it's meaningless because it's too lenient); a tiered policy lets the response scale proportionally to actual risk.

---

Error Budgets as a Negotiation Tool#

This is worth calling out explicitly because it's one of the most common scenario-based interview questions ("what would you do if Product wants to ship something risky and the budget is nearly gone?").

Diagram

The key insight to articulate: the SRE's role here is not to be the sole decision-maker who says "no" — it's to surface the data and enforce the pre-agreed process. This keeps the SRE function collaborative rather than adversarial, and it means that if leadership does override, it's an intentional, visible decision — not a silent violation nobody notices until the next outage.

---

Error Budgets and Release Velocity#

Error budgets don't just constrain velocity — they can also be used to actively encourage it when there's headroom, which is a nuance many candidates miss.

Diagram

Google explicitly recommends option D and E: an unused error budget is a wasted opportunity — either to ship faster (since the whole point of not targeting 100% was to enable velocity) or to proactively test resilience via controlled chaos experiments while there's room to absorb the risk. A team that always has a large budget surplus every quarter might actually be over-engineered or too conservative relative to their actual SLO — that's itself worth investigating, not just celebrating.

---

Error Budgets Across Multiple Teams and Dependencies#

In a microservices architecture, one team's error budget consumption can be caused by another team's dependency failing. This raises real organizational questions interviewers may probe:

Diagram

Best practice (and the answer interviewers want): the consuming team's (Team A's) error budget is debited, because the error budget reflects user-observed reliability, and the user doesn't care which internal team caused the problem. However, this should trigger a cross-team conversation: Team A's error budget policy might now specifically call out "review dependency reliability with Team B" as a Tier 3/4 action item, and Team B's own SLO/postmortem process should independently address the root cause on their side. Some mature orgs also define internal SLOs between teams (Team B promises Team A a specific reliability level, almost like an internal SLA) specifically to make this dependency risk explicit and negotiable.

---

Common Anti-Patterns With Error Budgets#

---

Case Study: Google's Error Budget Policy#

Google's publicly documented pattern (from the SRE book, Chapter 3) is frequently referenced directly in interviews. Paraphrased:

"If a service's error budget is exhausted, feature launches are frozen — except for launches specifically intended to improve reliability — until the service is back within its SLO. The VP of the affected product area can grant an exception, but doing so is a visible decision they own."

Key properties of this policy worth calling out:

• The freeze is specifically scoped — reliability-improving launches are exempt, since blocking those would be counterproductive.
• The override authority is senior and specific (a named VP), not "whoever argues hardest in the room."
• The override is visible/logged, which creates accountability — if the VP overrides and the service has another incident, that's now a decision they consciously made, which naturally discourages casual overrides.

---

Worked Practice Problems#

Problem 1: A service has an SLO of 99.9% over a rolling 28-day window and averages 20,000,000 requests over that window. So far this window, 25,000 requests have failed. What percentage of the error budget has been consumed, and what's the current burn rate (assume evenly spread over exactly 14 of the 28 days so far)?

Solution:

• Total error budget = 0.001 × 20,000,000 = 20,000 failed requests allowed.
• Failures so far = 25,000, which is already 125% of the total budget — the SLO has already been breached with 14 days still remaining in the window.
• Sustainable pace at day 14 (half the window) would be 10,000 failures. Actual = 25,000. Burn rate ≈ 25,000 / 10,000 = 2.5x.
• This should have triggered a Tier 4 (frozen) response well before this point.

Problem 2: Your team's error budget policy has a Tier 3 threshold at 25% remaining. Today the dashboard shows 30% remaining, but the last 24 hours alone consumed 15% of the total budget. Should the team be worried even though they're technically still in Tier 2?

Solution: Yes — this is exactly why burn rate matters more than a static remaining-percentage snapshot. A 15%-of-total-budget burn in just 24 hours out of a 28-day (672-hour) window implies a burn rate of roughly (15% / (24/672)) ≈ 28x the sustainable pace. At that rate, the remaining 30% would be fully exhausted in about 1 more day, even though the raw "% remaining" number alone looks only moderately concerning. A mature team monitors burn rate trend, not just the remaining balance — this is precisely the argument for multi-window burn-rate alerting instead of a single threshold check.

Problem 3: Two services, A and B, both have a 99.9% availability SLO over 28 days. Service A gets 1,000 requests/day; Service B gets 100,000,000 requests/day. A single 10-minute total outage occurs on each. Which one is in more danger of breaching its SLO, in absolute error-budget terms?

Solution: Trick question setup — convert to the time-based framing, which is traffic-independent for a pure availability SLO: both services have the same 40.32-minute time-based budget for 28 days, and a 10-minute outage consumes the same fraction (10/40.32 ≈ 24.8%) of either service's budget, regardless of request volume. However, if you instead think in request-based terms (which matters more for release-risk sizing, canary planning, etc.), Service B's absolute failed-request count from that same 10-minute outage is vastly larger (100,000,000/day ÷ 144 ten-minute-intervals/day ≈ 694,444 failed requests) vs. Service A's (1,000/day ÷ 144 ≈ ~7 failed requests) — so while their SLO risk is identical, their blast radius in absolute user impact is wildly different. This distinction — SLO breach risk vs. absolute user impact — is worth explicitly separating in an interview answer.

---

Summary and What's Next#

• Error budget = 100% − SLO, expressible as time (for availability SLOs) or as a count of allowed bad events (more general, works for latency/correctness SLOs too).
• Error budgets convert the dev-velocity-vs-stability tradeoff from a political argument into a shared, objective number.
• Burn rate = actual error rate ÷ allowed error rate — the key metric for knowing not just "how much budget is left" but "how fast are we losing it," which is what modern multi-window alerting is built on.
• An error budget without a pre-agreed policy is meaningless — the policy should be tiered (graduated response), not binary, and should require senior, visible sign-off to override.
• A healthy budget surplus should be actively used (faster shipping, chaos experiments), not just left on the table.
• In multi-team architectures, the consuming team's budget is debited by convention (reflects user experience), which should trigger cross-team accountability conversations, not just internal blame.

Continue to Part 4 (04-toil-and-postmortems.md) to cover the other two foundational SRE practices: eliminating toil, and running blameless postmortems when incidents do happen.