Alert on Burn Rate, Not Raw Error Rate
Threshold alerts either page for noise or miss real outages. Multiwindow burn-rate alerts tie paging to how fast the error budget is being spent, with Prometheus rules you can adapt.
PowerTraces AI ·
Most paging alerts answer the wrong question. "Is the error rate above 1%?" says nothing about whether anyone should wake up. The question that matters is how fast you are spending the error budget, and whether the current pace will cost you the SLO.
Why simple thresholds fail
The Google SRE Workbook evaluates alerting strategies on four attributes: precision (the share of alerts that matter), recall (the share of significant events that alert), detection time, and reset time (how long an alert keeps firing after the problem is gone). Each familiar approach gives up at least one.
- A short-window threshold has low precision. The Workbook's example: a 0.1% error rate sustained for 10 minutes consumes only about 0.02% of a monthly budget, yet it can still page someone.
- A long window on its own has terrible reset time. In the Workbook's example, a total outage fires within about two minutes and then keeps firing for 36 hours after recovery.
- A
for:duration hurts recall and detection time. A complete outage waits as long as a 0.2% blip does, and an SLI that fluctuates may never stay bad long enough to fire. - A single burn-rate window has low recall. In the Workbook's example, a 35x burn rate never alerts, even though it empties a 30-day budget in about 20.5 hours.
Burn rate in one minute
The Workbook defines burn rate as how fast, relative to the SLO, the service consumes its error budget. A burn rate of 1 means you would use exactly the whole budget by the end of the SLO window. For a 99.9% SLO the budget is 0.1% of requests, so a burn rate of 14.4 means an error ratio of 1.44%.
The useful number to attach to a burn rate is the share of budget it consumes over a window: burn rate × window / SLO period. Over a 30-day (720-hour) period, a 14.4 burn sustained for one hour consumes 14.4 / 720 = 2% of the budget. A burn of 6 over six hours consumes 5%, and a burn of 1 over three days consumes 10%. Those three pairs are exactly the Workbook's recommended starting point.
Two windows per alert
The Workbook's recommended configuration pairs a long window, which gives precision, with a short window, which makes the alert reset quickly once the burn stops. The alert fires only while both exceed the burn-rate threshold. Its table for a 99.9% SLO:
| Severity | Long window | Short window | Burn rate | Budget consumed |
|---|---|---|---|---|
| Page | 1 hour | 5 minutes | 14.4 | 2% |
| Page | 6 hours | 30 minutes | 6 | 5% |
| Ticket | 3 days | 6 hours | 1 | 10% |
The short window is a fraction of the long one (the Workbook uses one twelfth for the paging pairs). That ratio is a convention, not a law; what matters is that the short window is short enough to stop the alert soon after the problem ends.
Fact vs. opinion. The table values come from the Workbook and assume a 30-day window and a 99.9% SLO. Whether they suit your service is a judgment call. My recommendation is to start with them unchanged, then tune only after you have seen how they behave against your real incident history.
The Prometheus rules
First record the error ratio over each window, then alert on combinations. The metric http_requests_total with a code label is an assumption; substitute your own SLI.
groups:
- name: slo-recording
rules:
- record: job:slo_errors_per_request:ratio_rate5m
expr: sum by (job) (rate(http_requests_total{code=~"5.."}[5m])) / sum by (job) (rate(http_requests_total[5m]))
- record: job:slo_errors_per_request:ratio_rate1h
expr: sum by (job) (rate(http_requests_total{code=~"5.."}[1h])) / sum by (job) (rate(http_requests_total[1h]))
- record: job:slo_errors_per_request:ratio_rate30m
expr: sum by (job) (rate(http_requests_total{code=~"5.."}[30m])) / sum by (job) (rate(http_requests_total[30m]))
- record: job:slo_errors_per_request:ratio_rate6h
expr: sum by (job) (rate(http_requests_total{code=~"5.."}[6h])) / sum by (job) (rate(http_requests_total[6h]))
- name: slo-alerts
rules:
- alert: ErrorBudgetBurnPage
expr: |
(
job:slo_errors_per_request:ratio_rate1h > (14.4 * 0.001)
and
job:slo_errors_per_request:ratio_rate5m > (14.4 * 0.001)
)
or
(
job:slo_errors_per_request:ratio_rate6h > (6 * 0.001)
and
job:slo_errors_per_request:ratio_rate30m > (6 * 0.001)
)
labels:
severity: page
annotations:
summary: "{{ $labels.job }} is burning its 99.9% error budget too fast"Notice there is no for:. In Prometheus, for keeps an alert pending until its condition has held for the full duration, and without it the alert fires on the first evaluation that returns results. The two windows already provide the stability that a duration clause would otherwise be asked to provide, without delaying detection of a real outage. The ticket tier (3 days and 6 hours at burn rate 1) is added the same way with severity: ticket, and needs matching recorded series for those windows.
Prometheus also supports keep_firing_for, which holds an alert in the firing state for a set time after its condition last matched. It can smooth flapping, but the short window already does much of that job, so reach for it only if you still see re-pages.
Where this bites
- Low-traffic services. With few requests, one failure moves a short-window ratio enormously. Consider aggregating across endpoints or alerting on a longer window for these services.
- Pages without an action. A burn-rate page should link to a runbook that says what to check first. Burn rate tells you how urgent it is, not what is wrong.
- A bad SLI. If the SLI counts requests users do not care about, perfect burn-rate math pages you for the wrong things. Fix the SLI first.
- Slow-burn blind spots. Burn rates below 1 never exhaust the budget, which is fine. Burns just above 1 are what the ticket tier is for, so route it somewhere that gets worked.
Rolling it out
- Pick one service with a well-understood SLI and write down its SLO and window.
- Run the rules for a week with severity
ticketonly, and compare firings against what actually hurt users. - Promote the fast-burn pair to paging once it matches real incidents.
- Delete the old threshold alerts it replaces, so on-call does not get both.
Takeaways
- Alert on how fast the budget is burning, not on a raw error rate.
- Use a long window for precision and a short window for fast reset.
- Drop
for:on these alerts; the window pair replaces it. - Page for fast burns, ticket for slow ones.
- Tune against your own incident history, not just the defaults.
If you want SLO error budgets tracked and alerted on without writing the recording rules yourself, PowerTraces documents SLOs, error budgets and alert rules as part of its health features.