Skip to content

Alerting, SLOs & Error Budgets

Metrics, dashboards, and traces only help once someone's looking. Alerting is what turns a crossed threshold into a notification on its own — and the vocabulary of SLIs, SLOs, and error budgets is what turns "how reliable should this be" from a feeling into a number.

TL;DR — in 30 seconds:

  • An alerting rule is a PromQL expression evaluated on a schedule; it fires once true for a configured duration — default to symptom-based alerts (a golden signal) over cause-based ones.
  • Alertmanager takes a firing rule and dedupes, groups, routes, and silences it into a notification.
  • SLI = the measured number, SLO = its target over a window, error budget = the allowed gap between the SLO and 100% — the number you're really defending in a 2am-incident conversation.

1. Alerting rules — evaluated on a schedule

A Prometheus alerting rule is a PromQL expression, evaluated on a schedule; when it's true for a configured duration, the rule fires. Prometheus itself only evaluates rules and hands firing alerts off — it doesn't decide who gets paged or how.

2. Symptom vs cause

The single most important choice in writing an alerting rule is what to alert on:

  • A symptom-based alert tests something user-facing — a golden signal like elevated error rate or latency. It fires because someone is actually experiencing pain.
  • A cause-based alert tests an internal condition — high CPU, a full disk — that might lead to user pain, but often doesn't, and sometimes user pain happens without it.

Default to symptom-based alerts: they page you exactly when it matters, and skip paging you for a cause that never actually hurt anyone. A cause-based signal is still useful — as a panel on a dashboard you check once a symptom-based alert has already fired, not as the thing that wakes you up.

3. Alertmanager — what happens after a rule fires

Alertmanager is the component that receives alerts Prometheus's rules have fired and decides what to do with them: deduplicating the same alert firing from multiple sources, grouping related alerts into one notification instead of a flood, routing an alert to the right destination, and silencing alerts during planned maintenance. Prometheus firing a rule and Alertmanager acting on it are two separate steps.

flowchart LR
    R["Alerting rule<br/>(PromQL, true for N minutes)"] -- "fires" --> AM["Alertmanager"]
    AM -- "dedupe / group / route / silence" --> N["Notification<br/>(page, chat, ticket)"]

4. SLI, SLO, and error budget

  • An SLI (service level indicator) is the actual measured number for a service — e.g. the ratio of requests that succeeded and returned within a latency target.
  • An SLO (service level objective) is a target for that SLI over a time window — e.g. "99.9% of requests succeed, measured over a rolling 30 days."
  • The error budget is the gap between 100% and the SLO — the amount of failure the service is allowed to have before it's violating its own target. A 99.9% SLO leaves a 0.1% error budget to "spend" on deploys, experiments, and the occasional incident.
flowchart LR
    P["100%<br/>perfect reliability"] -->|"error budget<br/>(the allowed gap, e.g. 0.1%)"| O["SLO target<br/>e.g. 99.9% over 30 days"]
    I["SLI<br/>actual measured number"] -.->|"measured against"| O

An error budget turns a vague reliability conversation ("should we ship this risky change?") into a concrete one ("we've spent 60% of this month's error budget — do we still have room?"). This is the same measurement discipline DORA (module 02) applies to throughput, applied here to reliability — and it's the number an on-call engineer is really defending when a capstone gate asks what they'd do if a service broke at 2am.

5. How this connects

You can defend this when you can explain why a symptom-based alert beats a cause-based one as the default, what Alertmanager adds on top of a firing rule, and define SLI/SLO/error budget in your own words.