Observability & Monitoring¶

Core (a cross-cutting discipline every real engagement runs, not an optional extra). · prereq 04 (Kubernetes & Helm), 05 (AWS) — this module names the general discipline that 04's probes and 05's CloudWatch/X-Ray each already showed you one platform's version of. · ~1 day (4 reference submodules — the three pillars & golden signals, Prometheus & PromQL, Grafana dashboards, and alerting & SLOs). Reality check: this is the 80/20 on-ramp; real fluency (tuning alert thresholds against real traffic, writing a production PromQL library, running Prometheus at scale) is built on the job, one incident at a time.

Every module so far has taught you a signal that's local to one tool: Kubernetes probes in module 04 (Kubernetes), CloudWatch in module 05 (AWS). Observability is the discipline those are instances of — the general skill of knowing which signal to reach for when something's wrong, regardless of which platform it's running on. This module teaches the cross-cutting vocabulary and the two tools (Prometheus, Grafana) that most teams use to practice it.

TL;DR — in 30 seconds:

The three pillars answer different questions: metrics tell you that something's wrong, logs tell you why, traces tell you where the time went.
Prometheus is pull-based — it scrapes a target's /metrics endpoint on an interval; PromQL queries the resulting time series.
Grafana visualizes; it doesn't alert. Alert on a symptom (latency, error rate), not a cause (CPU) — Alertmanager dedupes, routes, and silences what a firing rule hands it.

Learning objectives¶

By the end, the junior can:

Explain the three pillars (metrics, logs, traces) as three different questions, not three interchangeable tools, and name the four golden signals (latency, traffic, errors, saturation).
Explain Prometheus's pull-based scraping model — a target exposes a /metrics endpoint, Prometheus scrapes it on an interval — and read a basic PromQL query.
Explain what a Grafana dashboard is for (and, just as importantly, what it isn't a substitute for).
Distinguish alerting on a symptom from alerting on a cause, and explain what Alertmanager adds on top of a firing Prometheus rule.
Define SLI, SLO, and error budget, and explain how an error budget turns "how reliable should this be" into a number a team can spend.

Do it with Claude

Pair, Do and Prove run in Claude Code, with the onboarding-tutor skill. Click a button to copy the exact prompt, then paste it into Claude Code in this repo.

1. Learn · acquire the concept¶

No external course — read this brief, then go deeper in the four reference pages below.

Mental model (20 minutes):

The whole discipline is one loop: a target exposes signal, something collects it, something visualizes or alerts on it.

flowchart LR
    A["App / exporter<br/>exposes /metrics"] -- "scrape (pull)" --> P["Prometheus<br/>time-series DB"]
    P -- "PromQL query" --> G["Grafana<br/>dashboard"]
    P -- "alerting rule fires" --> AM["Alertmanager"]
    AM -- "route / dedupe / silence" --> N["Page / notification"]

Three pillars, three questions — metrics tell you that something's wrong, cheaply and in aggregate; logs tell you why, one discrete event at a time; traces tell you where the time went across services. None replaces the others.
Four golden signals — the widely-used shortlist of what to measure for any service: latency (how long), traffic (how much demand), errors (how much is failing), saturation (how full the resource is). They're a starting checklist for "what should I even watch," not a Prometheus-specific feature.

Infrastructure Monitoring¶

The metrics pillar: a target exposes numeric signal, something scrapes and stores it, something visualizes it or alerts on it.

Prometheus is pull-based — instead of your app pushing data somewhere, Prometheus scrapes a /metrics HTTP endpoint on each target at a fixed interval. An exporter is a small process that translates something that can't natively expose /metrics (a database, an OS) into Prometheus's format. PromQL is the query language you point at the resulting time series.
Grafana visualizes; it doesn't alert — a dashboard is a query rendered as a graph, built from a data source like Prometheus. Nobody watches a dashboard 24/7, so a dashboard is not a substitute for an alert — it's what you pull up after an alert to understand what's happening.
Alert on symptoms, not causes — a Prometheus alerting rule is a PromQL expression that fires when true for a duration. The rule should test a user-facing symptom (a golden signal — high latency, high error rate) rather than an internal cause (high CPU) that may or may not actually be hurting anyone. Alertmanager is the separate component that receives firing alerts and handles deduplication, routing, grouping, and silencing — Prometheus itself just evaluates rules and hands them off.
SLI / SLO / error budget — an SLI (service level indicator) is the actual measured number (e.g. the ratio of successful requests). An SLO (service level objective) is a target for that indicator over a window (e.g. 99.9% over 30 days). The error budget is what's left over (100% − the SLO) — the amount of failure a team is allowed to "spend" before the SLO is violated, turning a reliability conversation into a number instead of a feeling. This is the same measurement discipline DORA (module 02) teaches for throughput — applied here to reliability.

Logs Management¶

The logs pillar: a discrete record of one event, kept and made searchable across every service and host instead of living only in one machine's local files.

Log aggregation — centralizing logs from every service and host into one searchable store, instead of SSH-ing into individual machines to grep a file. The ELK/Elastic stack (Elasticsearch, Logstash, Kibana) and Grafana Loki are two widely-used approaches to this.
Structured logging — emitting each log line as structured key-value data (commonly JSON) rather than free-text, so an aggregator can filter and query on a specific field (a user ID, a status code) instead of relying on full-text search alone.
Retention — logs are kept for a bounded window, then rolled off or moved to cheaper storage. How long to keep them is a deliberate cost and (often) compliance tradeoff, not "store everything forever."

Go deeper: Three pillars & golden signals · Prometheus & PromQL · Grafana dashboards · Alerting, SLOs & error budgets

Read next: the Prometheus and Grafana documentation (linked above) — optional, not required to pass this module's gate.

Check yourself — a Grafana dashboard shows your service's error rate climbing right now. Is that enough on its own to know something's wrong?

No — a dashboard is something you pull up after you've been alerted, not something anyone watches 24/7. If nothing paged you, the dashboard's graph can climb for hours before a human happens to look at it. The alert is what tells you to look; the dashboard is what you look at.

Check yourself — an alerting rule fires when a node's CPU usage crosses 90%. Is that a good alert to page someone on?

Not by default — that's a cause, not a symptom. High CPU doesn't necessarily mean a user is affected; the rule should instead test a golden signal like latency or error rate, because that's what actually tells you whether the service is failing the people using it.

Done when: you can name, unprompted, which pillar answers which question (metrics/logs/traces), state the four golden signals, and explain the difference between an alert firing and Alertmanager acting on it — before moving to Pair.

2. Pair · passive → active¶

Drive the onboarding-tutor:

"Give me a scenario — high latency, a crash loop, a slow endpoint — and quiz me on which of the three pillars I'd reach for first."
"Walk me through why a dashboard isn't the same thing as an alert, and what each is actually for."
"Quiz me on pull vs push: why does Prometheus scrape targets instead of having them push data to it, and what does an exporter do for a target that can't expose /metrics on its own?"
"Give me an alerting rule and make me classify it as symptom-based or cause-based, and explain why."
"Walk me through an SLI/SLO/error-budget example for a service I pick, and quiz me on what 'spending the error budget' means."

Common gotchas

Treating a dashboard as if it were an alert. Fix: nobody watches a dashboard around the clock — it's for understanding an alert that already fired, not for catching the problem in the first place.
Alerting on a cause (CPU, memory) instead of a symptom (latency, error rate). Fix: a cause can spike without ever hurting a user; page on the golden signal that tells you whether the service is actually failing people, not on the internal metric that might be why.
Treating metrics, logs, and traces as interchangeable. Fix: pick the pillar for the question you're actually asking — metrics for "is something wrong," logs for "why did this one event fail," traces for "where did the time go across services."
Setting an SLO to 100%. Fix: a 100% target leaves zero error budget, which blocks any deploy that carries even a theoretical risk — a realistic SLO (e.g. 99.9%) leaves a budget the team can actually spend on shipping changes.

Done when: you can trace the full scrape → query → dashboard chain and the separate fire → route → notify chain unprompted, for a scenario the tutor gives you.

3. Do · produce an artifact¶

Exercise — stand up the loop and observe it:

Run Prometheus (in a container, using what module 03 (Containers) already taught you) with a scrape config pointed at a target exposing a /metrics endpoint — your own small app, or Prometheus's own self-metrics endpoint is enough to start.
In Prometheus's own UI, run a PromQL query against that target (a simple rate() over a counter is enough) and confirm it returns real data.
Stand up Grafana in a container, connect it to your Prometheus as a data source, and build one panel from the same query.
Write a Prometheus alerting rule that fires on a symptom (e.g. an elevated error rate or latency) rather than a cause, and write one sentence explaining what Alertmanager would do with that firing alert that the raw rule by itself doesn't.
Write a short paragraph naming an SLI you picked for this exercise, an SLO target for it, and what "spending the error budget" would mean here.

Done when: your Prometheus target shows as up, your PromQL query returns real data, your Grafana panel renders it, and you can point to the specific line in your alerting rule that makes it symptom-based rather than cause-based.

4. Prove · understanding gate¶

Mandatory. Pass bar in the Socratic gate.

Three pillars & golden signals: What question does each pillar (metrics, logs, traces) answer that the other two can't? Name the four golden signals and what each measures.
Prometheus's pull model: Why does Prometheus scrape targets instead of having them push data to it? What does an exporter do for a target that can't natively expose /metrics?
Alerting & SLOs: What's the difference between alerting on a symptom and alerting on a cause, and why does that distinction matter? What does an error budget let a team do that an SLO alone doesn't?
Grafana's role: What is a Grafana dashboard actually for, and why isn't it a substitute for an alert?
Provenance: You likely had the tutor walk you through at least one classification (symptom vs cause, or an SLI/SLO choice). Which specific claim did it make, and how did you verify it — against your own running Prometheus/Grafana setup — rather than taking the tutor's explanation on trust?

Pass bar: you can defend each piece with the underlying mechanism (why pull suits this discipline, why a symptom-based alert is the right default, what an error budget actually changes) — not just name the tool — and show how you checked a tutor-supplied claim against your own setup. "Prometheus does metrics and Grafana does dashboards" without the why = back to Pair.

5. Retain · fight forgetting¶

This module has no generated Anki deck yet — recall by re-reading the four reference pages and re-running the Do exercise against a different metric, query, or alerting rule until the scrape → query → dashboard chain and the fire → route → notify chain both come back to you unprompted. Done when you can sketch both chains, cold, for any service someone hands you.

Key takeaways¶

The three pillars answer three different questions — metrics tell you that something's wrong, logs tell you why, traces tell you where the time went — and none replaces the others; the four golden signals (latency, traffic, errors, saturation) are the starting checklist for what to watch on any service.
Prometheus is pull-based: it scrapes a target's /metrics endpoint on an interval instead of the target pushing data to it. An exporter translates something that can't natively expose /metrics into that format, and PromQL is the query language over the resulting time series.
Grafana visualizes; it doesn't alert. A dashboard is what you pull up after an alert fires to understand what's happening — nobody watches one around the clock, so it's never a substitute for an alert.
Alert on symptoms (latency, error rate), not causes (CPU, memory). A cause can spike without ever hurting a user; Alertmanager is the separate component that dedupes, routes, groups, and silences what a firing Prometheus rule hands it.
SLI / SLO / error budget turns "how reliable should this be" into a number a team can spend: the error budget (100% − the SLO) is the amount of failure still available before the SLO is violated — the same measurement discipline DORA (module 02) applies to throughput, applied here to reliability.

Go deeper — curated resources¶

Hand-picked external material to take this topic further — the best books, courses, talks, and writing. Cost is tagged Free, Free online (full text/course free on the web), or Paid.

Docs · Prometheus Documentation — prometheus.io. The canonical reference for the pull-based scrape model, exporters, and PromQL this module's Prometheus beat is built on. Free.
Book · Observability Engineering — Charity Majors, Liz Fong-Jones, and George Miranda. The book that popularized the modern three-pillars framing, with practical guidance on moving beyond dashboards-and-alerts monitoring toward real observability. Paid.
Chapter · Monitoring Distributed Systems — Site Reliability Engineering (Google SRE Book), Chapter 6. The original source for the four golden signals (latency, traffic, errors, saturation) this module teaches. Free online.
Video · Grafana Labs — the official Grafana Labs channel. Dashboard walkthroughs and PromQL/alerting deep-dives straight from the maintainers. Free.

Gate log (mentor fills on pass)¶

Date passed:
Passed by: / checked by:
Weak spots noticed (feeds system improvement):