Three Pillars & Golden Signals¶
Before reaching for any tool, observability starts with two shortlists: three pillars (what kind of signal exists) and four golden signals (what to actually watch on any service). Both are platform-agnostic — module 04, Kubernetes & Helm (Kubernetes probes) and module 05, AWS (AWS CloudWatch/X-Ray) are each one platform's implementation of the same two ideas.
TL;DR — in 30 seconds:
- Three pillars — metrics, logs, traces — each answer a question the other two can't: that (and roughly when), why (for one event), where (across services).
- Four golden signals — latency, traffic, errors, saturation — are what to measure on any service, regardless of what it does.
- Both are platform-agnostic: Kubernetes probes and AWS CloudWatch/X-Ray are just one platform's implementation of the same two ideas.
1. The three pillars — three different questions¶
Each pillar exists because it answers a question the other two can't:
| Pillar | Answers | Nature | Cost |
|---|---|---|---|
| Metrics | Is something wrong, in aggregate, right now? | A number over time (request count, latency, CPU) | Cheap to store, cheap to alert on |
| Logs | What exactly happened, at one point in time? | A discrete event (a request failed, a deploy ran) | More expensive to store and search at volume |
| Traces | Where did the time go, across services, for one request? | One request's path through multiple services, timed hop by hop | Most expensive; usually sampled |
None replaces the others. Metrics tell you that something broke and roughly when; logs tell you why, for one specific instance of it; traces tell you which service in a multi-hop request chain is actually the slow one — a question neither metrics nor logs, alone, can answer once a request crosses more than one service boundary.
flowchart LR
S["Something's wrong"] --> M["Metrics<br/>that? and roughly when?"]
M --> L["Logs<br/>why, for this one event?"]
L --> T["Traces<br/>where, across services?"]2. The four golden signals¶
If metrics/logs/traces are what kind of signal exists, the four golden signals are what to actually measure for any service, regardless of what it does:
| Signal | Question | Example |
|---|---|---|
| Latency | How long do requests take? | p50/p99 response time |
| Traffic | How much demand is the service under? | Requests per second |
| Errors | How much of that traffic is failing? | Error rate, HTTP 5xx count |
| Saturation | How full is a constrained resource? | CPU, memory, queue depth |
These four are a starting checklist, not a Prometheus- or Grafana-specific feature — they're the same four questions worth asking about any service before deciding what to build a metric, log, or trace around.
3. How this connects¶
- The four golden signals are exactly the kind of thing a metric is built to answer cheaply — they're the natural input to the alerting rules covered in Alerting, SLOs & error budgets.
- Module 04's probe results and module 05's CloudWatch metrics are both, underneath, instances of the "metrics answer that, logs answer why" split taught here — same discipline, different platform.
You can defend this when you can name, unprompted, which pillar answers which question, and state the four golden signals without needing to look them up.