Observability & debugging

A broken Pod rarely announces why it broke — it just stops serving traffic, or starts restarting, or never leaves Pending. Reading that correctly means knowing which of three signals to reach for, and knowing that the probe results driving the loop are a decision the kubelet made, not a fact about the app. Get either wrong and you debug the symptom instead of the cause.

Scope: the core-concepts page's get→describe→logs→events loop is the fast path for a broken Pod. This page goes one layer deeper: how the three probe types actually drive that loop, why kubectl logs alone isn't "observability," and how logs/metrics/events fit together as the cluster's three signal types.

TL;DR — in 30 seconds:

• Three signal types answer different questions and none replaces the others: logs ("what did it print"), metrics ("how's it trending"), events ("what did the control plane just do to it") — and none is durable by default (logs are node-local, events have a short TTL).
• Liveness failure → kubelet restarts the container; readiness failure → Pod is pulled from Service endpoints, no restart — mixing these two up is the single most common probe-tuning mistake.
• A Pod in CrashLoopBackOff with clean logs before the restart is usually the liveness probe, not the app — check the probe's tuning before assuming the app is broken.

1. The three signal types

Kubernetes (and the apps running on it) produce three different kinds of signal, and each answers a different question:

Signal	Answers	Source	Retention
Logs	"what did the app print?"	stdout/stderr of each container	node-local by default — gone once the Pod is deleted, unless shipped elsewhere
Metrics	"how is it trending over time?"	metrics-server (CPU/memory) or an app's own exported metrics	short window in-cluster; long-term needs Prometheus or a cloud equivalent
Events	"what did the control plane just do to it?"	the api-server's Event objects (scheduling, pulls, probe failures)	a short TTL (about an hour by default) — the freshest, most operational signal

None of the three replaces the others. A CrashLoopBackOff needs logs to see why the app died, events to see how the kubelet reacted, and — if it's a slow leak rather than a hard crash — metrics to see the trend that led there.

Note — logs are not durable. kubectl logs reads from the node, and only for as long as the container (or its last-restarted instance, via --previous) still exists there. Anything you need to keep past the Pod's lifetime — audits, historical debugging, cross-Pod correlation — needs a log aggregation pipeline (Fluent Bit/Fluentd shipping to a store) sitting outside the cluster's own short-lived storage.

2. Probes, in depth

The core-concepts page introduced liveness, readiness, and startup probes as the mechanism behind several broken-Pod states. Each probe is really the same idea — the kubelet runs a check on an interval and acts on the result — applied to a different question:

Probe	Question it asks	On failure
liveness	"is this container still working, or wedged?"	kubelet restarts the container
readiness	"can this container serve traffic right now?"	Pod is pulled from the Service's endpoints — no restart
startup	"has this slow-starting container finished booting yet?"	liveness/readiness are held off until it succeeds

A probe is one of three check types, in increasing order of how much of the app it exercises:

• exec — run a command inside the container; success = exit code 0. Cheapest to reason about, but only as good as the command you pick.
• httpGet — hit an HTTP path; success = a 2xx/3xx response. The common choice for web services — a dedicated /healthz endpoint can check its own dependencies.
• tcpSocket — just open a TCP connection; success = the port accepts it. The weakest signal (a stuck process can still accept connections), but useful when there's no HTTP surface.

flowchart LR
  s["startup probe<br/>succeeds once"] -->|"unblocks"| l["liveness probe<br/>on every periodSeconds"]
  s -->|"unblocks"| r["readiness probe<br/>on every periodSeconds"]
  l -->|"fails failureThreshold times"| restart["kubelet restarts container"]
  r -->|"fails failureThreshold times"| pull["Pod removed from Service endpoints"]
  r -->|"succeeds again"| rejoin["Pod re-added to endpoints"]

Each probe is tuned with the same handful of fields: initialDelaySeconds (grace period before the first check), periodSeconds (how often), and failureThreshold (consecutive failures before acting). Startup exists precisely so a slow-booting app doesn't need an artificially long initialDelaySeconds on its liveness probe — without it, a container still loading data could be killed for "being wedged" when it's simply still starting.

Warning — the classic liveness misconfiguration. A liveness probe set too aggressively (too short a periodSeconds/failureThreshold, or checking something that's briefly slow under load) restarts a container that was actually fine — which is exactly the CrashLoopBackOff the core-concepts page teaches you to investigate with logs --previous. If the logs show a healthy app with no error before the restart, suspect the probe, not the app. (Go deeper: the upstream Configure Liveness, Readiness and Startup Probes task — optional, not required to defend this.)

3. Extending the debug loop

The core-concepts page's loop — get → describe → logs --previous → get events — covers the large majority of cases. Two extra tools round it out for what that loop can't reach:

• kubectl top pod / kubectl top node — reads live metrics from metrics-server. Use it when the question is "is this Pod actually using the CPU/memory it's requesting" rather than "why did it crash" — the Pending/OOMKilled states from the core-concepts table are a resourcing question, and top is the fastest way to see current usage against requests/limits.
• Ephemeral debug containers (kubectl debug) — attaches a temporary container with a shell and debugging tools to a running Pod or Node, without changing the target's image. Useful for a minimal "distroless" image that has no shell of its own to exec into. (Go deeper: the upstream Debugging Running Pods task — optional, not required to defend this.)

4. How this connects

• The probe states in this page's diagram are the same Running but 0/1 Ready and CrashLoopBackOff rows in the core-concepts debug table — this page explains the mechanism; that page is the fast first-response checklist.
• Pending because of insufficient requests/limits is the same signal the autoscaling page's Cluster Autoscaler reacts to — kubectl top and describe are how you confirm it's a resourcing problem before waiting on that autoscaler.
• Long-term metrics and shipped logs are what a production cluster layers on top of the node-local defaults described here — the same three-pillars split (metrics/logs/traces) reappears in the AWS module's CloudWatch/X-Ray page, one layer up the stack.

Common gotchas

• Assuming kubectl logs is "the" observability signal. Fix: logs answer "what did it print," not "how is it trending" (metrics) or "what did the control plane do" (events) — a CrashLoopBackOff investigation usually needs logs and events, sometimes metrics too.
• Confusing a readiness failure with a liveness failure. Fix: liveness failing restarts the container; readiness failing just pulls the Pod from Service endpoints with no restart — treating a readiness dip as "the app crashed" sends you down the wrong debugging path.
• Tuning a liveness probe too aggressively and blaming the app for the resulting CrashLoopBackOff. Fix: if logs --previous shows a healthy app with no error before the restart, suspect the probe's periodSeconds/failureThreshold, not the application code.
• Expecting kubectl logs to show history from before the Pod's current lifetime. Fix: logs are node-local and gone once the Pod is deleted (or the prior container instance is gone, unless you use --previous) — anything you need to keep needs a log-aggregation pipeline shipping logs outside the cluster.

Check yourself — a Pod is CrashLoopBackOff, and kubectl logs --previous shows a perfectly healthy app right up until it's killed. What should you suspect, and why?

The liveness probe, not the app — a probe tuned too aggressively (too short a periodSeconds/ failureThreshold, or checking something that's briefly slow) restarts a container that was actually fine. Clean logs right up to the kill is the signature of a probe problem rather than an application crash.

Check yourself — a Pod shows Running but its readiness probe just started failing. Does the kubelet restart it?

No — readiness failing doesn't restart anything; it just pulls the Pod out of its Service's endpoints so it stops receiving traffic until the probe passes again. Only a liveness failure triggers a restart; mixing the two up is a common source of debugging the wrong mechanism.

You can defend this when you can name what each of logs/metrics/events tells you and why none substitutes for the others, explain the difference between a liveness and a readiness failure in terms of what actually happens to the Pod, and say why a startup probe exists instead of just raising initialDelaySeconds on liveness.