Skip to content

Management and Monitoring Patterns

Availability, resiliency, and scalability patterns keep a distributed system running under failure and load. This page covers the last piece: how operators actually see what a distributed system is doing, and how they change its behavior without a redeploy.

TL;DR — in 30 seconds:

  • Health endpoint monitoring exposes each instance's health check (page 1) to external monitoring — not just the load balancer — so operators (and dashboards, and alerts) can see instance-level health directly.
  • External configuration store keeps runtime configuration (feature flags, rate limits, connection strings) outside the deployed artifact, so it can change without a rebuild or redeploy.
  • Both patterns exist for the same reason as the observability module (17): a distributed system with many moving parts is only operable if you can see its state and change its behavior at runtime — not only by reading logs after something has already gone wrong.

1. Health endpoint monitoring: more than a load-balancer signal

flowchart LR
    I["Instance /healthz"] -->|"routing decisions"| LB["Load balancer"]
    I -->|"scraped on an interval"| Mon["Monitoring system<br/>(module 17)"]
    Mon -->|"dashboards + alerts"| Ops["Operator"]
  • Page 1 covered the health check as the signal a load balancer uses to route traffic. This pattern is the same endpoint, consumed by a second audience: the monitoring stack, so instance health becomes a dashboard and an alert, not just an invisible routing decision.
  • A useful health endpoint reports more than "up or down" — it can report the status of the specific dependencies it relies on (database reachable, downstream API reachable), so an alert points at what is unhealthy, not just that something is.

2. External configuration store: change behavior without a redeploy

flowchart LR
    Cfg["External config store<br/>(e.g. a key-value store, Secret Management module 19's tooling)"] -->|"read at startup /<br/>watched at runtime"| App["Application instances"]
    Op["Operator"] -->|"updates a value"| Cfg
  • Configuration that's baked into the deployed artifact (a container image, a compiled binary) can only change by rebuilding and redeploying it. An external store — read at startup, or watched for live updates — lets an operator change a rate limit, a feature flag, or a timeout value without touching the artifact.
  • This is closely related to module 19, Secret Management: a secret store is one kind of external configuration store, specialized for values that must stay confidential; this pattern is the broader idea, applied to any runtime-tunable setting, secret or not.
  • The tradeoff: the application now has a runtime dependency on the config store being reachable — most designs handle this by caching the last-known-good config locally so a brief config-store outage doesn't take the application down with it.

Common gotchas

  • A health endpoint that only checks the process is running. Fix: it should reflect the dependencies that actually matter (can it reach its database?) — see page 1's liveness-vs-readiness distinction.
  • Baking a value that changes often (a rate limit, a flag) into the deployed image. Fix: if it needs to change without a rebuild, it belongs in an external configuration store, not a compiled-in default.
  • No local fallback if the config store is briefly unreachable. Fix: cache the last successfully read configuration so a transient config-store blip doesn't cascade into an application outage.
Check yourself — a team wants to change a rate limit for one endpoint at 3am without deploying new code, and separately wants an alert the moment any instance can't reach its database. Which pattern covers each need?

The rate-limit change is an external configuration store — a runtime-tunable value read from outside the deployed artifact, changeable without a redeploy. The database-reachability alert is health endpoint monitoring — the instance's health check reporting a specific dependency's status to the monitoring stack, not just "up or down."

Done when: you can explain why health endpoint monitoring is the same check as page 1's load-balancer signal serving a second audience, and why an external configuration store trades a small runtime dependency for the ability to change behavior without a redeploy.