Config & Secrets (deep)

Scope: the core-concepts page introduced ConfigMap and Secret as the two objects for injecting config and credentials at run time. This page goes one layer deeper on how a Pod consumes them, what immutability buys you, how values rotate without a rebuild, and the external-secrets pattern that keeps the real secret store outside the cluster.

TL;DR — in 30 seconds:

• ConfigMap/Secret reach a Pod two ways: env var (read once at container start, frozen for the Pod's life) or projected volume (kubelet re-syncs, so a changed value shows up without a restart).
• Marking a ConfigMap/Secret immutable trades convenience for safety (no accidental in-place bad rollout) and scale (the kubelet stops watching it) — pair it with a name that includes a version/hash.
• Rotation is close to free on the volume-mount path; on the env-var path it requires recreating the Pod — this is why security-sensitive credentials generally prefer volumes.

1. Two ways in: env vars vs projected volume

You rotate a leaked database password in the Secret, watch the object update cleanly — and the Pod keeps authenticating with the old password for hours. Nothing is broken; the Pod just consumed the Secret the one way that can't see the change. A Pod can pull a ConfigMap or Secret's keys in through either of two paths:

Path	How it works	Updates when the source changes
Environment variable	Each key becomes a $VAR in the container's environment, set once at container start	Never — the process only reads its environment at startup; a changed ConfigMap/Secret has no effect until the Pod is recreated
Projected volume	The ConfigMap/Secret is mounted as a directory, one file per key	Automatically — the kubelet periodically re-syncs the mounted files, so an updated value appears on disk without recreating the Pod (usually within roughly a minute, though the exact timing isn't guaranteed)

The trade-off is simple: env vars are the easiest to wire up and to read in application code, but they freeze the value for the Pod's whole lifetime. A projected volume costs a little more application-side work (the app must notice the file changed, or simply re-read it on every use) but is the only path that lets a value change without a redeploy.

flowchart LR
  cm["ConfigMap / Secret<br/>(stored in etcd)"]
  cm -- "read once,<br/>at container start" --> env["env var<br/>frozen for Pod lifetime"]
  cm -- "kubelet re-syncs<br/>periodically" --> vol["mounted file<br/>(projected volume)"]
  env --> app1["app reads $VAR"]
  vol --> app2["app re-reads file"]

2. Immutability

A ConfigMap or Secret can be marked immutable — once set, its data can no longer be updated (a change requires creating a new object under a new name instead). This trades convenience for two real benefits:

• Safety — it's no longer possible to accidentally roll out a bad value in place; a new value means a new object name, which means a deliberate Pod spec change to reference it.
• Scale — the kubelet no longer has to watch an immutable ConfigMap/Secret for changes, which reduces load on the API server in clusters with a very large number of mounted config objects — the immutable ConfigMaps and Secrets docs cover both the safety and scale motivations in more depth.

The usual pattern pairs immutability with a name that includes a version or content hash (e.g. a suffix added by your templating tool), so "changing the config" becomes "create a new object, then update the Deployment to reference it" — the same rollout/rollback story a Deployment already gives you for images.

3. Rotation

Rotation — replacing a credential's value without downtime — behaves differently depending on which path from §1 an app uses:

• Volume-mounted: rotation is close to free. Update the Secret's data; the kubelet syncs the new file to every Pod that mounts it; an app that re-reads the file (rather than caching it in memory forever) picks up the new value with no restart.
• Env-var-consumed: rotation requires recreating the Pod, because the environment was only read once at startup. A rolling restart of the Deployment is the usual way to pick up a new value.

This is the main reason security-sensitive workloads (anything holding a database password, an API token, a TLS certificate) generally prefer the volume-mount path — it makes "rotate this credential" a data change instead of a redeploy.

4. The external-secrets pattern

Storing the real secret only inside Kubernetes has a problem: it's now the single source of truth, with no independent audit trail, and every cluster that needs it needs its own copy kept in sync by hand. The external-secrets pattern fixes this by keeping an external secret manager (a cloud provider's secret store, or a dedicated vault product) as the actual source of truth, and letting a sync controller running in the cluster keep a Kubernetes Secret in sync with it:

flowchart LR
  ext[("External secret store<br/>(source of truth)")] -- "sync controller<br/>reads on a schedule" --> ctrl["Sync controller<br/>(runs in-cluster)"]
  ctrl -- "creates / updates" --> sec["Kubernetes Secret"]
  sec -- "mounted as usual" --> pod["Pod"]

The Pod's own manifest doesn't change at all — it still just mounts or reads-env-from a plain Kubernetes Secret. The controller is the only thing that talks to the external store, which means the app team never needs direct credentials to that store, and rotating the external secret automatically rotates the in-cluster copy on the controller's next sync.

5. How this connects

• This extends the core-concepts page's ConfigMap/Secret table — that page covers what they are; this page covers how a Pod consumes them and how their values change over time.
• A projected volume is the same Volume mechanism the storage page covers for Pod-scoped data — a ConfigMap/Secret volume is just a special source for it, not a different mounting concept.
• The external-secrets pattern is where this module meets AWS Secrets Manager / KMS and IAM — the sync controller needs its own least-privilege credentials to read the external store, the same least-privilege idea RBAC applies inside the cluster.

Common gotchas

• Rotating a Secret and expecting an env-var-consuming Pod to pick it up automatically. Fix: env vars are read once at container start — nothing changes until the Pod is recreated (a rolling restart of the Deployment is the usual fix); only the projected-volume path updates in place.
• Assuming an immutable ConfigMap/Secret can be edited "just this once." Fix: immutable means immutable — a change requires creating a new object under a new name and updating the Pod spec to reference it, not patching the existing one.
• An app caches a volume-mounted Secret's value in memory instead of re-reading the file. Fix: the kubelet syncs the new file to disk, but rotation is only "free" if the app actually re-reads it — caching indefinitely means you still need a restart, undoing the benefit of the volume-mount path.
• Reaching for the external-secrets pattern and thinking the Pod's manifest has to change. Fix: the Pod still just mounts or reads-env-from an ordinary Kubernetes Secret — only the sync controller talks to the external store, so nothing about how the Pod consumes it changes.

Check yourself — you rotate a Secret's value. An app consuming it as an env var and one consuming it as a projected volume behave differently. Why?

The env var was read once at container start and is frozen for the Pod's lifetime, so that app keeps using the old value until the Pod is recreated. The projected volume is periodically re-synced by the kubelet, so the new value appears on disk without a restart — as long as the app actually re-reads the file rather than caching it forever.

Check yourself — what does marking a ConfigMap immutable actually buy you, and what does it cost?

It buys safety (no accidental in-place bad-value rollout — a change means a deliberate new object plus a Pod spec update) and scale (the kubelet stops watching it for changes, less API-server load at large fleet size). The cost is convenience — you can't edit values in place; the usual pattern pairs it with a version/hash suffix in the name.

You can defend this when you can say which consumption path (env var vs volume) picks up a changed value without a Pod restart and why, explain what marking a ConfigMap/Secret immutable actually buys you, and describe what problem the external-secrets pattern solves that a plain in-cluster Secret doesn't.