The release DAG

On Monday's apply, cert-manager finishes installing before external-dns. On Tuesday's rerun — same file, same commit — external-dns finishes first. Nothing is broken: neither release's needs: mentions the other, so Helmfile never promised an order between them in the first place. The course page's §12 shows needs: as a straight chain (ingress-nginx → cert-manager → demo-app) and a matching linear diagram; real stacks are rarely one chain — they're a graph with branches. This page slows down on what Helmfile actually builds from needs:, how independent branches run, what --concurrency bounds, and what happens when one release in the graph fails.

TL;DR — in 30 seconds:

• needs: builds a DAG for the whole run, not a fixed top-to-bottom order — releases with no dependency relationship to each other have no ordering guarantee, and Helmfile can run them at the same time.
• --concurrency bounds how many eligible releases run at once; it doesn't change what the DAG says must finish before what.
• A mid-run failure only blocks releases downstream of it — already-started siblings finish, unrelated branches are unaffected, and a needs: cycle is a hard error before anything installs.

1. needs: builds a graph, not a queue

Each release's needs: list names other releases (namespace/name) that must finish successfully first. Helmfile collects every release's needs: into a single directed acyclic graph (DAG) for the whole run — it is not a fixed top-to-bottom install order, and releases with no dependency relationship to each other have no ordering guarantee between them:

flowchart LR
    ingress["ingress-nginx"]
    certmgr["cert-manager"]
    externaldns["external-dns"]
    demo["demo-app"]
    worker["worker"]

    ingress --> certmgr
    ingress --> externaldns
    certmgr --> demo
    externaldns --> demo
    certmgr --> worker

cert-manager and external-dns both depend only on ingress-nginx — neither depends on the other, which is exactly why their finish order can flip between runs like the Monday/Tuesday case above. Helmfile is free to run them at the same time. demo-app depends on both, so it waits for whichever of the two finishes last. This is the extension worth internalising over the course page's straight chain: a real stack is usually a small tree or diamond, not a single line, and the shape of that graph — not the order releases happen to be written in the file — is what decides who waits for whom.

2. Releases at the same depth run in parallel — up to --concurrency

Helmfile doesn't install the graph one release at a time. Once a release's every needs: entry has succeeded, it becomes eligible to start — and any number of eligible releases start together, bounded only by --concurrency:

Flag	Meaning	Default
--concurrency N	Cap on how many releases Helmfile installs/upgrades at once, across the whole run	Unlimited

In the diagram above, cert-manager and external-dns are both eligible as soon as ingress-nginx succeeds. With the default (unlimited) concurrency, Helmfile starts both immediately. With --concurrency 1, Helmfile still respects the DAG but now serialises everything, one release at a time — useful when the cluster API or a rate-limited provider (a fresh EKS control plane, a cloud DNS API) is the actual bottleneck, not the dependency logic itself. --concurrency and needs: answer two different questions: needs: answers what must finish before this can start; --concurrency answers how many things are allowed to run right now, DAG permitting.

3. Working on one release without re-running its whole upstream

Three flags let you narrow a run to part of the graph, each answering a different question:

Flag	Question it answers
--skip-needs	"Just the releases I selected — ignore their needs: entirely."
--include-needs	"My selection, plus whatever it directly needs:."
--include-transitive-needs	"My selection, plus its entire upstream chain, however deep."

Selecting just demo-app (-l app=demo) with --skip-needs re-applies only demo-app, trusting that cert-manager and external-dns are already healthy — the fast loop while iterating on one release. Adding --include-needs instead pulls in cert-manager and external-dns too (one hop up); --include-transitive-needs would also pull in ingress-nginx beneath them. Pick the narrowest one that still gives you a trustworthy apply.

4. What happens when one release in the graph fails

A needs: graph changes the blast radius of a failure — it isn't all-or-nothing, and it isn't independent either:

• Releases that had already started before the failure (siblings at the same depth, or unrelated branches) keep running to completion — Helmfile doesn't abort the whole run the instant one release errors.
• Releases that were waiting on the failed one never start — their needs: entry never becomes healthy, so they stay pending.
• Releases with no path to the failed one, in an unrelated branch of the graph, are unaffected and can succeed normally.
• A cycle in needs: (A needs B, B needs A) is a hard error at render time, before anything is installed — Helmfile refuses to guess an order.

Applying those four rules to §1's own graph, with cert-manager as the release that fails mid-run:

flowchart LR
    ingress["ingress-nginx<br/>already succeeded"]
    certmgr["cert-manager<br/>FAILED"]
    externaldns["external-dns<br/>unrelated branch — succeeds"]
    demo["demo-app<br/>waiting on cert-manager — never starts"]
    worker["worker<br/>waiting on cert-manager — never starts"]

    ingress --> certmgr
    ingress --> externaldns
    certmgr --> demo
    externaldns --> demo
    certmgr --> worker

    style certmgr stroke-width:3px

ingress-nginx already finished before the failure, so it stays succeeded. external-dns shares no edge with cert-manager — it's the "no path to the failed one" case — so it keeps running and succeeds normally. demo-app and worker both sit downstream of cert-manager in the DAG, so both stay pending forever: exactly the "waiting on the failed one" case, even though demo-app also needs: external-dns, which did succeed — one failed dependency is enough to block it.

So "release 2 of 3 failed" doesn't mean release 1 rolled back or release 3 was skipped for safety — it means release 1 is exactly as it finished, release 3 never ran if it depended on release 2 (directly or transitively), and anything outside that dependency path is untouched. Diagnosing "why didn't X run" in a partially-failed apply starts with tracing X's needs: chain back to whatever actually errored.

5. How this connects

• Secrets with SOPS, the previous page in this course, already named this page for where needs: ordering meets --suppress-secrets: a release with a secrets: entry decrypts exactly like any other release, then queues in this DAG like any other node — decryption doesn't change scheduling.
• Terraform's dependency graph (module 07, Terraform, "Providers & lifecycle" §5) builds the same DAG-not-queue shape from plan, but most of its edges are implicit — inferred from one resource reading another's attribute — with depends_on reserved for the rare case no attribute reference expresses the ordering. Helmfile has no implicit-inference equivalent: every edge in this page's graph is a needs: line you wrote by hand.
• systemd (module 09, Linux (RHEL), "systemd & journald") splits what this page's needs: does in one line into two separate directives: Wants=/Requires= answer does it need the other unit at all, After=/Before= answer in what order — a unit can have one without the other. needs: always answers both questions at once, so don't assume it behaves like just one half of that systemd split.

Common gotchas

• Assuming the order releases are written in helmfile.yaml is the apply order. Fix: order comes from needs:, not file position — releases with no needs: relationship can finish in a different order on different runs.
• Treating --concurrency 1 as disabling the DAG. Fix: it still respects needs:, it just serializes execution to one release at a time — useful when the bottleneck is a rate-limited API, not the dependency logic.
• Assuming one release failing aborts the whole apply. Fix: already-started siblings and unrelated branches keep running to completion — only releases waiting on the failed one stay pending.
• Re-running the full graph to fix one release. Fix: --skip-needs (just the selection), --include-needs (plus one hop up), or --include-transitive-needs (plus the whole upstream chain) each narrow the run — pick the narrowest one that's still trustworthy.

Check yourself — cert-manager and external-dns both needs: [ingress-nginx] and nothing else. On Monday cert-manager finishes first; on Tuesday's identical rerun, external-dns does. Is something broken?

No — neither release needs: the other, so Helmfile never promised an order between them. It's free to run both concurrently once ingress-nginx succeeds, and their relative finish order can flip between runs.

Check yourself — release 2 of 3 fails mid-apply. What state is release 1 in, and does release 3 run?

Release 1 is exactly as it finished — it already started before the failure and isn't rolled back. Release 3 only fails to run if it (directly or transitively) needs: release 2; if it's in an unrelated branch, it's unaffected.

You can defend this when you can point to two releases in a graph that have no needs: relationship to each other and explain why Helmfile is free to run them concurrently, why --concurrency doesn't change what runs but only how many at once, and — given a needs: graph and a mid-run failure — say precisely which releases already ran, which never got the chance, and which were unaffected.

Go deeper: the full behavior of --concurrency, the selector flags (--skip-needs, --include-needs, --include-transitive-needs), and how failures propagate through a needs: graph are documented in the Helmfile docs — this page covers the 80/20 you need for the Do exercise.