Aeron¶

Advanced (distributed systems) · prereq Containers (03), Kubernetes & Helm (04) — Mechanical Sympathy (90) recommended before or alongside · ~1 day

Aeron is a low-latency messaging layer whose cluster mode keeps a service's state correct and available through node crashes. This elective builds the mental model for when — and why — to reach for it.

TL;DR — in 30 seconds:

Aeron Transport is a fast, reliable pipe with no fault tolerance; Aeron Cluster adds Raft-replicated fault tolerance on top of Transport, at the cost of throughput.
A message is committed — and only then processed — once a quorum (majority) of nodes has it durably in their log; this is exactly what makes an uncommitted message safe to lose on a crash.
The Clustered Service must be deterministic: a clock read, randomness, or external I/O has to be captured before it reaches the log, or replay (and recovery) can silently rebuild the wrong state.

Learning objectives¶

By the end, the junior can:

Distinguish Aeron Transport (ultra-fast, no fault tolerance) from Aeron Cluster (Raft-replicated, fault-tolerant state machine) and choose the right one.
Explain the commit path: a message is processed only after a quorum acknowledges it — and why that prevents split-brain.
State the determinism contract for a Clustered Service, what breaks it (time.now(), randomness, external I/O), and how to keep it deterministic.
Explain recovery: how the Archiver + snapshots + log replay reconstruct state after a crash.

Do it with Claude

Pair, Do and Prove run in Claude Code, with the onboarding-tutor skill. Click a button to copy the exact prompt, then paste it into Claude Code in this repo.

1. Learn · acquire the concept¶

No external course — everything you need to pass the gate is in the brief below (about 5 minutes). The linked docs are optional depth, not required reading.

Transport vs Cluster¶

Aeron Transport and Aeron Cluster are two different tools — choosing between them is the first decision.

	Aeron Transport	Aeron Cluster
What it is	a very fast pipe: a reliable, ordered stream over UDP/IPC (not TCP)	a deterministic replicated state machine built on Transport
Fault tolerance	none (no consensus)	Raft across 3–5 nodes, with automatic leader election
Throughput	tens of millions of msgs/sec	slower — it pays replication overhead

Reach for Transport when you want raw speed and don't need to survive a node failure. Reach for Cluster when the state must stay correct and available even as nodes die — at the cost of throughput.

The commit path¶

Only the leader accepts client messages. It appends each one to its log, replicates it to the followers, and the message is committed once a quorum (a majority of nodes) acknowledges it.

Only committed messages are processed by the Clustered Service — your business logic. So a message that a leader crash loses before it commits is safe: no node ever acted on it, and the client simply retries.

flowchart LR
  C["Client message"] --> L["Leader appends to log"]
  L --> R["Replicate to followers"]
  R --> Q{"Quorum of nodes acks?"}
  Q -- yes --> K["Committed"]
  K --> P["Clustered Service processes"]
  Q -- no --> D["Uncommitted — no node acted; client retries"]

The determinism contract¶

The Clustered Service must be deterministic — that is what lets every node replay the same log and land in the same state:

same starting state + same ordered log + same code ⇒ identical state on every node

Anything non-deterministic breaks it: reading the clock, randomness, or external I/O (e.g. a network read).

Note: the fix always has the same shape — do the non-deterministic thing outside the service, then bake the resulting value into a logged message, so every node replays the identical value.

Recovery¶

Two pieces rebuild a crashed node. The Archiver persists the log. A snapshot is a saved point-in-time state, so recovery replays only the log after the snapshot instead of from day one.

flowchart LR
  SNAP["Load latest snapshot — point-in-time state"] --> RE["Replay log entries after the snapshot"]
  ARCH["Archiver — persisted log"] --> RE
  RE --> OK["State reconstructed — node rejoins"]

The snapshot is what keeps recovery cheap: without it, a node would replay the entire log from the beginning.

Go deeper (optional): the Aeron Cluster tutorial — focus on the Consensus Module, the Archiver, and the ClusteredService callbacks. You don't need it to pass this module's gate; the brief above is enough. Retain deck (beat 5): SE Onboarding::91 Aeron.

Done when: you can state, from memory (brief closed) — (1) when to pick Transport vs Cluster and what Cluster costs, (2) why a message waits for a quorum before it is processed, (3) what breaks determinism and the single-shape fix, and (4) how the Archiver + snapshot + replay rebuild a crashed node.

2. Pair · passive → active¶

Drive the onboarding-tutor:

"Explain why an Aeron Cluster message isn't processed until a quorum acks it — and what would go wrong if it were processed on append."
"Walk me through what happens when the leader dies after appending but before commit."
"Quiz me on what breaks determinism in a Clustered Service and how to fix each."

Done when: you can explain, unprompted, the commit path and why order-of-reception is treated as truth.

3. Do · produce an artifact¶

Exercise — reason about a 3-node cluster (on paper, or a running cluster if you have one):

Draw the path of a message from a client through the leader's Consensus Module to the Clustered Service, marking exactly where commit happens.
Nodes A (leader), B, C: the leader appends message m and replicates to B, then crashes before C acks (no quorum). Write down — is m processed? what does the client do? which node can safely become leader, and why?
Point at one line of business logic that would break determinism (e.g. stamping System.currentTimeMillis() inside the handler) and rewrite it so the value is captured before the log.

Optional (deeper): stand up the tutorial's single-node cluster, take a snapshot, kill it, and show state reconstructs from log replay.

Done when: your diagram + answers correctly show commit before process, uncommitted-loss is safe, and one concrete determinism fix.

Common gotchas

Assuming a message is safe once the leader has appended it. Fix: it's only safe to act on after a quorum acks it — an append with no quorum is uncommitted, and safe to lose on a leader crash.
Reading the wall clock, generating a random value, or calling external I/O inside the Clustered Service handler. Fix: compute it once outside the service and carry the resulting value through the logged message — the fix has one shape regardless of which of the three the culprit is.
Assuming a snapshot alone reconstructs state. Fix: recovery needs both — load the snapshot, then replay the log entries after it; without snapshots, a node would replay the entire log from the start.
Reaching for Aeron Cluster by default. Fix: choose Cluster only when the state must survive a node failure — for raw throughput without fault tolerance, plain Aeron Transport is faster and simpler.

4. Prove · understanding gate¶

Mandatory. Pass bar in the Socratic gate.

Transport vs Cluster: When would you use Aeron Transport alone vs Aeron Cluster? What does the Cluster buy you, and what does it cost?
Commit path: Why wait for a quorum before processing (not just before acking)? Name the failure this prevents.
Determinism: Your service calls an external pricing API mid-handler. Why is that a bug, and what's the fix?
Recovery: Explain how the Archiver + a snapshot + log replay bring a crashed node back to correct state. Why does the snapshot exist at all?
Provenance: Which parts of your commit-path diagram or determinism fix did the tutor hand you, and which did you reason out yourself? Pick one claim it asserted — "that uncommitted message is safe to lose", or "that line breaks determinism" — and show how you checked it against the quorum rule or the determinism contract, not just that the agent said so.

Pass bar: you can defend commit-before-process, the determinism contract, and the recovery path — and say how you verified any claim the tutor handed you. "It replicates" (or "the agent worked it out") without the why = back to Pair.

5. Retain · fight forgetting¶

Study the module deck SE Onboarding::91 Aeron — its theme subdecks let you drill one area at a time:

1 Packaging & containers · 2 Cluster, Raft & consensus · 3 Determinism & ordering · 4 Persistence & recovery

The deck syncs to your Anki automatically on pull (anki-sync). Done when you can recall, from the deck: transport vs cluster, quorum/commit, what breaks determinism, and the Archiver/snapshot recovery path.

Check yourself — the leader appends message m and replicates to one follower, then crashes before a quorum acks it. Is m processed?

No — m is uncommitted, so no node ever acted on it; it's safe to lose. The client simply retries against whichever node becomes the new leader.

Check yourself — why must a message wait for quorum before being processed, not just before being acked by the leader?

Because "acked by the leader" alone doesn't guarantee any other node has the entry. If the leader processed it and then crashed before replicating, that state would be gone with no other node ever having agreed to it — waiting for quorum before processing guarantees a majority already has it durably logged, so processing it can never be silently undone.

Check yourself — your Clustered Service handler calls an external pricing API mid-handler. Why is that a bug, and what's the fix?

The API call is non-deterministic — it can return different answers on different nodes or replays, or fail on one and not another. Fix: call it once outside the deterministic service, then carry the resulting value in the logged message, so every node's replay reads the same already-decided value.

Check yourself — why does the snapshot exist at all — why not just replay the whole log?

Because replaying an ever-growing log from the very first entry would get slower and slower over time. A snapshot is a checkpoint of state at one log position, so recovery only replays what happened after it — bounded, cheap recovery instead of unbounded replay.

Key takeaways¶

Transport is a fast, reliable pipe with no fault tolerance; Cluster adds Raft-replicated fault tolerance on top of Transport, trading throughput for the ability to survive a node crash.
A message is committed — and only then processed — once a quorum of nodes has it durably logged; that ordering is exactly what makes an uncommitted message safe to lose on a leader crash.
The determinism contract (same start state + same ordered log + same code ⇒ identical state everywhere) is broken by reading the clock, randomness, or external I/O inside the handler — the fix always has the same shape: compute the value outside the service and carry it in the logged message.
Recovery rebuilds a crashed node from two pieces: the Archiver's persisted log and the latest snapshot; replaying only the log entries after the snapshot is what keeps recovery bounded and cheap.
Reach for Cluster only when state must survive a node failure — for raw throughput without fault tolerance, plain Transport is faster and simpler.
Verifying a tutor's claim (e.g. "that uncommitted message is safe to lose") means checking it against the quorum rule or the determinism contract yourself, not just trusting that the agent said so.

Go deeper — curated resources¶

Hand-picked external material to take this topic further — the best books, courses, talks, and writing. Cost is tagged Free, Free online (full text/course free on the web), or Paid.

Repo · Aeron — Adaptive Financial Consulting. The source, wiki, and design docs — start with the wiki's Transport and Cluster tutorials. Free.
Docs · Aeron Cookbook — Adaptive. Recipe-style guides for Transport, Archive, and Cluster — the fastest path from concept to working code. Free.
Talk · Cluster Consensus: When Aeron Met Raft — Martin Thompson, GOTO 2018. Aeron Cluster's Raft consensus explained by its author — maps straight onto this module's commit path and recovery. Free.
Talk · Aeron: The Next Generation in High-performance Messaging — Martin Thompson, QCon London 2015. The design story behind Aeron Transport: reliable, lock-free UDP and the data structures that make it fast. Free.

Gate log (mentor fills on pass)¶

Date passed:
Passed by: / checked by:
Weak spots noticed (feeds system improvement):