Skip to content

Cluster & Raft

Scope: the module overview already introduced the commit path at a high level — a leader appends, replicates, and a quorum acks before the message is processed. This page goes one layer deeper on how the cluster agrees on that path — and why its shape rules out split-brain.

TL;DR — in 30 seconds:

  • The Consensus Module (Raft: election, replication, quorum) is a separate concern from your Clustered Service — your business logic never negotiates consensus itself.
  • Only the leader accepts messages (single-writer, applied across a cluster); a follower becomes leader by winning a majority vote for a new term.
  • Majority quorum rules out split-brain: any two majorities of the same fixed node set must overlap, so two leaders can never both commit conflicting state in the same term.

1. The Consensus Module: a separate concern from your business logic

Just as Transport splits into your application and a separate Media Driver process, Aeron Cluster splits into two distinct concerns: the Consensus Module, which runs the Raft protocol — leader election, log replication, quorum tracking — and the Clustered Service, which is your business logic. Your service code never negotiates consensus itself; it only reacts to messages the Consensus Module has already decided are committed.

This separation matters for the same reason the Media Driver split does: it keeps the hard, easy-to-get-wrong part (agreement across unreliable, potentially-failed nodes) in one well-tested component, so the part you author stays simple and — per the Determinism page — replayable.

flowchart TB
  subgraph N1["Node — leader"]
    CM1["Consensus Module<br/>(Raft: log, election, quorum)"]
    CS1["Clustered Service<br/>(your business logic)"]
    CM1 -->|"delivers committed messages"| CS1
  end
  subgraph N2["Node — follower"]
    CM2["Consensus Module"]
    CS2["Clustered Service"]
    CM2 --> CS2
  end
  subgraph N3["Node — follower"]
    CM3["Consensus Module"]
    CS3["Clustered Service"]
    CM3 --> CS3
  end
  CM1 <-->|"replicate log"| CM2
  CM1 <-->|"replicate log"| CM3

2. Leader and followers: one writer, everyone else replicates

At any moment, a cluster has exactly one leader and the rest are followers. Only the leader accepts client messages — this is the same single-writer shape the Mechanical Sympathy module's Concurrency page argues for inside a process, now applied across a whole cluster: one place decides the order of events, so nobody has to reconcile conflicting orderings after the fact.

Raft organizes time into terms: a term has at most one leader. A follower that stops hearing from the leader (its heartbeat times out) starts an election — it votes for itself and asks the other nodes for their vote. A node becomes leader for the new term once it collects votes from a majority of the cluster, not just one other node. Requiring a majority is what guarantees at most one leader per term: two different nodes cannot both win a majority of the same set of votes.

3. Quorum commit: why "acked by a leader" isn't enough

The leader appends each accepted message to its own log, then replicates it to the followers. A message becomes committed only once a quorum — a majority of the cluster, including the leader — has the entry durably in its own log. Only committed entries are ever handed to the Clustered Service to process.

sequenceDiagram
  participant C as Client
  participant L as Leader
  participant F1 as Follower
  participant F2 as Follower
  C->>L: message
  L->>L: append to log
  L->>F1: replicate
  L->>F2: replicate
  F1-->>L: ack
  Note over L: quorum reached (leader + F1)
  L->>L: mark committed
  L->>C: process + respond
  F2-->>L: ack (arrives late — no effect on commit)

Waiting for a majority rather than for every node is what keeps the cluster available when a minority of nodes are slow or unreachable — the leader doesn't stall waiting on a node that may never answer.

Cluster size Tolerates losing Quorum needed
3 nodes 1 node 2
5 nodes 2 nodes 3

Clusters are sized with an odd number of nodes for exactly this reason: every additional node should buy more fault tolerance, and only odd sizes do. A 4-node cluster tolerates losing exactly 1 node — no more than a 3-node cluster — so the fourth node buys nothing; a 5-node cluster is where tolerance actually rises, to 2.

4. Why majority quorum prevents split-brain

Any two majorities of the same fixed set of nodes must share at least one node — that's simple counting, not a design choice. So if two different leaders, in two different terms, each won a majority, those two majorities overlap. The shared node cannot have voted for both leaders, or acked both leaders' conflicting log entries at the same log position, because it only ever acts within one term at a time. This is the property that rules out split-brain: two nodes each believing themselves the leader and independently committing conflicting state.

It also explains a detail from the module overview: a message the leader appended but that never reached a quorum before a crash is safe to lose. No follower had it committed, so no Clustered Service ever processed it — the client simply retries against whichever node becomes the new leader.

flowchart LR
  subgraph A["Majority A — term 1, leader elected here"]
    A1["Node 1"]
    A2["Node 2"]
    A3["Node 3"]
  end
  subgraph B["Majority B — term 2, leader elected here"]
    B3["Node 3"]
    B4["Node 4"]
    B5["Node 5"]
  end
  A3 -.->|"same physical node — can't vote for both"| B3
  style A3 fill:#f96,stroke:#333,stroke-width:2px
  style B3 fill:#f96,stroke:#333,stroke-width:2px

Node 3 sits in both majorities — over a fixed 5-node cluster, any two majorities (any two sets of 3 or more) are guaranteed to share at least one node. That shared node can only act in one term at a time, so it can't have voted for two different leaders, which is exactly why split-brain can't happen.

5. How this connects

  • This builds directly on Transport. The replication traffic between the leader and followers is ordinary Aeron Publications and Subscriptions — the reliable, ordered delivery the Transport page describes is what the Consensus Module relies on to move log entries around; Raft adds the agreement on top of that reliable pipe.
  • Recovery picks up from here. What a follower does after being unreachable for a while — catching up the log, or restoring from a snapshot if it fell too far behind — is the Archiver & Recovery page.
  • The single-writer shape recurs. Exactly one leader accepting messages per term is the same single-writer reasoning the Mechanical Sympathy module's Concurrency page uses to avoid contention inside a process, now solving the same problem across a cluster of machines instead of across threads.

Common gotchas

  • Assuming any node can accept client messages. Fix: only the leader does — followers only replicate; this single-writer shape is what avoids reconciling conflicting orderings after the fact.
  • Assuming a 4-node cluster tolerates more failures than a 3-node one. Fix: it doesn't — a 4-node cluster still tolerates only 1 node down (quorum of 3), same as a 3-node cluster; only odd sizes buy more tolerance per added node.
  • Assuming "acked by the leader" means committed. Fix: a message is committed only once a quorum — a majority including the leader — durably has it in its log; the leader's own append alone isn't enough.
  • Worrying that two leaders could each win a majority vote in the same term. Fix: that's structurally impossible — any two majorities of the same fixed node set must share at least one node, and that node can't vote for two leaders in the same term.
Check yourself — why is the Consensus Module kept as a separate concern from the Clustered Service?

It keeps the hard, easy-to-get-wrong part — agreement across unreliable, potentially-failed nodes — in one well-tested component, so your business-logic code stays simple and (per the Determinism page) replayable.

Check yourself — a 3-node cluster loses one node. Can it still commit messages? What about a 5-node cluster losing two?

Yes to both — a 3-node cluster needs a quorum of 2, achievable with 1 node down; a 5-node cluster needs a quorum of 3, achievable with 2 nodes down.

Check yourself — why does the module size clusters with an odd number of nodes?

Because every additional node should buy more fault tolerance, and only odd sizes do — a 4-node cluster tolerates the same 1-node loss as a 3-node cluster, so the fourth node buys nothing.

Check yourself — using the overlapping-majority argument, why can't two nodes in two different terms both become leader and commit conflicting entries?

Any two majorities of the same fixed node set must share at least one node. That shared node can only act within one term at a time, so it cannot have supported both leaders' conflicting entries at the same log position — ruling out split-brain.

You can defend this when you can say why the Consensus Module is kept separate from the Clustered Service; explain what a Raft term is and why a leader election requires a majority of votes, not just one; say why a message is committed only after a quorum acks it rather than after the leader alone accepts it; and explain, using the overlapping-majority argument, why majority quorum makes split-brain impossible.