Archiver & Recovery

Scope: the module overview already sketched recovery at a high level — a snapshot plus the log entries after it rebuild a crashed node. This page goes one layer deeper on how the Archiver persists that log and why snapshots are what keep recovery cheap.

TL;DR — in 30 seconds:

• The Archiver is a separate component that continuously persists the in-memory Raft log to disk, so the cluster's history survives past any single node's process lifetime.
• A snapshot is a checkpoint of state at one log position — it turns recovery into "load the snapshot, then replay only what happened after it," instead of replaying the entire log.
• Archiving and snapshotting happen alongside normal operation, not on the commit path a client is waiting on — they make recovery fast without slowing down everyday processing.

1. The Archiver: durable storage for the log, not just replication

Quorum commit (the Cluster & Raft page) guarantees a committed entry survives on a majority of nodes' in-memory logs — but a log that only lives in memory disappears if every node in that majority restarts around the same time. The Archiver is a separate component that continuously persists the log to durable storage (disk), so the cluster's history survives past any single node's process lifetime.

This is the same separation-of-concerns shape as Transport's Media Driver and Cluster's Consensus Module: a dedicated component owns one hard problem — durability — so the Clustered Service never has to think about how its log is stored, only that it will be there on restart.

flowchart LR
  CM["Consensus Module<br/>(in-memory log)"] -->|continuously persists| ARC["Archiver<br/>(durable log on disk)"]
  ARC -->|replay on restart| CS["Clustered Service"]

2. Snapshots: a checkpoint, not a full history

A snapshot is a serialized copy of the Clustered Service's state at one point in the log — everything the service needs to resume from that exact position, without replaying anything before it. The Consensus Module records which log position each snapshot corresponds to.

Without snapshots, recovering a node would mean replaying the entire log from the very first committed entry — for a long-running cluster, that could be hours of replay before the node is useful again. A snapshot turns recovery into: load the snapshot, then replay only what happened after it.

Recovery input	What it restores	Cost
Snapshot only	State as of the snapshot's log position	Fast — one deserialize
Log entries after the snapshot	Everything since that checkpoint	Proportional to time since last snapshot
Full log (no snapshot)	Entire cluster history	Proportional to cluster lifetime — avoided in practice

3. Replay: rebuilding a node step by step

Recovery composes the two pieces in order:

flowchart TB
  START["Node starts (after crash or restart)"] --> LOAD["Load latest snapshot"]
  LOAD --> STATE["Service state = snapshot's state"]
  STATE --> FETCH["Fetch log entries after the snapshot's position from the Archiver"]
  FETCH --> REPLAY["Replay each entry through the Clustered Service, in order"]
  REPLAY --> DONE["State reconstructed — node rejoins the cluster"]

Replay only works because the Clustered Service is deterministic (the Determinism page): replaying the same ordered entries against the same starting state must land on the same result every time, on every node. If the service read the clock or called out to the network during normal processing, replaying that same entry later would not reproduce the original state — recovery would silently diverge.

4. Why this doesn't slow down the normal commit path

Archiving and snapshotting happen alongside normal operation, not on the critical path a client is waiting on. A message is committed as soon as a quorum of in-memory logs has it (the Cluster & Raft page); persisting it to the Archiver's disk, and periodically folding the log into a fresh snapshot, are background concerns that make recovery fast without slowing down everyday processing.

5. How this connects

• This builds directly on Cluster & Raft. The log the Archiver persists is the same log the Consensus Module replicates for quorum commit — Archiving durability and Raft agreement are two different guarantees layered on the same underlying log.
• Determinism is what makes replay trustworthy. The Determinism page's contract — same starting state
• same ordered log + same code ⇒ identical state — is the exact property replay depends on; break determinism anywhere and recovery can silently rebuild the wrong state.
• The snapshot/replay split is a mechanical-sympathy trade-off. Paying a small, regular cost (taking a snapshot) to avoid a much larger one (replaying an entire history) is the same amortized-cost reasoning the Mechanical Sympathy module applies elsewhere.

Common gotchas

• Assuming quorum commit alone makes the log durable. Fix: quorum commit only guarantees a majority's in-memory logs have the entry — the Archiver is what persists it to disk so it survives a restart.
• Assuming recovery replays the entire log from the beginning. Fix: it loads the latest snapshot first, then replays only the entries after it — full-log replay is exactly what snapshots exist to avoid.
• Assuming archiving/snapshotting slows down normal message processing. Fix: they run alongside normal operation, not on the critical commit path — a message commits as soon as a quorum of in-memory logs has it.
• Trusting replay to reconstruct correct state regardless of the handler's code. Fix: replay only reproduces the original result if the Clustered Service is deterministic — a non-deterministic handler makes recovery silently diverge, with no error.

Check yourself — quorum commit already guarantees a majority of nodes have an entry in their log. Why do you still need the Archiver?

Quorum commit only guarantees the entry lives in-memory on a majority of nodes — if that majority restarts around the same time, an in-memory-only log disappears. The Archiver persists the log to disk so it survives past any single node's process lifetime.

Check yourself — what does a snapshot restore, and what do the log entries after it restore?

The snapshot restores state as of one specific log position (a checkpoint); the log entries after it restore everything that happened since — replaying just those, instead of the full history, is what keeps recovery fast.

Check yourself — in what order does a restarting node rebuild its state?

Load the latest snapshot first (state = snapshot's state), then fetch and replay, in order, the log entries after the snapshot's position from the Archiver.

Check yourself — a Clustered Service handler reads the wall clock mid-processing. Why does that put recovery at risk, specifically?

Replay only reconstructs correct state if replaying the same ordered entries against the same starting state always produces the same result. A clock read produces a different value on replay than it did originally, so the replayed state can silently diverge from what the original run produced.

You can defend this when you can say why the Archiver is a separate component from the in-memory Raft log; explain what a snapshot restores versus what the log entries after it restore, and why that split keeps recovery fast; walk through the load-snapshot → replay-log order a restarting node follows; and explain why replay only reconstructs correct state if the Clustered Service is deterministic.