A replicated, strongly consistent key-value store. The Raft consensus protocol, implemented from scratch in Go — no consensus libraries.
Leader election · log replication · crash-safe persistence · snapshotting · linearizable reads · exactly-once client sessions.
RaftKV implements Raft directly from Ongaro & Ousterhout’s
Figure 2 — no hashicorp/raft, no
etcd/raft. Leader election with randomized timeouts, log
replication with fast conflict backup, a bbolt write-ahead log that fsyncs
on the commit path, snapshot-based log compaction with
InstallSnapshot catch-up, ReadIndex linearizable reads, and
exactly-once client sessions that survive compaction. Every deviation a real
system forces is deliberate and documented.
The design hinges on one seam: the Raft core depends on exactly
two interfaces, Transport and Persister.
Swap in the in-memory implementations and the same core runs under a
deterministic, seedable adversarial network — dropped, delayed,
reordered, partitioned messages that reproduce exactly. Swap in gRPC and
bbolt and the identical core runs a real cluster —
TestGRPCReplication passes the same election and replication
checks over real gRPC, and the five-node docker-compose deployment runs the
same binary. The tests and the deployment exercise one implementation, not
two.
Log indices are 1-based; the entry at index 0 is the snapshot boundary sentinel.
The Raft paper’s safety properties are not assumed — each one is asserted by a named oracle, test, or chaos script. All tests run under the Go race detector in CI.
| Property | Guarantee | Verified by |
|---|---|---|
| Election Safety | At most one leader per term. | checkOneLeader oracle, asserted across the election
suite including TestReElection and
TestElectionUnreliable (~10% message loss, delays). |
| Log Matching | Nodes agree on committed entries; no gaps in the log. | Gap and agreement checks in the replication suite
(TestLeaderChangeKeepsCommitted,
TestDeposedLeaderEntriesOverwritten). |
| State Machine Safety | No two nodes apply different commands at the same log index. | Per-index oracle in every harness apply drain, plus
checkStoresAgree across command and snapshot applies. |
| Leader Completeness | Committed entries survive leader changes and partitions. | chaos/partition.sh: a unique per-run committed value
survives partition and heal on the live 5-node cluster. |
| Linearizability | Every operation appears atomic between call and return. | TestLinearizability: Porcupine per-key register
model, 210 concurrent ops across 3 keys, 3 runs under
-race. |
| Exactly-once | A retried write applies once, even across leader changes. | TestExactlyOnceRetry, TestZeroSeqDedup;
the session table is snapshotted, so dedup survives compaction. |
Two Docker-native chaos scripts run against the real 5-node docker-compose cluster — not a simulation. Both pass, with assertions on data, not just liveness.
Kills the leader container. Asserts the cluster re-elects and a follow-up write commits through the new leader, then restarts the killed node.
Disconnects the leader from the cluster network. Asserts the majority elects a new leader and stays writable, the isolated minority refuses reads (no quorum, so no stale read), and a unique committed value survives the heal.
Found honestly: after many back-to-back chaos cycles on Docker-for-Windows, accumulated churn plus host latency spikes pushed heartbeats past the 150–300 ms election timeout and the cluster fell into a split-vote election storm — a fresh cluster is stable at term 1 every time. The lesson: on latency-spiky hosts the election timeout is marginal; scaling it up (or adding Pre-Vote) is the lever for high-latency deployments.
| Metric | Value |
|---|---|
| Write throughput | 486 writes/s |
| Failed requests | 0 |
| Latency p50 | 32 ms |
| Latency p99 | 53 ms |
| Latency p99.9 | 71 ms |
Measured with
cmd/loadtest: 5-node docker-compose cluster on a Windows dev
box, 16 concurrent clients, 5 s of sustained PUTs. Every write pays an
HTTP round-trip, a Raft replication round, and a bbolt fsync on
the commit path — this build is tuned for durability, not
latency.