A production-grade, fault-tolerant distributed key-value store built in Go, implementing the Raft consensus algorithm for consistency, Write-Ahead Logging (WAL) for durability, log compaction, snapshot-based recovery, and gRPC for inter-node communication.
Inspired by industry-standard consensus-replicated systems like etcd and Consul, this system is designed for high reliability, strong consistency (linearizability), and sub-10ms latency.
- Consensus-Driven Replication: Automated leader election, term management, and heartbeat-driven node discovery utilizing the Raft consensus algorithm.
- Strong Consistency (Linearizability): Linearizable read/write semantics via log commitment quorum and leader lease verification.
- Durability & Recovery: Write-Ahead Logging (WAL) utilizing CRC32 checksums, record length prefixes, and synchronous disk flushing (
fsync) to prevent any data loss across node crashes. - Log Compaction & Snapshotting: Periodic compaction of the WAL/Raft log by serializing current state machine maps to snapshot files. Automatic snapshot streaming via
InstallSnapshotRPC to sync lagging peers. - Transparent Client Redirection: Dynamic KV service client CLI automatically reconnecting and redirecting requests to the elected leader node.
- Observability: Built-in Prometheus metrics exporter publishing metrics on terms, node states, commit/applied indexes, client proposals, and RPC latencies.
- Chaos Testing Harness: Simulated network partition (split-brain testing), node crash reelection, and packet drop chaos tests included in the test suite.
graph TD
Client[Client / gRPC Gateway] -->|PUT/GET/DELETE| Leader[Raft Leader]
Leader -->|1. Append WAL| LocalWAL[(Local WAL)]
Leader -->|2. Replicate Log| Follower1[Follower Node 1]
Leader -->|2. Replicate Log| Follower2[Follower Node 2]
Follower1 -->|Append WAL| F1WAL[(Follower 1 WAL)]
Follower2 -->|Append WAL| F2WAL[(Follower 2 WAL)]
Leader -->|3. Commit & Apply| StateMachine[In-Memory KV Engine]
StateMachine -->|Periodic Snapshot| SnapshotManager[Snapshot Manager]
Inter-node RPC layer definitions:
RequestVote: Invoked by candidates during elections to gather votes.AppendEntries: Invoked by the leader to replicate log entries and send heartbeats.InstallSnapshot: Invoked by the leader to stream snapshot state to lagging followers.
├── cmd/
│ ├── server/ # Server entrypoint initializing Raft node & gRPC services
│ └── client/ # CLI client wrapper to interact with the KV cluster
├── internal/
│ ├── storage/ # KV storage engine, WAL, and file persistence
│ ├── raft/ # Raft consensus state machine & KV service handlers
│ ├── grpc/ # Generated Protobuf Go code
│ ├── snapshot/ # Snapshot serialization and compaction
│ └── metrics/ # Prometheus observability metrics exporter
├── proto/ # Protocol Buffers definition files (.proto)
├── deploy/ # Deployment scripts (Docker Compose, Dockerfile)
├── benchmarks/ # Concurrency benchmarking suite
├── run_local_demo.ps1 # Automated localhost cluster execution script
├── go.mod # Go module definition
└── README.md # User manual
- Go 1.22+ installed.
- Protobuf compiler (
protoc) if compiling proto files from scratch.
The demo script automatically initializes a 3-node cluster on localhost, performs a key-value write/read, executes a concurrent load benchmark, and terminates the cluster cleanly:
powershell -ExecutionPolicy Bypass -File .\run_local_demo.ps1To run unit and consensus integration tests (including leader crash reelection, WAL corruption recovery, and partition chaos):
go test -v ./internal/...Start three separate server nodes in different terminals:
- Terminal 1:
.\distributed-kv.exe -id node-1 -addr 127.0.0.1:50051 -peers node-1=127.0.0.1:50051,node-2=127.0.0.1:50052,node-3=127.0.0.1:50053 -metrics-port 2112 - Terminal 2:
.\distributed-kv.exe -id node-2 -addr 127.0.0.1:50052 -peers node-1=127.0.0.1:50051,node-2=127.0.0.1:50052,node-3=127.0.0.1:50053 -metrics-port 2113 - Terminal 3:
.\distributed-kv.exe -id node-3 -addr 127.0.0.1:50053 -peers node-1=127.0.0.1:50051,node-2=127.0.0.1:50052,node-3=127.0.0.1:50053 -metrics-port 2114
Interact with the cluster using the CLI client:
# Write data
.\kv-client.exe -op put -key company -val google
# Read data
.\kv-client.exe -op get -key company
# Delete data
.\kv-client.exe -op delete -key companyEach server node runs a Prometheus metrics exporter. Scrape endpoints:
- Node 1:
http://localhost:2112/metrics - Node 2:
http://localhost:2113/metrics - Node 3:
http://localhost:2114/metrics
Exposed Metrics:
raft_node_role: Role of the node (0=Follower,1=Candidate,2=Leader).raft_current_term: Active consensus term.raft_commit_index: Index of highest committed log entry.raft_last_applied: Index of highest applied log entry.raft_proposals_total: Total number of client write proposals proposed.raft_grpc_latency_seconds: Histogram of inter-node RPC durations.
Deploy a 5-node containerized cluster locally using Docker Compose:
cd deploy
docker-compose up --buildThis maps the following endpoints:
- gRPC Ports:
50051,50052,50053,50054,50055 - Metrics Ports:
2112,2113,2114,2115,2116 - Data Directories: Bound to named volumes (
node1-data,node2-data, etc.) to persist logs.
A concurrent benchmarking script is included to measure QPS and latency:
.\kv-benchmark.exe -addr 127.0.0.1:50051,127.0.0.1:50052,127.0.0.1:50053 -concurrency 50 -ops 2000 -read-ratio 0.5Under concurrent load, in-memory lease-reads serve responses in sub-millisecond latency, while write operations maintain strong safety by flushing WAL records to disk synchronously before replying.