Building a Distributed Key-Value Store in C++ (Part 1)

🚀 Why Build a Distributed Key-Value Store?

#

There are few better ways to get hands-on with distributed systems and C++ than by building a distributed key-value store (KV store) from scratch. This journey is both a learning experience and an exploration of fundamental ideas that power modern distributed databases like Redis, Etcd, and Consul.

In this blog series, I’ll attempt to walk through each step of designing and implementing a distributed KV store, from single-node basics to a replicated, fault-tolerant system.

🧠 The Big Idea

#

A Key-Value Store is a simple database that stores data as a collection of key-value pairs. Think of it as a glorified dictionary, but with persistence, concurrency control, replication, and more.

Eventually, I want this to support:

• Multi-node communication
• Replication and eventual consistency
• Leader election
• Crash recovery
• Client-server networking

But we’re going to get there step by step.

Architecture Diagram

#

The Architecture looks as follows

🔧 Step 1: Start Simple

#

The journey begins with a very basic Hello World in-memory KV store. Here’s what I’ve implemented so far:

File Structure

#

distribkvstore/
├── CMakeLists.txt
├── src/
│   ├── kvstore.cpp
│   ├── kvstore.hpp
│   └── main.cpp
└── tests/
    └── test_kvstore.cpp  (planned)

🧪 Basic Operations

#

We currently support 3 basic operations:

• put(key, value) - Insert or update a value
• get(key) - Retrieve a value
• del(key) - Delete a key

Example Use (main.cpp)

#

int main() {
  KVStore store;
  store.put("hello", "world");

  if (auto val = store.get("hello")) {
    std::cout << "GET hello: " << *val << "\n";
  }

  if (store.del("hello")) {
    std::cout << "deleted hello\n";
  }

  if (!store.get("hello")) {
    std::cout << "hello not found after deletion\n";
  }

  return 0;
}

The code compiles cleanly using CMake, and serves as the foundation for what’s to come.

🗺️ Roadmap Ahead

#

1. Phase 1: Local Store

   ✅ Done

   Basic In-Memory KV Store

   • Implemented a KVStore class
   • Supports put, get, and del operations
   • Command-line usage for demoing
2. Phase 2: Persistence

   Next

   Durability with Append-Only Log

   • Store data on disk for crash recovery
   • Simple log file to persist updates
   • Future support for replay and compaction
3. Phase 3: Networking

   Planned

   Client-Server Communication

   • Expose KVStore via TCP sockets
   • Implement a lightweight protocol
   • Build a CLI
4. Phase 4: Multi-Node Architecture

   Planned

   Cluster Mode

   • Run multiple instances
   • Replication between nodes
   • Setup eventual consistency or synchronous replicas
5. Phase 5: Consensus

   Planned

   Leader Election and Coordination

   • Implement Raft or Gossip protocol
   • Handle node crashes and partitions
   • Ensure quorum and write safety
6. Phase 6: Testing & Resilience

   Final

   Hardening the System

   • Add fuzz and chaos testing
   • Graceful failure handling and optimizations

📌 Next Steps

#

In the next post, I’ll add unit tests and discuss how I plan to integrate persistence using a simple append-only log file. This will pave the way toward recoverability, one of the key challenges in distributed design.

Let’s see how far I’ll spiral down into insanity 🥲