Building a Distributed Key-Value Store in C++ (Part 2)
- An append-only log is the simplest durable storage primitive - replay it on startup and you get crash recovery for free.
- Flushing after every write is slow but safe; optimise only after correctness is established.
- Using a dedicated temporary log file in tests prevents test runs from corrupting production data.
Phase 1 had a store that vanished the moment you killed the process. Phase 2 fixes that with an append-only log and a test suite so regressions don't slip through.
The append-only log
Every PUT and DEL gets appended to a flat text file before the in-memory map is updated. On startup, load_from_log() replays the file top-to-bottom to rebuild state. Same pattern Redis uses for AOF, and countless write-ahead logs before it.
Writing to the log
void KVStore::append_log(const std::string& op, const std::string& key, const std::string& value) {
if (log_out.is_open()) {
log_out << op << " " << key;
if (op == "PUT") {
log_out << " " << value;
}
log_out << "\n";
log_out.flush();
}
}flush() after every write is conservative, but data loss is worse than latency at this stage.
Replaying on startup
void KVStore::load_from_log(){
std::ifstream in(log_file_path);
std::string line;
while (std::getline(in, line)) {
std::istringstream iss(line);
std::string op, key, value;
iss >> op >> key;
if (op == "PUT") {
iss >> value;
store[key] = value;
} else if (op == "DEL") {
store.erase(key);
}
}
}The constructor calls load_from_log() before accepting any commands, so the in-memory map always reflects the last persisted state.
Values with spaces aren't supported yet - the parser splits on whitespace. A proper quoting scheme or length-prefixed binary format can come later.
Testing with Catch2
I pulled in Catch2 via CMake's FetchContent:
FetchContent_Declare(
Catch2
GIT_REPOSITORY https://github.com/catchorg/Catch2.git
GIT_TAG v3.5.2
)
FetchContent_MakeAvailable(Catch2)
enable_testing()
add_subdirectory(tests)First test
TEST_CASE("Basic KVStore operations") {
KVStore store("test_store.log");
store.put("a", "1");
REQUIRE(store.get("a").value() == "1");
store.put("b", "2");
REQUIRE(store.get("b") == std::make_optional(std::string("2")));
REQUIRE(store.del("a") == true);
REQUIRE_FALSE(store.get("a").has_value());
}Pass a dedicated path like "test_store.log" in every test. Keeps test data separate from production logs, and cleanup is trivial.
A SECTION block for crash recovery can open a second KVStore instance pointing at the same file and assert state is restored - no mocking needed.
What's next
Part 3 puts the store on the network. A KVServer will accept TCP connections and handle PUT, GET, and DEL from any terminal.
...starting point - it covers the in-memory store and the overall roadmap. Parts [2](/notes/building-distributed-kv-store-pt2), [3](/notes/building-distributed-kv-store-pt3), and [4](/notes/building-distrib...
...e log is replayed on startup for crash recovery." num={1} /> ## What's next [Part 2](/notes/building-distributed-kv-store-pt2) adds unit tests and an append-only log for persistence. Without that, the store...
...I/O. </KeyTakeaways> Parts [1](/notes/building-distributed-kv-store-pt1) and [2](/notes/building-distributed-kv-store-pt2) gave us a persistent, tested in-process store. Part 3 wraps it in a TCP server...
Data is my veggies - healthy, versatile, and sometimes hard to digest, but in the end, it always brings value.