Web fundamentals, an interactive architecture walkthrough
Distributed key-value store: how the architecture works
How a Dynamo-style key-value store places keys on a hash ring, replicates them with tunable quorums, and repairs replicas that fall behind.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A distributed key-value store spreads keys across many machines and keeps several copies of each one, so it keeps working while individual machines fail. This design follows the Dynamo approach: keys are placed on a consistent-hash ring with virtual nodes, every key has three replicas on different racks, and any node can coordinate a request. Each request chooses how many replicas must answer, which trades consistency against latency.
On each node, writes are appended to a commit log, applied to a sorted in-memory table, and later flushed as immutable sorted files that background compaction merges. Membership and failure detection use gossip instead of a master, and replicas that fall behind are fixed by hinted handoff, read repair and scheduled Merkle-tree repair. This is a simplified learning design, not any database's complete implementation.
Trade-offs and what changes at scale
Quorum reads and writes give read-your-writes behaviour while tolerating one replica down, but they cost two round trips per request; reading or writing at ONE is faster and may return stale data. Last-write-wins needs synchronised clocks and silently drops one of two concurrent writes, so systems that cannot accept that use version vectors or conditional writes instead. The log-structured engine makes writes cheap but adds compaction I/O and read amplification. At larger scale the operational work is repair scheduling, keeping partitions small, and spreading replicas across racks and regions.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
The Application Service calls put or get through the Client Driver, which knows the token ring and sends the request to a node that holds the key. Picking a replica directly saves one network hop.
The Coordinator Node asks the Partitioner & Token Ring to hash the key onto the ring and walk it to find three replicas on different racks. Consistent hashing with many virtual nodes per machine means adding a machine moves only a small share of keys.
A write goes to all three replicas, and the coordinator replies once a quorum of two has acknowledged. On each replica, the write is appended to the Commit Log first so it survives a crash.
The write is then applied to the Memtable, a sorted structure in memory, and acknowledged. Writes never update files in place, which keeps them fast and sequential.
When the Memtable fills, it is flushed to disk as a new immutable file among the SSTables, and the matching commit log segments can be discarded. A key can now exist in several SSTables at once, each with its own timestamped version.
A read at quorum asks two replicas; on each, the Bloom Filters rule out SSTables that cannot contain the key, so only a few files are opened. The coordinator returns the version with the newest timestamp (last write wins).
If the replicas disagree, Read Repair writes the newer value back to the stale one. Because writes and reads both use a quorum (2 + 2 > 3), every read overlaps at least one replica that has the latest acknowledged write.
Nodes exchange state through Gossip about once a second, and each node's Failure Detector judges peers up or down from the gaps between heartbeats. No central master decides membership, so there is no single point of failure.
If Replica C is down, the coordinator keeps the write in the Hint Store and replays it when C comes back. Hints shorten the time C is behind but are best effort, so they are not relied on alone.
Scheduled Anti-entropy Repair builds Merkle trees over key ranges on each replica and streams only the ranges whose hashes differ. This catches anything hints and read repair missed.
Compaction merges SSTables in the background, keeping the newest version of each key and dropping deleted data once it is safe. Without it, reads would open more and more files over time.
A new node takes ownership of some tokens, announced through gossip, and Bootstrap Streaming copies those ranges from existing replicas before it serves reads. Capacity grows without a full reshuffle of the data.
Because SSTables never change once written, snapshots are cheap and are copied to the Backup Store. The Change Stream exposes writes from the commit log to other systems, such as search or analytics.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (19 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Clientslayer
- Application Service: reads and writes keys
- Client Driver: token-aware routing, retries
Request Routingdomain
- Coordinator Node: any node; fans out to replicas
- Partitioner & Token Ring: hash key to token, vnodes to nodes
Other Replicaslayer
- Replica B: second copy, another rack
- Replica C: third copy, another rack
Replica A: Storage Engineboundary
one of the key's replicas
- Bloom Filters: skip SSTables without the key
- Commit Log: append-only, replayed after a crash
- Compaction: merges SSTables, drops old versions
- Memtable: sorted in-memory writes
- SSTables: immutable sorted files on disk
Repairdomain
- Anti-entropy Repair: Merkle tree comparison
- Hint Store: writes held for a down replica
- Read Repair: fixes stale replicas seen on reads
Cluster Membershipdomain
- Failure Detector: phi accrual, per node
- Gossip: membership, token ownership, schema
Operationslayer
- Backup Store: SSTable snapshots (s3)
- Bootstrap Streaming: moves ranges to new nodes
- Change Stream: change data capture (kafka)
- Observability: latency, pending compactions, hints
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Distributed cacheHow a distributed cache places keys with consistent hashing, replicates shards, evicts entries, invalidates on writes and resists stampedes.
- Load balancerHow a two-tier load balancer spreads traffic with ECMP and consistent hashing at L4, then balances each request at L7 using health checks.
- Message queueHow a partitioned, replicated log stores messages durably, keeps them ordered per key, and lets consumer groups read at their own pace.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.