Web fundamentals, an interactive architecture walkthrough
Distributed cache: how the architecture works
How a distributed cache places keys with consistent hashing, replicates shards, evicts entries, invalidates on writes and resists stampedes.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A distributed cache holds a copy of frequently read data in memory across many servers, so most reads never reach the database. Clients place each key on a hash ring so any client can find its server without asking, shards are replicated for failover, and a worker deletes entries when the database changes. The harder parts are not the lookups but the failure cases: a popular key expiring under load, a stale value written after a newer one, and a failed server sending all its traffic to the database.
Trade-offs and what changes at scale
A cache trades freshness for speed, so every design needs an answer to 'how stale can this be?' and a way to bound it with expiry times plus invalidation. Consistent hashing limits data movement when servers change, but hot keys still concentrate on one shard unless they are replicated or split. Asynchronous replication keeps writes fast and can lose the latest writes on failover, which is fine for a cache but not for a primary store. Look-aside caching puts loading logic in every application, while read-through puts it in the cache layer at the cost of coupling. Lease tokens and a gutter pool, as described in Facebook's published memcache paper, protect the database during stampedes and shard failures. Cache capacity is a cost decision: memory is expensive, so a hit-ratio target tells you what size to buy. This is a simplified teaching model based on public information, not any company's actual architecture.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
The application needs a value. It uses a look-aside pattern: ask the cache first through the Cache Client Library, and go to the database only on a miss. The application, not the cache, owns the logic for loading and storing values.
The client library fetches the ring map from the Cluster Manager and caches it. Keys are placed on a hash ring with many virtual nodes per server, so adding or removing a server moves only a fraction of the keys instead of reshuffling everything. Redis Cluster uses a related scheme with a fixed set of 16384 hash slots.
The client hashes the key to find the shard that owns it and sends the read there. A hit returns straight away. Keys that are read far more than others can also be served from replicas, spreading one hot key across several machines.
On a miss the application reads the Primary Database. When a popular key expires, thousands of requests could miss together and hit the database at once (a stampede). To prevent it, the shard hands one client a lease token and tells the others to wait briefly or use slightly stale data, so only one request recomputes the value.
The client stores the loaded value, presenting its lease token. If the key was invalidated after the lease was issued, the token is void and the store is refused, so a slow reader cannot put an old value over a newer one. Expiry times get a small random spread so many keys do not all expire together.
Each primary copies writes to a replica asynchronously, which keeps writes fast but means a failover can lose the latest acknowledged writes (Redis documents the same for its clusters). For a cache this is acceptable, because the database is the source of truth and a lost entry is just a later miss.
On the write path the application updates the Primary Database. Every committed change flows into the Commit Log, so invalidation is driven by what actually committed rather than by what the application hoped to write.
The Invalidation Worker consumes the log and works out which cache keys each change affects. Because it replays from the log, an invalidation that was lost or failed can be retried instead of leaving a stale entry until its expiry.
The worker deletes the keys rather than writing new values into them. Deleting is safer than updating, because racing updates can leave the older value in place, and the next reader reloads the fresh one. Expiry times remain as a backstop for any missed deletion.
When the shard is full, it evicts entries by policy. Least-recently-used suits skewed access, where a small set of keys gets most reads; least-frequently-used copes better when occasional large scans would otherwise flush the hot keys. Redis approximates LRU by sampling a few keys rather than tracking exact order, which saves memory. A cache with no eviction rejects writes once full.
The Cluster Manager health-checks the shards. When a primary stops responding, it promotes the replica and publishes a new ring map. Until clients catch up, requests to the dead shard fail.
During that gap, clients send the failed request to a small Gutter Pool with short expiry times instead of the database. This absorbs most of the miss traffic that would otherwise hit the database, at the cost of briefly stale data. It also avoids rehashing the dead shard's keys onto healthy ones, which could overload them in turn.
Observability tracks hit ratio, eviction rate, per-shard load (to spot hot keys) and invalidation lag. A falling hit ratio after a deployment usually means the key format or expiry changed.
Sequence diagram
The same walkthrough as a UML sequence diagram: components left to right in the order a request reaches them, numbered by step. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Application Tierlayer
- Application Server: look-aside: checks the cache, falls back to the database
- Cache Client Library: hash ring, retries, lease tokens
Cache Clusterdomain
- Cache Shard (primary): owns a slice of the key space, in memory (redis)
- Cache Shard (replica): copy for reads and failover (redis)
- Cluster Manager: membership, ring map, failure detection
- Gutter Pool: short-lived stand-in while a shard is down
Source of Truthdomain
- Commit Log: committed changes, in order
- Invalidation Worker: deletes cache keys after each commit
- Primary Database: authoritative data (postgresql)
Observability
- Observability: hit ratio, evictions, invalidation lag
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Distributed key-value storeHow a Dynamo-style key-value store places keys on a hash ring, replicates them with tunable quorums, and repairs replicas that fall behind.
- URL shortenerHow a link shortener creates unique short keys, serves redirects from a cache in a few milliseconds, and counts clicks off the redirect path.
- Search autocomplete (typeahead)How search autocomplete answers in milliseconds: prefix indexes, caching, offline ranking from query logs, trending queries and policy filtering.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.