Web fundamentals, an interactive architecture walkthrough
Distributed lock service: how the architecture works
How a distributed lock service grants locks through a consensus log, frees them with expiring leases, and uses fencing tokens to stop stale holders.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A distributed lock service lets processes on different machines agree that only one of them is doing something at a time, such as running a job or acting as leader. This design runs a small cluster that replicates every write through a consensus log, so a lock granted by a majority of members survives the loss of a minority.
Each client holds a session lease that it renews with keepalives; lock keys are attached to that lease, so a crashed client's locks are released when the lease expires. Waiters line up by key creation revision and watch only the key ahead of them. Because a paused process can outlive its lease, the lock revision is also used as a fencing token that the protected resource checks on every write. This is a simplified learning design, not any specific product's implementation.
Trade-offs and what changes at scale
Consensus makes locks safe across failures but costs a majority round trip per acquire and does not scale writes by adding members, so lock services are kept small and used for coordination, not data. Lease length trades fast failover against false expiry during pauses or network blips. A single Redis key with an expiry is much cheaper and fine for avoiding duplicate work, but it can grant a lock twice during failover, so correctness-critical work needs fencing either way. At larger scale contention, not size, is the limit: hot locks are split by resource or replaced with partitioned ownership.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
Before locking, the Lock Client Library opens a session with a 10-second lease, and the Session Keepalive renews it every few seconds. Every key the client creates is tied to this lease, so the client's death eventually frees its locks.
Worker A and Worker B both ask for the job-x lock, and each Lock Client Library writes its own key under the lock's prefix, attached to its lease. The request goes through the Cluster Endpoint to whichever member is leader.
The Leader Node appends each key write to the Replicated Log and replies only after a majority of members has stored it. A lock granted this way survives the loss of any one member.
Committed writes are applied to the Key-Value State, which gives each key a create revision from one cluster-wide counter. The key with the lowest revision under the prefix holds the lock, here Worker A's.
Worker B does not poll; through the Watch Service it waits for the deletion of the key just ahead of its own. Watching only the predecessor means a release wakes one waiter, not every waiter at once.
Worker A sends its lock revision as a fencing token with every write to the Storage Service. Holding a lease does not by itself prove a client still holds the lock, so the resource has to check.
The Storage DB records the highest token it has seen and rejects writes carrying a lower one. If Worker A pauses past its lease and Worker B takes over, A's late write is refused instead of corrupting data.
If Worker A crashes, its keepalives stop, and the Lease Manager expires the lease and deletes its key. The deletion is a normal committed write, so every member agrees on the moment the lock changed hands.
If the Leader Node fails, Follower 1 and Follower 2 elect a new leader that has every committed entry. Locks and leases are part of the replicated state, so they carry over to the new leader.
The same mechanism gives leader election: each instance of a Leader-elected Service campaigns for one key, and only the holder acts. The others wait on the watch and take over when the holder's lease ends.
A Single Redis Lock with SET NX PX and a random value is simpler and faster, but asynchronous replication means a failover can grant the same lock twice. It suits locks that only avoid duplicate work, not ones that protect correctness.
The cluster snapshots its state to the Snapshot Store for disaster recovery, and Observability tracks lock wait times and leader changes. Frequent leader changes usually point to slow disks or an overloaded network.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (18 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Lock Contenderslayer
- Admin CLI: inspect holders, force release
- Lock Client Library: acquire, release, fencing token
- Session Keepalive: renews the lease every few seconds
- Worker A: wants exclusive access to job X
- Worker B: also wants job X
Cluster Endpoint
- Cluster Endpoint: routes clients to any member
Lock Service Clusterboundary
odd number of members, majority quorum
- Follower 1: votes, replicates the log
- Follower 2: votes, replicates the log
- Key-Value State: lock keys with create revision
- Leader Node: orders every write
- Lease Manager: expires leases, deletes their keys
- Replicated Log: writes committed by a majority
- Watch Service: notifies waiters of deletes
Protected Resourcedomain
- Storage DB: data plus highest token seen (postgresql)
- Storage Service: accepts writes with a fencing token
Other Uses and Alternativesdomain
- Leader-elected Service: one active instance via the same lock
- Single Redis Lock: SET NX PX, efficiency-only locking (redis)
Observability
- Observability: lock wait time, lease expiries, leader changes
Snapshot Store
- Snapshot Store: periodic cluster snapshots (s3)
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Distributed job schedulerHow a distributed job scheduler turns cron schedules into runs, shares work with row locks, and retries jobs safely when workers fail.
- URL shortenerHow a link shortener creates unique short keys, serves redirects from a cache in a few milliseconds, and counts clicks off the redirect path.
- Distributed rate limiterHow an API rate limiter shares token buckets across gateways, answers 429 with a retry hint, fails safely and sheds load when the system is stressed.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.