Storage & collaboration, an interactive architecture walkthrough
File sync and storage (Dropbox-style): how the architecture works
How a file sync service chunks and hashes files, uploads only missing blocks, orders changes in a journal and keeps every device in step.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A file sync service keeps a folder identical on several devices and in the cloud. The client splits each file into blocks, hashes them, and sends a commit that lists the hashes. The server asks for only the blocks it does not already hold, then records the new version in an ordered journal. Metadata (paths, versions, permissions) and file content (blocks addressed by hash) live in separate systems. Other devices are nudged by a notification, read the journal from their last cursor, and download only the blocks they lack.
Trade-offs and what changes at scale
Block hashing gives deduplication and cheap partial updates but needs a block index that grows with every unique block, and unreferenced blocks need garbage collection (not drawn). Fixed-size blocks are simple but handle inserted bytes poorly; content-defined chunking fixes that at the cost of client CPU. Keeping the journal strongly ordered makes conflicts detectable but limits the write rate of a single very busy shared folder. Replication is quick for new data and erasure coding is cheaper for old data, so storage tiers by age. Cross-account deduplication saves storage but can leak whether content exists. Streaming sync, prefetching blocks before the commit finishes, shortens large uploads for other devices. This is a simplified teaching model based on public information, not Dropbox's actual architecture.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
The client notices a changed file through operating-system file events. It splits the file into fixed-size blocks (public Dropbox writing describes blocks of up to 4 MiB) and hashes each one. Editing the middle of a large file changes only the blocks it touches, so only those are sent again. Inserting bytes shifts later block boundaries, which is where content-defined chunking does better.
The client sends a commit to the Metadata Service: a path in a namespace plus the file's list of block hashes. The service checks access and which hashes it already knows, then answers with the blocks it still needs. A new file usually needs all of them.
The client uploads only the missing blocks to the Block Service, each with its hash. The service recomputes the hash and rejects a mismatch, which catches corruption in transit and keeps clients from naming blocks they did not send.
Verified blocks are written to the Block Store, replicated across machines, and the Block Index records which location each hash maps to. Blocks are immutable and named by content, so a retried upload of the same block is harmless.
The client retries the commit. Now every block is known, so the Metadata Service appends a row to the File Journal and the new version becomes visible. The journal is an ordered log, which gives every device a single history to follow.
After the journal write, the Metadata Service tells the Notification Service that the namespace changed. The signal carries no file data, just a nudge that there is something to fetch.
The notification reaches the account's other devices (a long-lived connection or poll that the device holds open). A device that is offline misses it and catches up from the journal when it reconnects, so notifications are an optimisation rather than the source of truth.
The other device asks for journal rows after the last cursor it has seen. The cursor is how it gets exactly the changes it missed, in order, however long it was away.
The device compares the new blocklist with blocks it already has locally and downloads only the missing ones. The Block Service looks up each hash in the Block Index and reads the block from the Block Store.
When a different user uploads a file that shares blocks with stored ones, the commit finds the hashes already known and nothing needs uploading. This deduplication saves bandwidth and storage, but matching across accounts can reveal that a particular file exists, so some designs restrict it.
If two devices edit the same file offline, the second commit names a version that is no longer the latest. A common approach is to reject it and keep both: the later one is saved as a separate conflicted copy for the user to merge, rather than silently overwriting.
In the background the Storage Compactor groups blocks into larger units and, once data is older, erasure codes it, which needs less disk than keeping several full copies. It updates the Block Index with the new locations. This is not on the upload or download path.
Observability tracks commit latency, block verification failures, notification delay and how far devices lag behind the journal. Sync lag is what users experience as 'my file has not arrived'.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (11 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Clientslayer
- Other Device: second device of the same account
- Sync Client: watches folder, chunks, hashes, uploads
Block Planedomain
- Block Index: block hash to storage location
- Block Service: put and get blocks by hash, verifies hashes
- Block Store: compressed, encrypted blocks; replicated then erasure coded
- Storage Compactor: groups blocks into buckets, erasure codes older data
Metadata Planedomain
- File Journal: append-only change log per namespace
- Metadata Service: namespaces, paths, commits, permissions
- Notification Service: tells devices that something changed
Edgelayer
- API Gateway: TLS, auth, rate limiting
Observability
- Observability: commit latency, block errors, sync lag
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Real-time collaborative document editingHow collaborative editors keep several people's edits in sync: ordering operations, OT versus CRDT, snapshots and live cursors.
- Video sharing platform (YouTube-style)How a video platform takes resumable uploads, encodes segments in parallel into several renditions, and serves them via a CDN with adaptive bitrate.
- Web crawlerHow a large-scale web crawler manages a URL frontier, stays polite per host, filters duplicate URLs and pages, and decides when to revisit.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.