The prompt, scoped
"Design Dropbox" is "keep a folder identical across a user's devices, efficiently and without losing data." We keep the spine: upload and store files durably, sync changes across devices moving only what changed, and resolve conflicts. Out of scope, said up front: sharing and permissions, real-time collaborative editing (that is Google Docs), and search. The hard parts are dedup, the sync protocol, and conflicts.
Step 1
Requirements: functional and non-functional
The functional core:
- Upload / download files; store them durably.
- Sync changes across all of a user's devices.
- Move only deltas: a one-byte edit should not re-upload a 1GB file.
- Resolve conflicts when two devices edit the same file offline.
Out of scope (say it): sharing/permissions, collaborative editing, search, versioning UI.
The non-functional shape:
Two facts drive it: never lose a file, and never move bytes you do not have to. The first pushes toward durable, content-addressed storage; the second pushes toward chunking and dedup so sync ships deltas, not whole files.
Maps to S2 (requirements) and S1 (consistency split: metadata strong, file sync eventual).
Step 2
Estimation: the numbers that bound the design
| Quantity | Assumption | Result |
|---|---|---|
| Files | large blobs, many users | storage is dominated by file bytes |
| Edits | usually change a small part of a file | deltas are tiny relative to file size |
| Chunk size | ~4 MB content-defined chunks | a file is a list of chunk hashes |
| Dedup | identical chunks across files/users | store each unique chunk once; large savings |
| Metadata ops | per sync: list versions + chunks | frequent small reads/writes, must be consistent |
The number that decides the architecture
The ratio of edit size to file size. Because an edit usually touches a small fraction of a file, the design is built to detect and ship only the changed chunks. Combined with global dedup (store each unique chunk once), this is what makes sync fast and storage cheap. Chunking is the whole game.
Maps to S2 (capacity estimation).
Step 3
API: chunks and a commit
| Endpoint | Does | Notes |
|---|---|---|
POST /v1/chunks/:hash | upload a chunk | content-addressed; if the hash already exists, it is a no-op (dedup) |
POST /v1/files/commit | commit a file version | a list of chunk hashes + parent version; atomic metadata write |
GET /v1/delta?cursor= | what changed since my cursor | the sync pull: returns changed files and their chunk lists |
GET /v1/chunks/:hash | download a chunk | only the chunks the device is missing |
The shape is "upload the chunks you have that the server lacks, then commit a version that references them by hash." Other devices learn via delta and download only missing chunks.
Maps to S5 (API design): content-addressed, idempotent uploads, a delta cursor. See the API cheat sheet.
Step 4
Data model and store choices
Namespace(id PK, owner) a user's folder tree File(id PK, namespace_id, path) FileVersion(id PK, file_id, parent_version, chunk_hashes[], device_id, ts) Chunk(hash PK) -> blob content-addressed = automatic dedup DeviceCursor[device_id] -> last synced version stores Chunks ........ immutable, content-addressed blobs, huge -> object storage (key = hash) Metadata ...... file tree, versions, cursors; needs ACID -> relational / strongly-consistent Notify ........ "your namespace changed" fan-out -> a lightweight pub/sub
Why content-addressing is the trick
Storing a chunk under the key hash(chunk) gives you dedup for free: two identical chunks, from the same file, different files, or different users, resolve to one key and one stored copy. It also makes uploads idempotent: re-uploading a chunk you already have is a no-op. The metadata (which hashes make which file version) is the only thing that must be transactional.
Maps to S3 (store from access patterns: blobs by hash vs relational metadata) and S4 (object storage scales the bytes). See the data-stores cheat sheet.
Step 5
High-level architecture
A device chunks a changed file, uploads only the chunks the server lacks (via the block service into content-addressed object storage), then commits a new version to the metadata service. The metadata service is the source of truth; on a commit it tells the notification service, which nudges the user's other devices. Those devices pull the delta from metadata and download only their missing chunks.
Maps to S7 (architectural styles) and S6 (idempotent content-addressed writes, pub/sub).
Step 6
Deep dive A: content-defined chunking and dedup
The core technique, and the subtle part is where the chunk boundaries fall. Fixed-size blocks break the moment you insert a byte; content-defined chunking does not.
fixed-size chunks (naive)
insert one byte at the start -> every following block shifts ->
every chunk hash changes -> you re-upload the whole file. bad.
content-defined chunking (CDC)
- slide a rolling hash over the bytes; cut a boundary when the hash
hits a pattern (e.g. low bits all zero)
- boundaries depend on CONTENT, not position
- so an insert only changes the chunk it lands in; the rest keep
the same boundaries and the same hashes
dedup
- store each chunk under key = hash(chunk)
- identical chunks (across files / users) collapse to one copy
- upload protocol: "here are my chunk hashes; which do you NOT have?"
-> upload only the missing ones
Why CDC, not fixed blocks
Fixed blocks make a one-byte insert re-hash the entire file, defeating the whole point of delta sync. A rolling-hash boundary moves with the content, so an edit is local: only the affected chunk changes. Then content-addressing dedups the unchanged chunks globally. This pair is what turns "sync a 1GB file" into "ship a few MB."
Maps to S4 (chunking and dedup) and S1 (bandwidth vs compute: hash to save transfer).
Step 6 · continued
Deep dive B: the sync protocol
How a device learns of a change and catches up, moving the minimum. The metadata service is the source of truth; devices hold a cursor and pull deltas.
each device holds a CURSOR = the last version it has seen on a change (Device A commits) metadata bumps the namespace version, notifies subscribers Device B catches up 1. notified (or it long-polls) -> "you are behind" 2. GET /delta?cursor=X -> list of changed files + their chunk hashes 3. diff against local: which chunk hashes am I missing? 4. download only those chunks; reassemble the files 5. advance cursor missed the notification? no problem: the cursor + delta pull is self-correcting, so a dropped nudge just means catch up on next sync
The senior point
The notification is an optimisation, not the source of truth. Correctness comes from the cursor plus delta: a device can always ask "what changed since X?" and converge, so a lost or duplicated notification is harmless. That separation, a best-effort nudge over a reliable pull, is what makes the sync robust.
Maps to S6 (idempotent catch-up, at-least-once notifications) and S9 (self-correcting sync).
Step 6 · continued
Deep dive C: conflict resolution
Two devices edit the same file while offline, then both sync. You cannot silently pick a winner without losing someone's work, so you detect the fork and keep both.
both devices started from version V (the parent)
Device A commits V -> A (parent = V)
Device B commits V -> B (parent = V) <-- same parent!
detection
the second commit's parent_version is no longer the current head
=> the metadata service sees a fork, not a linear history
resolution (Dropbox's actual choice)
- keep BOTH: accept the first as the new head, and write the
second as "filename (conflicted copy from Device B)"
- never silently overwrite; let the human reconcile
(richer option: 3-way merge for mergeable types; files generally are not)
Why "conflicted copy", not last-write-wins
Last-write-wins silently destroys data, which is unacceptable for a system whose whole promise is "never lose my files." Detecting the fork by parent version and materialising a conflicted copy is the honest move: nothing is lost, and a human resolves the ambiguity a machine cannot. This is the same parent-check idea as optimistic concurrency control.
Maps to S6 (version vectors / optimistic concurrency) and S1 (simplicity vs capability: conflicted copy over auto-merge).
Step 7
Reliability & failure design
| If this fails… | Mode | What happens |
|---|---|---|
| Chunk upload interrupted | resume | content-addressed + idempotent; re-upload only the missing chunks |
| Metadata commit | fail closed | atomic version bump; never a half-committed file version |
| Notification lost | fail open | cursor + delta pull self-corrects on the next sync |
| Concurrent edits | conflicted copy | fork detected by parent version; both kept, nothing lost |
| Object storage node | degrade | replicated chunks; another replica serves the hash |
Durability is the backbone: chunks are content-addressed and replicated, so a re-upload is a no-op and a lost replica is recoverable. The metadata commit is the one strongly-consistent, fail-closed step. Everything else (notifications, downloads) is best-effort and self-correcting via the cursor, so dropped messages cost a little latency, never data.
Maps to S9 (designing for failure): fail closed on the commit, self-correcting sync. See the resilience cheat sheet.
Step 7 · continued
Observability & operability
- SLI / SLO: sync latency (a change on one device becoming visible on another) under target, and upload/download success rate.
- Dedup ratio: stored bytes over logical bytes. It is a cost SLI and a health signal; a sudden drop means chunking or dedup is misbehaving.
- Conflict rate: conflicted-copy creation; a spike suggests a sync bug or a misbehaving client.
- The three pillars: metrics for sync latency and dedup ratio, traces across block and metadata services, structured logs keyed by namespace and version.
- Alert on experience and cost: page on sync-latency budget burn and metadata error rate, not raw CPU.
Maps to S10 (observability). See the observability cheat sheet.
Step 8
Trade-offs along the six core axes
| Axis (S1) | The call here | Why it is least-wrong |
|---|---|---|
| Bandwidth vs compute | chunk + hash to ship deltas | spend CPU hashing to avoid moving whole files |
| Space vs time | content-addressed dedup | store each unique chunk once; cheap storage, fast sync |
| Consistency vs availability | split by layer | metadata strongly consistent; chunk sync eventual and self-correcting |
| Push vs pull | notify then pull | a best-effort nudge over a reliable cursor-based delta pull |
| Simplicity vs capability | conflicted copy, not auto-merge | never lose data; let a human reconcile what a machine cannot |
| Durability vs cost | replicate the chunk store | the product promise is "never lose my files"; pay for replication |
The whole design is one idea applied twice: move and store as little as possible. Content-defined chunks plus content-addressed dedup minimise both bytes transferred and bytes stored; a cursor-based delta makes sync self-correcting; and a conflicted copy keeps the promise that nothing is ever lost. Say that and the boxes follow.
Maps to S1 (the six core trade-offs) and S12 (present and defend).
Use this in the cohort
- Slot: Week 6 worked design. The dedup and conflict ideas reappear in many designs; this is the cleanest place to teach them.
- Run it live: walk steps 1 to 5, then ask "you change one byte of a 1GB file, what gets uploaded?" and let the room discover content-defined chunking.
- The one sentence: "chunk by content so edits stay local, address chunks by hash so identical ones store once, sync deltas off a cursor, and on a fork keep both copies."