A file sync service sounds like a glorified upload button until you count the bytes. The naive version is a single PUT /file that stores a blob and a GET /file that returns it - and it works right up to the moment a user edits one paragraph in a 2GB video project and your service happily re-uploads all 2GB over their home DSL line. Multiply that by 700M users across 4 devices each, all expecting a saved file to appear on every other device within seconds, and the upload button collapses. The entire engineering problem of Dropbox is this: move the minimum number of bytes to keep N devices holding identical file trees, while storing those bytes once no matter how many users or versions contain them, and reconciling edits that happen concurrently on devices that were offline and never saw each other.

Everything hard here - chunking, deduplication, the metadata/blob split, sync conflict handling, delta sync - is machinery to drive that byte count toward zero. You do not sync files; you sync a small stream of chunk-hash lists, and you move the actual chunk bytes only when nobody, anywhere, already has them. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Upload and download files. A user puts a file into their Dropbox folder; it is durably stored in the cloud and retrievable.
  • Sync across devices. Any change on one device (add, edit, rename, move, delete) propagates to all the user’s other devices and to shared collaborators, automatically and quickly.
  • Delta sync. Editing part of a large file transfers only the changed portion, not the whole file, in both directions.
  • Deduplication. Identical content - the same file uploaded by many users, or the same chunk repeated across files/versions - is stored physically once.
  • Offline edits and reconciliation. A device edits while offline; on reconnect its changes merge with everything that happened remotely, and genuine conflicts are surfaced (never silently lost).
  • Version history. Prior versions of a file are retained and restorable for a retention window.
  • Sharing. A folder or file can be shared with other users, who then sync it into their own tree with permissions (view/edit).

Explicitly out of scope (say this out loud so you own the scope):

  • Real-time collaborative editing inside a document (Google Docs / OT / CRDT). Dropbox syncs whole files; concurrent character-level merge is a different problem. I sync files, not keystrokes.
  • Rich file preview / thumbnail rendering / full-text search indexing. Feature layers on top of storage; I will note where they hook in.
  • Auth, billing, and the account service. Assume a service answers “who is this user and what is their quota.”
  • The desktop client’s filesystem watcher internals (inotify/FSEvents plumbing). I assume the client can detect local changes; the interesting part is what it does with them.

The one decision that drives everything: this is a write-optimize-for-bytes-moved, read-and-write-both-matter, per-file-tree stateful system where correctness means every device converges to the same file tree and no acknowledged version is ever lost. Unlike a feed (read-heavy, staleness fine) the sync must be prompt and lossless; unlike chat (route a message once) the same bytes recur constantly and must be stored once. That combination - dedup on storage, minimal delta on the wire, convergence across offline devices - is the whole problem.

Non-Functional Requirements (NFR)

  • Scale: ~700M registered users, ~500M files touched per day, average ~4 devices per active user, hundreds of billions of files stored, exabyte-scale total bytes (before dedup). Peak concurrent connected clients in the tens of millions (each device holds a long-poll/notify channel).
  • Latency: a change on device A should appear on device B within a few seconds when both are online (notification p99 under ~1s; the transfer itself depends on file size). File open/download starts streaming in under ~200ms to first byte from the nearest region.
  • Availability: 99.99% on the metadata and notify path. A device must never be blocked from editing locally; sync is asynchronous and resumes after any outage.
  • Consistency: strong consistency on metadata (the file tree, version pointers, and namespace are the source of truth and must never disagree with themselves), eventual consistency on propagation (a remote device may be a few seconds behind, which is fine). Read-your-writes within a device is immediate because the local filesystem is authoritative locally.
  • Durability: 11 nines on stored bytes (an acknowledged upload is never lost). Metadata is replicated and quorum-committed. Version history is durable for the retention window.
  • Bytes-moved is the real budget. It is acceptable for propagation to take seconds; it is never acceptable to re-transfer or re-store bytes the system already has. Dedup and delta sync are correctness-of-cost requirements, not nice-to-haves.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Users and files:

700M registered, assume ~100M daily active users (DAU).
Each DAU touches ~5 files/day (edit, add, rename).
=> 100M * 5 = 500M file-change events/day
   500M / 86,400 sec ≈ 5,800 writes/sec (steady, metadata level)
   Peak (bursty, ~5x) ≈ 30,000 metadata writes/sec

Chunk traffic (the byte layer, where the money is):

Fix a chunk size of ~4MB (content-defined, average). A file change does not move the whole file - only the changed chunks. Say the average change touches ~2 chunks (8MB) of new/modified content, but dedup eliminates a large fraction - many chunks already exist:

500M changes/day * ~2 changed chunks * 4MB = 4 PB/day of candidate chunk uploads
Dedup + delta typically kills 60-90% of that (identical/unchanged chunks skipped).
Assume 70% eliminated: ~1.2 PB/day actually crosses the wire and lands new.
   1.2 PB/day / 86,400 ≈ ~14 GB/sec of new chunk ingest (steady)
   Peak ~5x ≈ ~70 GB/sec

That is why dedup and delta sync are not optional - without them you 3-5x your storage and egress bill.

Storage:

Say 100B files stored, average 1MB effective after dedup (many are small; big ones dedup hard).
   100B * 1MB = 100 PB of unique chunk data.
   With erasure coding (1.4x overhead) instead of 3x replication:
      ≈ 140 PB physical for the blob store.
Metadata is tiny by comparison:
   100B files * ~1KB metadata (path, size, chunk-hash list, version ptr)
   ≈ 100 TB of metadata, before replication (RF=3 ≈ 300 TB).

The split is stark: exabyte-ish blob store, ~100TB metadata store. They have wildly different access patterns and cost profiles, which is exactly why they must be separate systems.

Read vs write:

Writes (metadata): ~5,800/sec steady, ~30K/sec peak.
Reads (metadata):  device sync polls + listings + shared-folder fan-out.
   100M DAU * 4 devices * a few list/poll ops/min ≈ ~100K-200K metadata reads/sec.
   => metadata is read-heavy (~10-30:1 read:write). Cache and replicas matter.
Chunk reads (downloads): dominated by first-sync and shared-file pulls;
   tens of GB/sec egress, served from object storage + CDN edge.

Notification channels:

Tens of millions of connected devices holding a long-lived notify channel.
At ~500K-1M channels per tuned notify server: 20M / 500K ≈ 40 notify servers
just to hold the connections. Payloads are tiny (a "namespace changed" ping).

The takeaways: metadata is a high-QPS, read-heavy, strongly-consistent small-record store; the blob layer is an exabyte-scale, dedup-first, erasure-coded object store; and the single biggest architectural lever is bytes-moved, which chunking + dedup + delta sync attack directly. Those three, not the raw scale, are the interview.

High-Level Design (HLD)

The architecture splits hard along one seam: metadata (small, structured, strongly consistent, the source of truth for the file tree) is completely separate from blob data (huge, immutable, content-addressed chunks). The desktop/mobile client chunks files locally, hashes each chunk, and talks to a Metadata Service to reconcile the file tree; it uploads only chunks the Block Service says it does not already have. A Notification Service tells other devices “your namespace changed, come sync.”

   ┌────────────┐     ┌────────────┐     ┌────────────┐
   │  Client A  │     │  Client B  │     │ Web / API  │   (each client: local
   │  (chunker, │     │  (chunker, │     │  clients   │    chunker + hasher +
   │   hasher,  │     │   hasher,  │     │            │    local index DB)
   │   watcher) │     │   watcher) │     │            │
   └─────┬──────┘     └─────┬──────┘     └─────┬──────┘
         │  HTTPS            │  notify chan     │
   ┌─────▼───────────────────▼──────────────────▼─────┐
   │            Load Balancer  /  API Gateway           │
   └───┬───────────────┬──────────────────┬─────────────┘
       │ metadata ops  │  chunk up/down   │  subscribe
 ┌─────▼──────────┐ ┌──▼───────────────┐ ┌▼──────────────────┐
 │ Metadata       │ │  Block Service   │ │ Notification      │
 │ Service        │ │  (has-chunk?     │ │ Service           │
 │  - file tree   │ │   put chunk,     │ │  - long-poll /    │
 │  - versions    │ │   get chunk,     │ │    push channels  │
 │  - chunk lists │ │   dedup check)   │ │  - "namespace N   │
 │  - namespaces  │ │                  │ │    changed" ping  │
 └───┬────────────┘ └───┬──────────────┘ └───────┬───────────┘
     │                  │                          │
 ┌───▼──────────┐  ┌────▼───────────────┐   ┌──────▼─────────┐
 │ Metadata DB   │  │  Chunk Index /     │   │  Pub/Sub bus    │
 │ (sharded SQL/ │  │  Dedup Store       │   │  (per-namespace │
 │  strongly     │  │  hash -> location, │   │   change fanout)│
 │  consistent)  │  │  refcount          │   └─────────────────┘
 │  files, chunks│  └────┬───────────────┘
 │  versions,    │       │ points into
 │  namespaces   │  ┌────▼───────────────────────────────────┐
 └───────────────┘  │  Object / Block Storage (S3-like)        │
                    │  immutable, content-addressed chunks,    │
                    │  erasure-coded, multi-region, CDN-fronted│
                    └──────────────────────────────────────────┘

Upload flow (device A saves an edited file):

  1. The client’s filesystem watcher detects the file changed. The client chunks the file locally (content-defined chunking, ~4MB average), computes a strong hash (SHA-256) per chunk, and diffs the new chunk-hash list against the version it last synced - producing the set of added/changed chunks only.
  2. The client asks the Block Service has-chunks? for that set of hashes. The service returns which hashes already exist (globally, across all users). Only the truly-new chunks need uploading.
  3. The client uploads the new chunk bytes to the Block Service, which writes them to object storage (if not present) and increments a reference count for each chunk hash.
  4. The client sends the new file metadata (path, size, ordered chunk-hash list, its known base version) to the Metadata Service. The Metadata Service commits a new file version pointing at that chunk list, atomically, in the strongly-consistent Metadata DB, and bumps the namespace’s monotonically increasing cursor.
  5. The Metadata Service publishes “namespace N changed to cursor X” to the Notification Service.

Download / sync flow (device B, online):

  1. Device B holds a long-poll channel; the Notification Service pings it: “namespace N changed.”
  2. B calls the Metadata Service: “give me changes since my cursor Y.” It gets the new file’s metadata - path and chunk-hash list.
  3. B diffs that chunk list against what it has locally (its own local chunk store). It downloads only the chunks it is missing from the Block Service / CDN, reassembles the file, and writes it to the local filesystem. Cursor advances to X.

The key insight: the client, not the server, does the chunking and hashing, and the wire only ever carries chunk-hash lists plus the chunks nobody has yet. Metadata is the small strongly-consistent brain that tracks “which chunks make which file version”; the blob store is a dumb, huge, immutable, content-addressed bag of chunks. Separating them lets each scale and be consistent on its own terms.

Component Deep Dive

The hard parts, naive-first then evolved: (1) chunking + deduplication - the byte-count engine; (2) metadata vs blob separation and the schema that ties them; (3) sync conflict handling across offline devices; (4) delta sync - moving only changed bytes.

1. Chunking and deduplication: why naive fails, then fixed-size, then content-defined

Naive approach: store each file as one opaque blob, dedup whole files by hashing the entire file. Upload a file, hash it, if that hash exists point to the existing blob; otherwise store it.

Where it breaks:

  • Whole-file dedup catches almost nothing after the first edit. Change one byte in a 2GB file and its full-file hash changes completely - zero reuse, you re-store and re-upload all 2GB. The common case (editing existing files) gets no benefit at all.
  • No delta sync is possible. With one opaque blob there is no unit smaller than the file to transfer, so the wire always moves the whole file.

Evolution A: fixed-size chunking. Split every file into fixed 4MB chunks, hash each chunk, dedup and transfer at the chunk level. Now editing the middle of a file only changes the chunks it touches.

Where it breaks - the boundary-shift problem:

Fixed offsets are brittle under insertion. Insert a few bytes near the front of a file and every subsequent chunk boundary shifts by that offset, so every downstream chunk’s content changes, so every downstream chunk gets a new hash - and dedup/delta collapse to nearly zero again.

File v1 (fixed 4MB boundaries):  [chunk0][chunk1][chunk2][chunk3]
Insert 10 bytes at offset 100 (inside chunk0):
File v2:                         [chunk0'][chunk1'][chunk2'][chunk3']
                                  ^ everything after the insert shifted by 10 bytes
=> chunk1, chunk2, chunk3 all have DIFFERENT bytes now (shifted content)
=> new hashes for ALL of them. You re-upload the whole tail for a 10-byte insert. ❌

Fixed chunking only works when edits are pure overwrites that do not change length. Real edits insert and delete. This is exactly why rsync-style tools use a rolling boundary.

Evolution B: content-defined chunking (CDC) with a rolling hash. Instead of cutting at fixed offsets, slide a small rolling-hash window (Rabin fingerprint) over the bytes and cut a boundary wherever the rolling hash matches a pattern (e.g. low N bits are zero), targeting ~4MB average chunks. Because boundaries are decided by content, not position, an insert only changes the one or two chunks around it - the boundaries downstream re-align to the same content.

Rolling hash over a sliding window; cut when (hash & 0xFFF... ) == 0  -> ~4MB avg.
Bound chunk size to [min, max] (e.g. 1MB..8MB) so pathological inputs are safe.

File v1:  [A][B][C][D]   (boundaries fall at content markers)
Insert 10 bytes inside B:
File v2:  [A][B'][C][D]  <- only B changed; C and D re-sync to the SAME boundaries
                            because the rolling hash re-finds the same cut points.
=> only ONE new chunk (B') to hash, upload, and store. ✅

Deduplication on top of CDC. Every chunk is content-addressed by its strong hash (SHA-256). The dedup store is a map chunk_hash -> {storage_location, refcount}. On upload the client asks “do these hashes exist?”; only misses are transferred. Dedup is global (across all users and files) for maximum savings - the same Linux ISO uploaded by a million users is stored once. Two guards make global dedup safe:

  • Reference counting for deletes. A chunk is only physically deleted when its refcount hits zero (no file version anywhere still points to it). Deleting one user’s file just decrements refcounts.
  • Hash collision safety and privacy. SHA-256 collisions are astronomically unlikely, so equal hash means equal content. For the privacy concern (global dedup can leak “does this file already exist” via timing), production systems scope dedup per-storage-zone or add server-side checks; I would mention convergent-encryption trade-offs but keep global dedup for the savings.
Client:  chunks = CDC(file); hashes = [SHA256(c) for c in chunks]
         missing = BlockService.has_chunks(hashes)   # returns only the misses
         for c in missing: BlockService.put(hash(c), bytes(c))   # refcount++ inside
         MetadataService.commit_version(file_id, chunk_list=hashes, base=known_ver)

The result: an insert or a small edit to a huge file uploads and stores a couple of chunks, dedup makes repeated content free, and the chunk-hash list is the compact currency both the wire and the metadata trade in.

2. Metadata vs blob separation

Naive approach: one database row per file holding the file bytes as a BLOB column, plus its path and version. Everything in one store, one query gets you the file and its metadata.

Where it breaks:

  • You cannot put exabytes in a transactional database. A strongly-consistent SQL store tuned for small hot records chokes on multi-GB blobs - buffer pools thrash, replication lag explodes, backups become impossible. The access patterns are opposite: metadata is tiny/hot/relational/consistent; blobs are huge/immutable/streamed.
  • Dedup becomes impossible. If bytes live inside per-file rows, the same chunk in two files is stored twice. Content addressing needs a shared chunk store, not per-file blobs.
  • Wrong scaling axis. Metadata QPS scales with user activity; blob bytes scale with content volume. Coupling them forces you to scale both when only one is hot.

The fix: two purpose-built stores joined by chunk hashes.

  • Metadata Service + Metadata DB (strongly consistent, sharded SQL / a NewSQL store). Holds the namespace tree, files, versions, and the ordered chunk-hash list per version. This is the source of truth for “what the file tree looks like.” It is small (~1KB/file), relational (files belong to namespaces, versions belong to files), and needs transactions and strong consistency - two devices must never see contradictory trees. So it is SQL/NewSQL, sharded by namespace_id.
  • Block/Object Store (S3-like, erasure-coded, content-addressed). Holds immutable chunks keyed by their hash. No transactions, no mutation (a chunk never changes - a new content means a new hash), just durable, cheap, massively scalable byte storage, CDN-fronted for reads.
  • Chunk Index / Dedup Store. Maps chunk_hash -> storage_location + refcount. Bridges metadata (which references chunks by hash) to physical bytes.

The join is elegant: a file version in metadata is just an ordered list of chunk hashes; the bytes live once in the block store; many versions and many users’ files share chunks by referencing the same hashes. Metadata mutates constantly (renames, new versions) but is tiny; blobs are immutable and huge. Each store gets exactly the consistency model and cost profile it needs.

Metadata DB (strong consistency, sharded by namespace):
   file:    id, namespace_id, path, latest_version_id
   version: id, file_id, size, chunk_hashes=[h1,h2,h3...], created_at, device_id
Block store (immutable, content-addressed, erasure-coded):
   h1 -> bytes...     (stored ONCE, referenced by every version that contains it)
Dedup index:
   h1 -> { location, refcount:  N }

3. Sync conflict handling across offline devices

Naive approach: last-writer-wins on the whole file by wall-clock timestamp. Whichever version has the latest mtime overwrites the others.

Where it breaks:

  • You silently destroy work. Device A (offline on a plane) edits report.docx; device B (at the office) edits the same file. Both come online. LWW keeps one, the other person’s edits vanish with no trace. For a service whose promise is “we never lose your files,” this is fatal.
  • Clocks lie. Device clocks skew by minutes; “latest timestamp” is not even reliably the latest edit. You can lose the newer edit to a device with a fast clock.
  • No causality. LWW cannot tell “B’s edit is based on A’s version” (a legitimate update to keep) from “A and B both edited the same base independently” (a true conflict). It treats both identically.

The fix: version vectors / causal versioning per file, and materialize genuine conflicts as separate files rather than resolving them destructively.

Each file version records the base version it was derived from (a version id or a per-namespace cursor the device had when it started editing). The Metadata Service commits a new version only if it is a clean fast-forward from the current latest:

Commit rule (at the Metadata Service, inside a transaction on the file row):
  incoming.base_version == file.latest_version ?
     YES -> fast-forward: commit incoming as the new latest.        (clean update)
     NO  -> two edits diverged from a common ancestor.              (CONFLICT)

On a fast-forward the update just applies - this is the overwhelmingly common case (a device syncing its own sequential edits, or picking up a remote change before editing). On divergence, we do not pick a winner and delete the loser. Instead:

  • Both versions are kept. The base-winning version stays as report.docx; the losing device’s version is materialized as a conflicted copy: report (Device-B's conflicted copy 2026-07-07).docx. Both are real files the user can see, compare, and merge by hand. Nothing is lost - this is Dropbox’s actual behavior.
  • The decision is per file, atomic on the file row. The conflict check and version commit happen in one transaction in the strongly-consistent Metadata DB, so two devices racing to commit cannot both “win” - one fast-forwards, the other is told “you diverged” and creates the conflicted copy.
  • Directory-level operations (a file moved on A, edited on B) reconcile at the namespace level: moves/renames are metadata ops with their own base-cursor check; a move plus a concurrent edit both apply because they touch different attributes (path vs content), merged in the tree.

Why not OT/CRDT merge like Google Docs? Because Dropbox syncs opaque files - it has no idea how to merge two .docx or .psd files at the byte level, and guessing would corrupt them. The honest, lossless answer for arbitrary file types is: fast-forward when causally clean, and surface a conflicted copy when not, letting the human or the app do the semantic merge. Correctness here means never losing an acknowledged version, and conflicted copies deliver exactly that.

A offline (base=v5) edits report.docx -> wants to commit A' (base v5)
B online     (base=v5) edits report.docx -> commits B' cleanly (v5 -> v6)
A reconnects, tries to commit A' with base=v5, but latest is now v6:
   base(v5) != latest(v6)  -> CONFLICT
   keep v6 as report.docx
   materialize A' as "report (A's conflicted copy 2026-07-07).docx" (its own new file)
   both sync to every device; the user reconciles. Zero data lost. ✅

4. Delta sync: moving only the changed bytes

Naive approach: on any change, re-send the whole file to the server and re-download the whole file to peers.

Where it breaks:

  • Bandwidth is the bottleneck. A one-line change to a 500MB log file re-transfers 500MB up and 500MB down per peer device. On residential upload speeds that is minutes for a trivial edit, and it destroys the “saves appear instantly” feel.
  • It wastes the dedup you already built. You computed chunk hashes; re-sending the whole file throws that information away.

The fix: chunk-level delta on both the upload and download paths, driven by the chunk-hash lists.

  • Upload delta. The client already chunked the file with CDC and holds the chunk-hash list of the previous version (in its local index DB). It computes the new version’s chunk list, set-diffs against the old, and uploads only chunks whose hashes are new - after a has-chunks? check that also skips any chunk that exists globally via dedup. A 10-byte insert into a 2GB file uploads one ~4MB chunk (often less if the edit is smaller than a chunk; the client can additionally do sub-chunk rsync-style rolling diffs within the single changed chunk for tiny edits).
Upload delta:
  old_hashes = local_index[file].chunk_hashes         # what the server already has
  new_hashes = CDC_and_hash(current_file_bytes)
  to_send    = [h for h in new_hashes if h not in old_hashes]
  to_send    = BlockService.has_chunks(to_send)        # dedup: drop globally-existing
  upload only to_send; then commit new_hashes as the version's chunk list.
  • Download delta. The peer device receives the new version’s chunk-hash list from metadata, diffs it against the chunk hashes it already holds locally (from the prior version or any other file - its local chunk store is content-addressed too), and downloads only the chunks it is missing. Reassembly is “concatenate chunks in list order.”
Download delta (device B):
  new_hashes = MetadataService.get_version(file).chunk_hashes
  have       = local_chunk_store.keys()
  missing    = [h for h in new_hashes if h not in have]
  fetch missing from Block store / CDN; reassemble file = concat(chunks in order).
  • Both directions reuse one primitive: the chunk-hash list as a compact file recipe. The wire never carries a file - it carries a list of hashes plus the bytes that are genuinely new. This is why the earlier BoE showed 60-90% of candidate bytes eliminated: most chunks in any given edit already exist somewhere.

The combination of CDC (stable boundaries), global dedup (store once), and chunk-list diffing (transfer once) is what turns “sync a 2GB file” into “move a few kilobytes of hash lists plus one changed chunk.” That is delta sync.

API Design & Data Schema

Most sync traffic is metadata calls plus chunk transfers; a long-poll/notify channel handles change signaling. REST-ish HTTPS with signed URLs for chunk I/O.

REST / RPC (metadata + block)

POST /api/v1/blocks/probe
  -> "which of these chunk hashes do you already have?" (dedup check)
  Body: { "hashes": ["h1","h2","h3", ...] }
  Response 200: { "missing": ["h2"] }                # only h2 needs uploading

PUT  /api/v1/blocks/{hash}
  -> upload one chunk's bytes (idempotent; keyed by content hash). refcount++ .
  Response 201: { "hash": "h2", "stored": true }

GET  /api/v1/blocks/{hash}
  -> download one chunk (redirects to a signed CDN/object-store URL).
  Response 302 -> https://cdn.../blocks/h2?sig=...

POST /api/v1/files/commit
  -> create/update a file version pointing at an ordered chunk-hash list.
  Body: { "namespace_id":"ns7", "path":"/report.docx", "size":812340,
          "chunk_hashes":["h1","h2b","h3"], "base_version":"v5", "device_id":"A" }
  Response 200: { "version_id":"v6", "namespace_cursor": 90412 }      # fast-forward
  Response 409: { "conflict": true, "latest_version":"v6",            # diverged
                  "action":"create_conflicted_copy" }

GET  /api/v1/namespaces/{ns}/changes?since_cursor=90410
  -> the delta of metadata changes since the client's cursor (pull after a notify).
  Response 200: { "changes":[ {"path":"/report.docx","version_id":"v6",
                  "chunk_hashes":["h1","h2b","h3"],"op":"update"} ],
                  "cursor": 90412 }

GET  /api/v1/files/{id}/versions
  -> version history for restore. Response: { "versions":[ {...} ] }

POST /api/v1/files/{id}/restore   { "to_version":"v3" }
  -> restore as a NEW forward version (history stays immutable).

Notification channel

// client -> notify server: subscribe to its namespaces, hold open
{ "type":"SUBSCRIBE", "namespaces":["ns7","ns12"], "cursor":90410 }

// notify server -> client: a cheap "go sync" ping (no payload data)
{ "type":"CHANGED", "namespace":"ns7", "cursor":90412 }

The notify frame carries no file data - just “namespace changed to cursor X.” The client then pulls the actual delta via GET /changes?since_cursor. This keeps the notify path a tiny, cheap fan-out and the metadata pull a precise, cursor-based catch-up.

Data store schemas

1. Metadata DB - sharded SQL / NewSQL, strongly consistent, sharded by namespace_id. Relational, small records, transactional (the conflict check and version commit must be atomic).

Table: namespaces           -- a user's root or a shared folder
  namespace_id   BIGINT   PRIMARY KEY / SHARD KEY
  owner_id       BIGINT
  cursor         BIGINT             -- monotonic change counter (sync watermark)

Table: files
  file_id        BIGINT   PRIMARY KEY
  namespace_id   BIGINT   SHARD KEY, INDEX
  path           TEXT               -- logical path within the namespace
  latest_version BIGINT             -- FK -> file_versions.version_id
  is_deleted     BOOL
  INDEX (namespace_id, path)        -- list a folder, resolve a path

Table: file_versions          -- immutable; a version = an ordered chunk-hash list
  version_id     BIGINT   PRIMARY KEY
  file_id        BIGINT   INDEX
  base_version   BIGINT             -- causal parent (conflict detection)
  size           BIGINT
  chunk_hashes   TEXT / ARRAY       -- ordered list of SHA-256 chunk hashes
  device_id      TEXT               -- who wrote it (attribution)
  created_at     TIMESTAMP
  INDEX (file_id, created_at DESC)  -- version history, newest first

2. Chunk / Dedup Index - wide-column KV (Bigtable / Cassandra / a KV store), keyed by chunk hash. Huge key count, simple point lookups, high write rate, refcount updates.

Table: chunks
  chunk_hash     STRING   PARTITION KEY      -- SHA-256, content address
  location       STRING                      -- object-store key / shard pointer
  refcount       BIGINT                      -- # of versions referencing it (GC)
  size           INT
  Read:  point get by hash (probe/dedup)     -- O(1)
  Write: put-if-absent + refcount atomic incr/decr

3. Block / Object Store - S3-like, erasure-coded, content-addressed, CDN-fronted. Immutable blobs keyed by hash. No schema beyond hash -> bytes; durability from erasure coding across racks/zones, reads accelerated by CDN edge caches.

Why SQL/NewSQL for metadata but NoSQL/object for blobs: metadata is relational (namespaces -> files -> versions), small, and demands transactions and strong consistency (the atomic conflict-check-then-commit, and the invariant that no two devices see contradictory trees) - that is textbook SQL, sharded by namespace_id so one user’s tree is co-located and transactional. Blobs are immutable, enormous, and need only content-addressed durability and cheap streaming reads - that is an object store, where SQL would be catastrophically wrong. The chunk index is a simple high-volume KV map - NoSQL. Each layer gets the exact consistency and cost model its access pattern needs; forcing one store to do all three is the naive mistake.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Metadata hot namespace / commit contention (breaks first). A large shared folder with many active collaborators funnels all its version commits through one namespace shard, and the atomic conflict-check-then-commit serializes on the file row. Fix: shard by namespace_id so different namespaces scale out independently; within a namespace, the atomic commit is per-file row, so distinct files commit in parallel - only concurrent edits to the same file serialize, which is rare and exactly where you want serialization (that is the conflict detector). Very large shared folders can be split into sub-namespaces. Keep the commit transaction tiny (a compare-and-set on latest_version plus an insert), so throughput per shard stays high.

2. Chunk upload/download bandwidth (the byte firehose). Fix: this is what chunking + dedup + delta sync exist for - the has-chunks? probe kills 60-90% of candidate bytes before they hit the wire. Uploads go directly to the object store via signed URLs (bytes never transit the metadata tier). Downloads are served from CDN edge caches and multi-region object storage, so popular chunks (shared files) are served near the user. Erasure coding (1.4x) instead of 3x replication cuts storage overhead by more than half at the same durability.

3. Notification fan-out to tens of millions of devices. Fix: notify servers hold cheap long-poll/WebSocket channels (hundreds of thousands each), fed by a per-namespace pub/sub bus. The ping carries no data (just a cursor), so fan-out is tiny; the actual delta is pulled on demand by each device. Devices back off with jitter to avoid a thundering herd when a big shared folder changes.

4. Dedup index lookups at ingest rate. Fix: the hash -> location/refcount index is a horizontally-sharded KV store, sharded by chunk_hash (uniform distribution by construction, since hashes are random). Point lookups are O(1) and cache extremely well - a Bloom filter in front of the probe endpoint cheaply answers “definitely-not-present” for the common brand-new-chunk case, skipping the index read.

5. Reconstructing / listing state efficiently. Fix: the namespace cursor gives clients an incremental sync watermark - GET /changes?since_cursor returns only the delta, never a full tree re-list. First sync of a huge folder streams chunk lists then pulls missing chunks in parallel. Metadata reads (read-heavy, ~10-30:1) are served from read replicas and a cache in front of the Metadata DB.

6. Chunk garbage collection and hot deletes. Fix: deletes decrement refcounts; a background GC sweeps chunks at refcount 0 (with a grace period so a concurrent upload racing a delete does not lose a chunk - the refcount is atomic and GC only collects chunks that stayed at zero across the window). Version history retention pins chunks referenced by any version inside the retention window, so restore stays possible.

7. Shard keys, stated plainly. Metadata DB -> namespace_id (a user’s/shared-folder’s tree is one co-located, transactionally-consistent unit; enables the atomic per-file conflict commit and cursor-based sync). Chunk/dedup index -> chunk_hash (uniformly distributed, point-lookup, global dedup across all users). Object store -> content hash (content addressing is the placement). Sharding metadata by user_id would split shared folders across shards and break their transactional tree; sharding chunks by anything but hash would break global dedup. The namespace is the unit of metadata consistency; the chunk hash is the unit of storage identity.

8. Hot chunks (a viral shared file everyone downloads). Fix: content-addressed chunks are trivially cacheable - push them to CDN edges and replicate hot chunks to more object-store zones. Because chunks are immutable, caching is safe forever with no invalidation problem (a changed file is new chunks with new hashes, never a mutation of an existing hash).

9. Single points of failure. API gateways and notify servers are stateless and interchangeable. The Metadata DB is quorum-replicated (RF=3) with automatic failover; a committed version survives node loss. The object store is erasure-coded across zones for 11-nines durability. The dedup index is replicated. No single box whose loss loses data - at worst clients reconnect and re-poll their cursor.

10. Multi-region and offline resilience. Metadata has a home region per namespace (co-located editors get fast commits; the strong-consistency invariant needs a single authoritative commit path, so you do not run two independent commit masters for one namespace). Chunks replicate cross-region and are CDN-fronted so downloads are always near-user. Offline devices queue changes locally against their last cursor and reconcile on reconnect via the same fast-forward-or-conflict machinery - correctness never depends on connectivity, only propagation speed does.

Wrap-Up

The trade-offs that define this design:

  • Chunk-level content addressing over whole-file storage. Content-defined chunking (a rolling hash cutting boundaries by content, not offset) makes inserts cheap and enables both global dedup and delta sync - at the cost of per-chunk hashing on the client and a chunk index to maintain. Fixed chunking was rejected because insertions shift every downstream boundary; whole-file storage was rejected because a one-byte edit re-moves the whole file.
  • Metadata and blobs are separate systems, joined by chunk hashes. Metadata is small, relational, strongly consistent SQL sharded by namespace; blobs are immutable, exabyte-scale, erasure-coded object storage keyed by content hash. Coupling them (a BLOB column) would break dedup, cap scale, and force one consistency model on two opposite workloads.
  • Lossless conflict handling over automatic merge. Because Dropbox syncs opaque files it cannot semantically merge, so it fast-forwards causally-clean updates and materializes genuine divergence as conflicted copies - trading the convenience of auto-merge for the absolute guarantee that no acknowledged version is ever silently lost. Last-writer-wins was rejected because it destroys work and trusts skewed clocks.
  • Bytes-moved as the primary budget. Delta sync in both directions, driven by chunk-hash-list diffing plus a global dedup probe, eliminates 60-90% of candidate bytes before they hit the wire - turning “sync a 2GB file” into “move a few chunks.”
  • Cheap notify, precise pull. A dataless “namespace changed to cursor X” ping fans out to tens of millions of devices, each pulling only its incremental delta - keeping the fan-out tiny and the catch-up exact.

One-line summary: a file sync service that chunks files on the client with content-defined boundaries, deduplicates and stores every chunk exactly once in a content-addressed erasure-coded object store, tracks the file tree in a separate strongly-consistent metadata database that records each version as an ordered chunk-hash list, moves only the chunks nobody already has in either direction via delta sync, signals changes through a cheap cursor-based notify channel, and reconciles concurrent offline edits by fast-forwarding causally-clean updates and preserving genuine conflicts as separate files - so N devices converge to one identical tree while the bytes on the wire and on disk both trend toward the theoretical minimum.