Almost everyone reaches for Google Docs the moment they hear “document editor,” and almost everyone is answering the wrong question. Google Docs is a real-time collaboration problem: two people typing into the same sentence at the same millisecond, operational transforms, CRDTs, convergence. That is a specific and hard sub-problem, and it is not what Notion or Wikipedia are. Notion and Wikipedia are async document systems: you open a page, you edit it, you save it. Someone else might edit it an hour later, or a minute later, but the design does not live or die on sub-200ms convergence of concurrent keystrokes. It lives or dies on three other things - how you store a document made of structured blocks, how you keep a full version history without exploding storage, and how you search across tens of millions of documents fast.

So I am deliberately taking the road away from OT and CRDTs. The interviewer said async edits and version history, not real-time collaboration, and that is a gift: it means I can use boring, durable, well-understood building blocks (a database with optimistic concurrency, a copy-on-write version chain, an inverted index) instead of the exotic convergence machinery. The hard parts move from “make concurrent edits converge” to “model a document as a tree of blocks, version it cheaply, and make it searchable at scale.” Let me build it properly.

Functional Requirements (FR)

In scope:

  • Create, read, update, delete documents. A user creates a document, opens it, edits its content, and saves. The document is a structured thing (a tree of blocks: headings, paragraphs, lists, tables, images), not a flat blob of text.
  • Block-structured editing. A document is composed of ordered blocks. Editing means adding, deleting, reordering, and modifying individual blocks - not rewriting the whole document each time. This is the Notion model and it matters for both storage and versioning.
  • Version history. Every save creates a new immutable version. Users can view the timeline, diff two versions, see who changed what, and restore an old version. This is Wikipedia’s “View history” tab and Notion’s page history.
  • Async concurrency. Two users may edit the same document at different times (or with overlapping-but-not-keystroke-simultaneous sessions). We need a sane conflict policy - detect the conflict, do not silently clobber - but we do NOT need real-time operational transforms.
  • Full-text search. Search across all documents by content, title, and metadata, ranked by relevance, with permission filtering (you only see results in docs you can read).
  • Hierarchy and links. Documents live in a tree (Notion workspace / page nesting) or a namespace (Wikipedia), and link to each other. Backlinks matter.

Explicitly out of scope (say this so you own the scope):

  • Real-time collaborative editing (OT/CRDT). No live cursors, no keystroke-level convergence. If you want that, it is a different system - I wrote that one separately. Here, edits are session-scoped and saved as whole revisions.
  • Rich-text rendering / WYSIWYG front end. How the browser paints a block is a client concern.
  • Auth and permission internals. Assume an auth service answers “can user U read/write doc D.” We consume it, especially in search.
  • Media transcoding. Images and files are stored in object storage; we hold references, not the pipeline that resizes thumbnails.

The one decision that drives everything: this is a read-heavy, async-write, versioned document store where the document is a structured tree of blocks and the two hard problems are cheap version history and full-text search - not concurrency convergence. That framing is the whole interview. Say it out loud and the rest follows.

Non-Functional Requirements (NFR)

  • Scale: 50M documents total. 10M edits/day (writes). Reads dominate heavily - assume a 100:1 read:write ratio, so ~1B document reads/day. A workspace/wiki has deep nesting; a single popular document (a company handbook page, a trending Wikipedia article) can be read millions of times a day.
  • Latency: document read p99 under 150ms (it is on the critical path of a user opening a page). Save/edit p99 under 300ms. Search p99 under 300ms. Version-history list under 200ms.
  • Availability: 99.9%+ on reads (a wiki being down is a visible outage). Writes can tolerate marginally lower availability - a failed save that the client can retry is acceptable, a lost save is not.
  • Consistency: read-your-writes for the editing user (after I save, I must see my save). Across users, eventual consistency is fine - if my teammate’s edit shows up to me a few seconds late, nobody is harmed. Search is allowed to lag the write path by seconds. This looseness is exactly what buys us the boring, scalable design.
  • Durability: an acknowledged save is never lost. Version history is immutable and durable - you cannot silently lose an old revision, because “restore” depends on it. This is the strongest guarantee in the system.

Back-of-the-Envelope Estimation (BoE)

Let me ground the design in numbers.

Write QPS (edits):

Edits per day       = 10,000,000
Seconds per day     = 86,400
Average write QPS   = 10M / 86,400 ≈ 116 writes/sec
Peak (say 5x)                        -> ~580 writes/sec

Writes are tiny. 116/sec average is nothing - a single well-tuned database primary handles that. Writes are NOT the scaling problem here, which is a key realization. The problem is reads, versions, and search.

Read QPS:

Read:write ratio    = 100:1 (docs are read far more than edited)
Reads per day       = 10M edits * 100 ≈ 1,000,000,000 reads/day
Average read QPS    = 1B / 86,400 ≈ 11,600 reads/sec
Peak (5x)                            -> ~58,000 reads/sec

~12K reads/sec average, ~58K peak. This is where caching and read replicas earn their keep. A cache hit rate of 90%+ on hot documents takes most of this off the database.

Storage - documents:

Avg document: ~30 blocks, each block ~200 bytes of text/JSON
Avg doc size (current version)  ≈ 30 * 200 = 6 KB (call it 10 KB with metadata)
50M docs * 10 KB                = 500 GB for the current-version corpus

500 GB of live document content. Trivial - fits comfortably in a sharded relational or document store. The current state is small.

Storage - version history (this is the real number):

Assume avg 50 versions retained per document over its lifetime.
Naive full-snapshot per version: 50M docs * 50 versions * 10 KB
   = 50M * 500 KB = 25 TB
With block-level dedup / delta versions (only changed blocks stored):
   avg edit touches ~2 blocks of 30 -> ~7% of the doc changes per version.
   50M * 50 * (10 KB * 0.07) ≈ 1.75 TB

Full snapshots per version blow up to 25 TB; block-level delta versioning cuts it to ~1.75 TB - a 14x reduction. That single decision (do not snapshot the whole doc every save) is what makes version history affordable, and it is a Component Deep Dive below.

Storage - search index:

Full-text index is typically ~30-50% of the raw text size.
Raw current text ≈ 300 GB (text only, minus block JSON overhead)
Inverted index ≈ 0.4 * 300 GB ≈ 120 GB
Replicated 2x for HA + query throughput -> ~240 GB

~120 GB of index, replicated. Sits in an Elasticsearch/OpenSearch-class cluster of a handful of nodes.

Cache size:

Hot working set: say 5% of docs are "hot" on a given day
5% of 50M = 2.5M docs * 10 KB = 25 GB of hot document bodies
Add rendered/parsed forms + metadata -> ~50-75 GB cache

A ~64 GB Redis tier (sharded) holds the hot document working set. At a 90% hit rate it absorbs the bulk of the 12K reads/sec.

Bandwidth:

Read egress: 12K reads/sec * 10 KB = 120 MB/sec average, ~600 MB/sec peak.
Write ingress: 116/sec * 10 KB = ~1.2 MB/sec. Negligible.

The takeaway from the BoE: writes are trivial, current-state storage is trivial, and the entire engineering challenge is (1) read throughput at ~12-58K/sec, (2) version-history storage without a 25 TB blowup, and (3) a 120 GB full-text index searched with permission filtering. The design is shaped by those three, not by write load.

High-Level Design (HLD)

The system is a read-heavy versioned store with a search sidecar. The write path is simple and synchronous; the search path is async off a change stream; the read path is cache-first. Here is the end-to-end architecture.

                         ┌───────────────┐
                         │    Clients     │  (web / mobile / API)
                         └───────┬────────┘
                                 │ HTTPS
                         ┌───────▼────────┐
                         │  API Gateway /  │  auth, rate limit, routing
                         │  Load Balancer  │
                         └───────┬────────┘
              ┌──────────────────┼───────────────────────┐
              │ read             │ write                  │ search
     ┌────────▼────────┐ ┌───────▼─────────┐    ┌─────────▼─────────┐
     │  Document Read   │ │  Document Write  │    │   Search Service  │
     │    Service       │ │     Service      │    │  (query + rank)   │
     └───┬─────────┬────┘ └───────┬──────────┘    └─────────┬─────────┘
         │ hit     │ miss         │ save                    │ query
   ┌─────▼───┐ ┌───▼──────────────▼─────────┐        ┌──────▼────────┐
   │  Redis  │ │      Document Store          │        │  Search Index │
   │  Cache  │ │  (sharded by document_id)    │        │ (Elasticsearch│
   └─────────┘ │  - current_version pointer   │        │  inverted idx)│
               │  - blocks / version deltas   │        └──────▲────────┘
               └───────────┬──────────────────┘               │ index docs
                           │ CDC / change stream               │
                           ▼                                   │
                  ┌─────────────────────┐   consume    ┌───────┴────────┐
                  │  Change Log (Kafka) │─────────────►│  Indexer Worker │
                  └─────────────────────┘              │ (parse blocks → │
                           │                           │  text → index)  │
                           ▼                           └────────────────┘
                  ┌─────────────────────┐
                  │  Version History     │   immutable version records +
                  │  Store (append-only) │   block content-addressed store
                  └─────────────────────┘

     ┌────────────────────────────────────────────────────────────────┐
     │  Supporting: Object Storage (images/files), Auth Service,        │
     │  Permission/ACL Service, CDN (static + cacheable rendered docs)  │
     └────────────────────────────────────────────────────────────────┘

Read flow - opening a document, end to end:

  1. Client requests GET /documents/{id}. The gateway authenticates and checks with the permission service that the user can read the doc.
  2. The Document Read Service checks Redis for the current rendered/assembled version of the doc keyed by doc:{id}:v{current_version}. On a hit (the common case, ~90%), return immediately - this is the p99-under-150ms path.
  3. On a miss, read the document’s current_version pointer and assemble the block tree from the Document Store (resolving version deltas if needed - see deep dive), populate Redis, and return.
  4. Reads are served from read replicas, not the primary, since reads outnumber writes 100:1.

Write flow - saving an edit, end to end:

  1. Client sends PUT /documents/{id} with the new block set (or a block-level patch) and the base_version it started editing from (for optimistic concurrency).
  2. The Document Write Service checks that base_version == current_version. If not, it is a conflict (someone saved in between) - reject with 409 and let the client reconcile (deep dive 2).
  3. On no conflict: write the changed blocks to the version store (content-addressed), create a new immutable version record, and atomically bump the document’s current_version pointer. This is a single transaction on the document’s shard.
  4. Invalidate/refresh the Redis entry for the doc, and emit a change event to the Kafka change log.
  5. Return 200 with the new version number. The user immediately reads their own write (read-your-writes) because we updated the cache and the pointer synchronously.

Search indexing flow (async):

  1. The Indexer Worker consumes the change log, extracts plain text from the document’s blocks, and updates the Search Index (Elasticsearch). This lags the write by seconds, which the NFR explicitly allows.

The key architectural insight: the write path is synchronous and simple (optimistic concurrency + versioned append), and everything expensive or eventually-consistent - search indexing, cache warming, backlink computation - hangs off an async change stream. Because writes are only ~116/sec, we can afford a clean transactional write and let the fan-out be lazy.

Component Deep Dive

Three hard parts, each walked from naive to scalable: (1) the document/block storage model and how versioning is done cheaply, (2) async concurrency and conflict handling without OT, and (3) full-text search across 50M docs with permission filtering.

1. Block Storage + Version History Without a Storage Blowup

A Notion/Wikipedia document is not a blob - it is an ordered tree of blocks. How we store it dictates how versioning, editing, and reads work. And versioning is where naive designs explode.

Approach A: Store the whole document as one blob, snapshot every save (naive)

Model each document as a single JSON/HTML blob. Every save writes a brand-new full copy as a new version row.

document_versions
  doc_id | version | full_content (10 KB blob) | author | ts
   D1    |   1     | {...entire doc...}
   D1    |   2     | {...entire doc again...}   <- 99% identical to v1
   D1    |   3     | {...entire doc again...}

Where it breaks - two ways:

  • Storage blows up. The BoE showed 50 versions * 50M docs * 10 KB = 25 TB, and most of every version is byte-identical to the one before it. You are paying to store the same paragraph 50 times. It grows unbounded with edit count.
  • Edits are coarse and read-modify-write. To change one paragraph, the client must send the whole document and the server rewrites it whole. On a large doc (a 500-block wiki page) every keystroke-batch ships kilobytes, and two people editing different sections of the same big blob will collide on the whole blob even though their edits do not actually overlap.

The blob model is correct and simple, and it is fine at small scale, but it wastes 90%+ of its storage and makes fine-grained editing and diffing impossible.

Approach B: Blocks as first-class rows (the Notion model)

Decompose the document into individual blocks, each its own row with a stable ID, a type, content, and an ordering key. The document is a lightweight record pointing at an ordered list of block IDs (or blocks carry a parent_id + position to form the tree).

blocks
  block_id (PK) | doc_id | parent_id | type      | content        | position
   b1           | D1     | null      | heading   | "Onboarding"   | 1.0
   b2           | D1     | b1        | paragraph | "Welcome..."   | 2.0
   b3           | D1     | b1        | bullet    | "Step one"     | 3.0
  • Fine-grained edits. Changing one paragraph is an update to one block row, not a rewrite of the doc. Adding a block inserts a row. Reordering changes position (use a fractional/lexical ordering key like 2.5 between 2.0 and 3.0 so inserts never renumber siblings).
  • Independent sections. Two users editing block b2 and block b7 do not conflict at the block level at all - their saves touch disjoint rows.
  • Tree structure for free. parent_id gives you Notion’s nesting (toggle lists, columns, sub-pages) directly.

Where it strains: we still need versioning, and now “a version” is a snapshot of which blocks and which content existed at that time. Naively snapshotting all block rows per version reintroduces the 25 TB problem. So we need version storage that shares unchanged blocks.

Approach C: Content-addressed blocks + delta versions (the fix)

Make block content immutable and content-addressed, and make a version a cheap manifest that references block content by hash. This is copy-on-write, the same trick Git uses.

block_content            (immutable, content-addressed)
  content_hash (PK) | data
   h_a3f...         | {"type":"paragraph","text":"Welcome to the team"}
   h_9b1...         | {"type":"paragraph","text":"Welcome aboard"}   <- edited

document_versions        (immutable manifest per save)
  doc_id | version | block_manifest                       | author | ts | parent_version
   D1    |   1     | [b1→h_11, b2→h_a3f, b3→h_77]          | u1     |    | null
   D1    |   2     | [b1→h_11, b2→h_9b1, b3→h_77]          | u2     |    | 1
                     └── only b2 changed; b1,b3 point at SAME content as v1
  • Unchanged blocks are stored once. v2 differs from v1 only in b2’s content hash; b1 and b3 reuse the exact same immutable content rows. Editing one paragraph in a 500-block doc adds one small block_content row plus one manifest - not a full copy. This is the delta versioning that took the BoE from 25 TB to ~1.75 TB.
  • A version is immutable and cheap. Each document_versions row is a manifest (a list of block_id -> content_hash mappings) plus metadata. Restoring version N is just: make a new version whose manifest equals N’s manifest. Diffing two versions is a set-diff of their manifests - instantly tells you which blocks were added, removed, or changed.
  • The current version is a pointer. The documents table holds current_version; a read resolves the pointer -> manifest -> block contents. Assembly is a batched key-value fetch of the referenced content hashes, cached in Redis.
  • Garbage collection. When versions age out of the retention window (e.g. keep last 90 days + named versions), unreferenced block_content rows are GC’d via reference counting - content shared by any live version survives.

To avoid re-assembling a 500-block manifest on every read, we materialize the current version into the cache (and optionally a current_document denormalized row) so the hot read path is a single fetch, while history stays in the space-efficient delta form.

Version model Storage (50M docs, 50 ver) Fine-grained edit Diff/restore Verdict
Full-blob snapshot per save ~25 TB no (rewrite whole doc) expensive Rejected
Blocks as rows, snapshot all rows ~25 TB yes ok Half-fix
Content-addressed blocks + delta manifests ~1.75 TB yes trivial (manifest diff) The design

2. Async Concurrency: Detect Conflicts, Never Clobber

Two people edit the same document without real-time sync. We are explicitly NOT doing operational transforms. But “no OT” does not mean “let the last save win blindly” - that silently destroys the first person’s work. We need to detect concurrent edits and reconcile them.

Approach A: Last-writer-wins on the whole document (naive)

Both users load the doc, edit, and PUT it back. Whoever saves last overwrites the other. No version check.

Where it breaks: Alice loads v5, spends 20 minutes rewriting the intro. Bob loads v5, fixes a typo in the conclusion, saves (v6). Alice saves - and her save, based on v5, overwrites v6, erasing Bob’s typo fix. Neither user is warned. This is silent data loss, the one thing a system that promises “we never lose your writing” cannot do.

Approach B: Optimistic concurrency control with a version check

Every edit session records the base_version it started from. On save, the server compares base_version to the document’s current_version:

PUT /documents/D1  { base_version: 5, blocks: [...] }

server:  current_version == 5 ?
           yes -> apply, create v6, return 200
           no  -> 409 Conflict { current_version: 6 }
  • No lost updates. If someone saved in between, base_version (5) != current_version (6), the save is rejected with 409, and the client is told a newer version exists. Nobody is silently clobbered. This is an atomic compare-and-set on the document’s current_version, done inside the write transaction on the doc’s shard.
  • Optimistic, not locking. We do not lock the document while a user edits (pessimistic locks would kill a wiki - a page open in someone’s tab for an hour would block everyone). We assume conflicts are rare (they are, given async editing) and only pay a cost when one actually happens.

Where it strains: a bare 409 is a bad user experience if the whole document conflicts every time anyone touches it. On a large doc, Bob’s conclusion typo should not block Alice’s intro rewrite. We can do better because of the block model.

Approach C: Block-level optimistic merge (the fix, using the block model)

Because a document is a set of independently-versioned blocks (deep dive 1), we resolve conflicts at block granularity, not document granularity.

  • Per-block version check. The save carries, per changed block, the block’s base_content_hash. On save, for each block: if the block’s current hash still equals the base hash the editor started from, apply the change. Alice changed b2 (intro), Bob changed b7 (conclusion) - disjoint blocks, both saves succeed, auto-merged. This is the common case and it just works.
  • True conflict = same block edited by both. Only when Alice and Bob both edited the same block since a shared base do we have a real conflict. Then we surface it: return 409 for that block with both versions, and let the client show a merge UI (“your version” vs “their version,” like Wikipedia’s edit-conflict screen or Notion’s conflict handling). We never auto-pick a loser silently.
  • Wikipedia’s model as the reference. Wikipedia does exactly this: it attempts a three-way merge of the two edits against the common ancestor version; if the changed regions do not overlap, it merges automatically; if they do, it shows the editor a conflict screen. Our block boundaries make the “do the regions overlap” check trivial - two edits conflict iff they touch the same block.

So the concurrency story is: optimistic concurrency with compare-and-set at the block level, auto-merge disjoint blocks, and an explicit human-resolved conflict screen only when the same block is edited concurrently. No OT, no CRDT, no locks - just version checks over an already-decomposed document. This is the entire payoff of choosing the async framing.

3. Full-Text Search Across 50M Documents

Search is the other headline feature. “Find every doc mentioning kubernetes rollback,” ranked, filtered to what the user can read, fast.

Approach A: SQL LIKE '%term%' against the document store (naive)

SELECT doc_id FROM blocks WHERE content LIKE '%kubernetes%';

Where it breaks: a leading-wildcard LIKE cannot use an index - it is a full table scan of every block of every document, tens of millions of rows, per query. At 12K QPS this melts the database instantly. It also has no relevance ranking (which doc is the best match?), no stemming (rollback vs rolled back), no phrase matching. LIKE is not search; it is a scan that happens to match substrings.

Approach B: A dedicated inverted-index search engine

Move search off the primary store into an inverted index (Elasticsearch / OpenSearch, built on Lucene). The index maps each term to the list of documents containing it (a posting list), so a query is a lookup, not a scan.

term "kubernetes"  -> [doc_3, doc_17, doc_92, ...]   (posting list)
term "rollback"    -> [doc_17, doc_88, ...]
query "kubernetes rollback" -> intersect posting lists -> {doc_17, ...}
                            -> score by TF-IDF / BM25 -> ranked results
  • Sub-linear queries. Looking up two terms and intersecting their (pre-sorted) posting lists is orders of magnitude cheaper than scanning every row. The BoE index is ~120 GB, held across a few nodes in RAM/SSD.
  • Real search semantics. Analyzers do tokenization, lowercasing, stemming (rollback/rolled back match), stop-word removal, and synonyms. BM25 ranks results by relevance (term frequency, inverse document frequency, field boosts - a title match outranks a body match). Phrase and fuzzy queries fall out of the same index.
  • Kept fresh asynchronously. The Indexer Worker consumes the Kafka change log from the write path, extracts plain text from a document’s blocks (blocks -> concatenated text + title + tags), and upserts the document into the index. This lags writes by seconds - allowed by the NFR. We index the current version only; history is not searched.

The permission problem (the part interviewers push on)

Search must return only documents the user can read. A raw full-text match that ignores ACLs leaks private documents in the result snippets - a serious bug. Two strategies:

  • Filter-at-query-time (index the ACL). Store each document’s read-access principals (user IDs, group IDs, workspace ID) as a keyword field in the index. Every query is AND-ed with a terms filter: read_principals IN (user's groups). The engine intersects the text match with the permission filter in one pass. This is the standard approach and it scales because the filter is just another posting-list intersection.
  • Handle ACL churn. When a document’s permissions change (shared with a new group, page moved to a private space), we must reindex its read_principals. That is another event on the change log. The expensive case is a group membership change affecting many docs; we handle it by indexing group IDs (not expanded user lists) on the document, and resolving the user’s group set at query time - so adding a user to a group is a change to the user’s group set, not a reindex of every doc the group can see. This keeps ACL changes cheap.

For very large multi-tenant setups you also route/shard the index by workspace/tenant so a query only hits the shards for the user’s workspace, which both speeds queries and naturally bounds the permission scope.

So search is: an async-maintained inverted index (Elasticsearch/BM25) fed by the change log, with permissions enforced as an indexed group-filter intersected into every query, sharded by tenant. SQL LIKE is never on the path.

API Design & Data Schema

REST API

# Documents
POST   /api/v1/documents                      # create a document
        { "workspace_id": "...", "parent_id": "...", "title": "...", "blocks": [...] }
        -> 201 { "doc_id": "D1", "version": 1 }

GET    /api/v1/documents/{doc_id}             # read current version (cache-first)
        -> 200 { "doc_id", "version": 6, "title", "blocks": [...], "updated_at" }

PUT    /api/v1/documents/{doc_id}             # save an edit (optimistic concurrency)
        { "base_version": 5,
          "block_patches": [ { "block_id":"b2", "base_hash":"h_a3f", "content":{...} } ] }
        -> 200 { "version": 6 }
        -> 409 { "conflict_blocks": ["b2"], "current_version": 6 }   # same-block conflict

DELETE /api/v1/documents/{doc_id}             # soft-delete (trash), recoverable

# Version history
GET    /api/v1/documents/{doc_id}/versions    # paginated timeline
        -> [ { "version": 6, "author", "ts", "summary" }, ... ]
GET    /api/v1/documents/{doc_id}/versions/{v}          # read a specific version
GET    /api/v1/documents/{doc_id}/diff?from=5&to=6      # block-level diff
POST   /api/v1/documents/{doc_id}/restore     { "version": 4 }   # restore -> new version

# Search
GET    /api/v1/search?q=kubernetes+rollback&workspace_id=...&limit=20
        -> { "results": [ { "doc_id", "title", "snippet", "score" }, ... ] }

Data model - the right store for each job

1. Documents + version pointer (relational, sharded by doc_id):

documents
  doc_id           UUID    PK
  workspace_id     UUID          # tenant / shard-affinity, ACL scope
  parent_id        UUID          # tree/hierarchy (nesting)
  title            TEXT
  current_version  INT           # pointer to live version (compare-and-set on write)
  is_deleted       BOOL
  created_at       INT64
  updated_at       INT64
  INDEX (workspace_id, parent_id)   # list children of a page
  SHARD KEY: hash(doc_id)

2. Version manifests (append-only, immutable):

document_versions
  doc_id          UUID
  version         INT
  block_manifest  JSONB     # [ {block_id, content_hash, parent_id, position}, ... ]
  author_id       UUID
  created_at      INT64
  parent_version  INT
  summary         TEXT      # edit note (Wikipedia-style)
  PRIMARY KEY (doc_id, version)     # immutable; never updated
  SHARD KEY: hash(doc_id)           # a doc's history co-located with the doc

3. Block content (content-addressed, immutable, dedup):

block_content
  content_hash    BINARY(16)  PK    # hash of (type + content)
  data            JSONB             # the actual block payload
  ref_count       INT               # for GC of unreferenced content
  -- write-once; shared across all versions/docs that have identical content
  SHARD KEY: content_hash

Why relational + content-addressed KV, not pure NoSQL: the write path needs an atomic compare-and-set on current_version plus an insert of the version row - a single-row transaction on the doc’s shard, which relational stores (Postgres, or a sharded relational like Vitess/CockroachDB) do cleanly. Documents shard cleanly by doc_id, and a document’s blocks/versions co-locate on the same shard, so a read/write never spans shards. The block_content dedup table is a content-addressed KV workload (point lookups by hash) and could equally be a KV store (RocksDB/Cassandra/S3-for-large-blocks). We do NOT need a globally-distributed strong-consistency database because there are no cross-document transactions - each document is its own consistency island. That is what keeps writes simple at 116/sec.

4. Search index (Elasticsearch / OpenSearch):

document_index  (per current version)
  doc_id           keyword
  workspace_id     keyword      # routing + permission scope
  title            text  (boosted)
  body             text         # flattened block text, analyzed/stemmed
  tags             keyword[]
  read_principals  keyword[]    # user + group ids allowed to read (ACL filter)
  updated_at       date
  ROUTING: workspace_id         # query hits only the tenant's shards

5. Object storage: images/files referenced by blocks ({"type":"image","url":"s3://..."}); the DB holds the reference, S3/GCS holds the bytes.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. Read throughput (breaks first - 12K to 58K reads/sec). The document store cannot serve every open-a-page hit, especially for hot documents (a company handbook read millions of times/day). Fix: cache-first reads through a sharded Redis tier holding assembled current versions (~64 GB, 90%+ hit rate), backed by read replicas of the document store for misses. Put a CDN in front of public/immutable content (a specific version is immutable, so documents/{id}/versions/{v} is infinitely cacheable). Hot public wiki pages are served entirely from the edge.

2. Version-history storage blowup (25 TB). Full-snapshot-per-save stores the same content dozens of times. Fix: content-addressed blocks + delta manifests (deep dive 1) - store each unique block once, a version is a cheap manifest of hashes. 25 TB -> ~1.75 TB. GC unreferenced blocks by reference count once versions age out of retention.

3. Search cost and freshness. Searching the primary store with LIKE is a full scan; keeping search on the write path would slow saves. Fix: a dedicated inverted-index engine (Elasticsearch/BM25) fed asynchronously off the Kafka change log, so writes never wait on indexing and search lags by seconds (allowed). Shard/route the index by workspace_id.

4. Sharding and hot documents. One shard owning a viral document (a launch-day announcement) gets read-hammered. Fix: shard the document store by hash(doc_id) so writes distribute evenly; absorb read hot-spots in the cache/CDN layer (a hot doc is a hot cache key, not a hot shard - the DB barely sees it). For a genuinely hot write document (rare in async editing), the per-block optimistic model spreads contention across blocks rather than one row.

5. The ACL fan-out in search. A group-permission change could force reindexing every document the group can see. Fix: index group IDs on documents and resolve the user’s group set at query time. Adding a user to a group updates that user’s token set (cheap), not every document. Only a document’s own sharing change triggers a reindex of that one doc.

6. Write-path single points of failure. The document-store primary and the change log are on the critical write path. Fix: run the store with synchronous replication to a standby (failover primary) so an acked write survives a node loss; run Kafka replicated (RF=3) so the change stream is durable. The write is idempotent per (doc_id, version) - a retried save with the same version is a no-op, so at-least-once delivery of the change event is safe.

7. Large documents (a 5,000-block wiki page). Assembling thousands of blocks per read, or shipping the whole block set per edit, is slow. Fix: the block model already means edits ship only changed blocks (patches), not the whole doc. For reads, materialize the current version into a single cached document object so assembly happens on write (rare) not on read (hot); paginate/lazy-load very long documents block-range by block-range.

8. Indexer lag / backlog. A burst of edits (a bulk import) backs up the indexer, so search goes stale. Fix: partition the change log and scale indexer workers horizontally (consumer group); each worker owns a partition. Search staleness degrades gracefully (results a bit behind) rather than failing - and the read path never depends on the index.

Wrap-Up

The trade-offs that define this design:

  • Async over real-time was the whole game. By taking the interviewer’s “not Google Docs” seriously, I dropped OT/CRDT entirely and got to use boring, durable primitives - optimistic concurrency, copy-on-write versioning, an inverted index. The cost is no live collaboration; the payoff is a system built from parts that are individually well-understood and scale independently.
  • Copy-on-write versioning over snapshots. Content-addressed, immutable blocks referenced by cheap version manifests turn 25 TB of duplicated snapshots into ~1.75 TB of shared content, and make diff/restore trivial (a manifest set-diff). We pay a small assembly cost on read (mitigated by materializing the current version into cache) to buy 14x storage savings and instant history.
  • Optimistic block-level concurrency over locks or last-writer-wins. No pessimistic locks (a wiki cannot lock a page for an hour), no silent clobbering. Compare-and-set at the block level auto-merges disjoint edits and surfaces a human-resolved conflict screen only when the same block is edited concurrently - the rare case.
  • Search as an async sidecar over inline query. The inverted index is fed off a change log so writes stay fast and simple, permissions are enforced as an indexed group-filter intersected into every query, and staleness of a few seconds is an accepted, invisible cost.
  • Cache and CDN over database muscle. Reads outnumber writes 100:1 and immutable versions are perfectly cacheable, so the database is sized for the trivial 116 writes/sec while Redis and the CDN absorb the 12-58K reads/sec. The store never sees the hot traffic.

One-line summary: model the document as a tree of content-addressed immutable blocks, version it copy-on-write with cheap manifests, resolve async edits with block-level optimistic concurrency, feed a permission-filtered inverted index off an async change log, and put cache + CDN in front of a modest sharded store - trading real-time collaboration for a versioned, searchable document system that stays cheap and fast at 50M documents.