A document editor sounds trivial until you put two people in the same document. One editor is a text box that saves to a database: type, PUT /document, done. The interviewer nods, then asks the question that detonates the whole design: what happens when Alice and Bob both type into the same sentence at the same time? Alice inserts “Hello " at position 0 while Bob inserts “World” at position 5, their edits cross in flight over the network, and now each person’s copy of the document has to end up identical to the other’s - even though they applied the same two edits in a different order. Get that wrong and the document silently corrupts: characters land in the wrong place, text duplicates, people lose work. That is unacceptable in a system whose entire promise is “we will never lose your writing.”

The whole problem reduces to one hard fact: concurrent edits to shared, ordered state cannot be applied naively, because position indices shift under you. When Bob’s edit says “insert at position 5,” that 5 was computed against a document that no longer exists by the time Alice’s server processes it - Alice already inserted 6 characters at the front. Everything hard here - operational transformation, CRDTs, the real-time sync protocol, presence, versioning - is machinery for making concurrent edits converge to one identical final state on every screen, in real time, without a central lock that would kill the collaborative feel. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Rich-text document editing. Users type, delete, paste, and apply formatting (bold, headings, lists) into a shared document.
  • Real-time collaboration. Multiple users (say up to ~50 active editors in one doc) edit simultaneously and see each other’s changes within a couple hundred milliseconds.
  • Conflict-free convergence. No matter the order edits arrive at each client, every client ends at byte-identical document state. This is the core guarantee.
  • Cursor and selection presence. Each user sees the live cursors and text selections of every other collaborator, labelled and colored, moving as they move.
  • Offline edits and reconnection. A user who loses connection keeps editing locally; on reconnect their edits merge cleanly with everything that happened while they were gone.
  • Version history. The document has a revision timeline: named/auto-saved versions, the ability to see who changed what, and restore to a prior point.
  • Persistence and access. Documents are durably saved, shareable by link, with view/comment/edit permissions.

Explicitly out of scope (say this out loud so you own the scope):

  • The rendering engine / rich-text layout. How the browser paints the document (contenteditable, layout, pagination) is a front-end problem, not the distributed-systems problem.
  • Comments, suggestions mode, and chat. Each is a feature layer on top of the core edit-sync engine; I will note where they hook in but not design them.
  • Auth, sharing ACL internals, and the permissions service. Assume an auth service exists that answers “can user U edit doc D.”
  • Spreadsheets/slides. Same conflict-resolution family, different data model (cells, shapes). I am designing the text document.

The one decision that drives everything: this is a low-latency, symmetric read/write, per-document stateful system where correctness means convergence, not throughput. Unlike a feed (read-heavy, stale is fine) or chat (route a message to a person), here many writers mutate one shared ordered structure concurrently and every copy must reconcile to the same result. That convergence requirement is the whole problem.

Non-Functional Requirements (NFR)

  • Scale: ~100M daily active users, ~1B documents total, ~50M documents edited per day. A single hot document may have up to ~50 concurrent active editors (Google caps live editors around 100; most docs have 1-3). Peak ~1M concurrently open editing sessions.
  • Latency: local echo must be instant (0ms - the client applies its own keystroke immediately, never waiting for the server). Remote edits should appear on other screens within p99 under 200ms. Cursor updates within ~100ms.
  • Availability: 99.99% on the edit path. A user typing must never be blocked by the backend; the editor stays usable (queueing edits locally) even during a brief server blip.
  • Consistency: strong eventual consistency - all replicas that have received the same set of operations converge to the identical document, regardless of delivery order or duplication. Within a session the user sees their own edits immediately (read-your-writes); across users it is eventual but fast.
  • Durability: an acknowledged edit is never lost. Once the server acks an operation it must survive crashes. Version history is immutable and durable.
  • Correctness over freshness: it is acceptable for a remote edit to be 200ms late; it is never acceptable for two clients to disagree on the final text.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Edit operations (writes):

A person types roughly 5 characters/sec while actively writing (200 WPM bursts, but averaged with reading/thinking it is far less). Assume an active editor emits ~2 operations/sec while in a burst. But most open documents are idle at any instant. Say of 1M concurrently open sessions, ~10% are actively typing at once:

1,000,000 open sessions * 10% active * 2 ops/sec
= 100,000 * 2
≈ 200,000 edit ops/sec (steady)
Peak (3x) ≈ 600,000 ops/sec

Fan-out (the read side is a push):

Each op must be broadcast to every other editor in its document. Average doc has ~2 editors, but the hot ones dominate op volume. Assume average fan-out ~3:

200,000 ops/sec * ~3 recipients ≈ 600,000 op-pushes/sec (steady)
Peak ≈ 1.8M pushes/sec

So like chat, reads and writes are the same order of magnitude - there is no read-cache shortcut; every op must actually move to every peer. The asymmetry lever a feed enjoys does not exist.

Storage:

Two things to store: the current document state, and the operation log (the source of truth for history and convergence). An op is small:

op_id         16 bytes
doc_id         8 bytes
site_id        8 bytes   (which client)
lamport_ts     8 bytes   (logical clock)
type           1 byte    (insert/delete/format)
position/ref  ~24 bytes  (index or CRDT id)
payload       ~8 bytes   (a char or short run + attributes)
------------------------------------------------
≈ 80 bytes/op  (round to ~100 with overhead)

Ops per day and the log they generate:

200,000 ops/sec * 86,400 sec ≈ 17.3B ops/day
17.3B * 100 bytes ≈ 1.7 TB/day of raw op log

That is a lot to keep forever per doc, so the op log is compacted: periodically we snapshot the document state and garbage-collect the ops before the snapshot (history keeps periodic checkpoints, not every keystroke forever). Document bodies themselves are small:

1B documents * ~40 KB average (a typical doc, text + formatting)
≈ 40 PB of document state at rest, before replication.
With RF=3 ≈ 120 PB. Cold docs tier to cheaper storage.

Concurrent connections:

1M concurrent editing sessions over WebSocket.
At ~200K connections per tuned gateway box: 1M / 200K ≈ 5-10 gateway servers
just to hold the sockets. Small - the connection count here is far below chat's.

Bandwidth:

Peak 1.8M op-pushes/sec * ~100 bytes ≈ 180 MB/sec of op traffic.
Plus cursor/presence updates (high count, tiny payload).
Trivially handled across a handful of gateways.

The takeaways: op throughput (~600K/sec peak writes, ~1.8M/sec pushes) is the driver, not connection count. Storage is dominated by the op log, which must be compacted with snapshots. And the truly hard part is not any of these numbers - it is that every one of those ops must be transformed or merged so that all replicas converge. That algorithm, not the scale, is the interview.

High-Level Design (HLD)

The architecture is a per-document collaboration session: for each actively-edited document there is one authoritative in-memory session (owned by a specific server) that receives every op, orders them, transforms/merges them, persists them to the op log, and broadcasts the result to all connected clients. Clients hold a WebSocket to a stateless gateway; the gateway routes them to the document’s session server. A document-to-session registry maps doc_id -> session server so all editors of one doc land on the same authoritative sequencer.

   ┌────────────┐   ┌────────────┐   ┌────────────┐
   │ Client A   │   │ Client B   │   │ Client C   │   (browsers, local doc replica
   │ (local doc │   │ (local doc │   │ (local doc │    + edit queue each)
   │  replica)  │   │  replica)  │   │  replica)  │
   └─────┬──────┘   └─────┬──────┘   └─────┬──────┘
         │  persistent WSS │                │
   ┌─────▼─────────────────▼────────────────▼──────┐
   │        Load Balancer (L4)  +  Gateways         │  (stateless, hold sockets)
   └─────┬─────────────────┬────────────────┬───────┘
         │        route by doc_id via registry       │
   ┌─────▼───────────────────────────────────────────▼─────┐
   │              Doc-Session Registry (Redis)               │
   │              doc_id -> session_server_id                │
   └─────────────────────────┬──────────────────────────────┘
                             │
                 ┌───────────▼────────────┐
                 │   Collaboration Server   │  one authoritative
                 │   (per-doc session)      │  session per active doc:
                 │  - receive op            │  orders, transforms/merges,
                 │  - order (rev counter)   │  broadcasts, persists
                 │  - transform (OT) /      │
                 │    integrate (CRDT)      │
                 │  - broadcast to peers    │
                 └───┬──────────────┬───────┘
          append op  │              │  snapshot / read
       ┌─────────────▼──┐    ┌──────▼───────────────┐
       │  Op Log Store   │    │  Document Store       │
       │  (append-only,  │    │  (current state +     │
       │   per doc, the  │    │   periodic snapshots, │
       │   source of     │    │   sharded by doc_id)  │
       │   truth)        │    │                       │
       └────────┬────────┘    └──────────────────────┘
                │ async
        ┌───────▼─────────┐
        │ Version History  │  (named/auto versions, restore points,
        │  Service         │   built from snapshots + op ranges)
        └──────────────────┘

Edit flow (A types a character, others online):

  1. A presses a key. The client immediately applies it to its local replica (instant local echo - never waits for the server) and appends the op to a local outgoing queue tagged with A’s current known server revision.
  2. The client sends the op over its WebSocket to the gateway, which forwards it to the document’s Collaboration Server (found via the registry).
  3. The session server transforms the op against any ops that were committed after A’s known revision (the ops A had not seen yet), assigns it the next global revision number, and appends it to the op log (durable - the source of truth).
  4. The server acks A with the new revision number, and broadcasts the transformed op to B and C.
  5. B and C each transform the incoming op against their own local un-acked ops, apply it to their replica, and advance their known revision. All three replicas now match.

Edit flow (A is offline, then reconnects):

  1. A keeps editing; ops pile up in A’s local outgoing queue against A’s last-known revision (say revision 40).
  2. On reconnect, A sends its queued ops plus “I last saw revision 40.” The server has advanced to revision 57 meanwhile. It transforms A’s queued ops against revisions 41-57, commits them, and sends A the missed ops 41-57 (transformed against A’s pending ops). Both sides converge.

The key insight: one authoritative per-document sequencer turns “N clients editing concurrently” into “a single agreed order of operations,” and the convergence algorithm (OT or CRDT) makes every client’s out-of-order local view reconcile to that order. The server is the tiebreaker; the algorithm is the reconciliation.

Component Deep Dive

The hard parts, naive-first then evolved: (1) conflict resolution - OT vs CRDT, the actual heart of the problem; (2) the real-time sync protocol over WebSocket; (3) cursor and selection presence; (4) versioning and history.

1. Conflict resolution: why naive fails, then OT, then CRDT

Naive approach: last-write-wins on the whole document. Each client sends the full document text on save; the server keeps whichever arrived last.

Where it breaks:

  • You lose entire edits. Alice adds a paragraph at the top, Bob fixes a typo at the bottom, both save. Whoever saves second overwrites the other’s change completely. In a real-time editor where saves happen every keystroke, this destroys work constantly.
  • It does not merge; it clobbers. The entire promise of collaboration - both edits surviving - is gone.

Second naive approach: send positional operations, apply them in arrival order. Instead of the whole doc, send “insert ‘X’ at index i” / “delete index i”. Apply each op to the shared doc as it arrives.

Where it breaks - the central problem of this whole design:

Positions are computed against the document the client saw, but by the time the op reaches everyone, the document has changed. Concretely, start with "ABC". Alice and Bob act concurrently, each against "ABC":

Alice: insert 'X' at index 0   -> she sees "XABC"
Bob:   insert 'Y' at index 2   -> he sees  "ABYC"

Now the ops cross. Server applies Alice's then Bob's:
  "ABC" --insert X@0--> "XABC" --insert Y@2--> "XAYBC"   ❌ Y landed before B
Client that applied Bob's then Alice's:
  "ABC" --insert Y@2--> "ABYC" --insert X@0--> "XABYC"   ✓ Y after B

Two clients, same two ops, DIFFERENT final text. Divergence. Corruption.

The indices are stale. Bob’s “index 2” meant “after B,” but once Alice inserted at 0, “after B” is now index 3. This is exactly the problem OT and CRDTs exist to solve.

Evolution A: Operational Transformation (OT). OT keeps positional ops but adds a transform function that rewrites an incoming op’s positions to account for concurrent ops it did not see. Before applying Bob’s op on a replica that already applied Alice’s, transform it:

transform(op_bob, op_alice):
  if op_alice is insert at index i_a, and i_a <= op_bob.index:
      op_bob.index += length(op_alice.insert)   # shift right
  return op_bob

Bob's "insert Y@2" transformed against "insert X@0" becomes "insert Y@3".
Now:  "XABC" --insert Y@3--> "XABYC"   ✓  matches the other client.

Every replica applies the same logical operations, each transformed against whatever it had not yet seen, and they converge. The server is the ordering authority: it assigns each op a revision number, and every client transforms its pending local ops against the server’s committed sequence (Google Docs uses a central-server OT variant; Jupyter, Etherpad, and Google Wave popularized this). This is the classic and battle-tested approach.

Where OT is hard:

  • The transform function must be correct for every pair of op types (insert/insert, insert/delete, delete/delete, and formatting), and satisfy the convergence property TP1 (and TP2 for peer-to-peer). Writing and proving these transforms is notoriously fiddly - the Google Wave team wrote papers on the edge cases.
  • It leans on a central server. True decentralized OT (TP2) is very hard; in practice OT wants one authoritative sequencer, which we have.

Evolution B: CRDTs (Conflict-free Replicated Data Types). CRDTs sidestep transformation entirely by changing the data model. Instead of integer indices, every character gets a globally unique, immutable, densely-orderable identifier at insert time. A character is inserted “between id P and id Q,” not “at index 5.” Because the id never changes and encodes position by a total order, there is nothing to transform - you just insert it in id-order and deletes are tombstones.

Sequence CRDT (e.g. RGA / Logoot / Fugue) idea:
  Each char = (unique_id, value). unique_id sorts globally.
  Insert 'X' between existing ids  a  and  b  -> mint an id  m  with  a < m < b.
  "insert between" is commutative: apply in any order, same result.

  Alice inserts X with id  0.5|siteA   (between start and 'A')
  Bob   inserts Y with id  2.7|siteB   (between 'B' and 'C')
  These ids are independent and totally ordered, so BOTH replicas,
  applying both inserts in either order, sort to the same sequence:
      X(0.5) A(1) B(2) Y(2.7) C(3)   ->  "XABYC"   ✓ no transform needed

CRDT trade-offs:

  • No central server required for correctness. Ops are commutative, associative, and idempotent, so any delivery order (even duplicates) converges. Great for offline-heavy and peer-to-peer (this is why Figma, Apple Notes, and Automerge/Yjs-based tools use CRDTs).
  • Metadata overhead. Every character carries an id; tombstones for deletes accumulate. Naive CRDTs can bloat memory to many times the visible text. Modern CRDTs (Yjs, Fugue, diamond-types) mitigate with block/run encoding and tombstone GC, getting close to OT’s footprint.
  • Interleaving anomalies. Some early CRDTs interleave concurrently-inserted runs oddly; newer ones (Fugue) fix this.

The decision:

Dimension OT CRDT
Data model integer positions + transform fn unique per-char ids, commutative merge
Central server wants one (authoritative sequencer) not required (P2P-capable)
Correctness effort fiddly transform proofs (TP1/TP2) commutativity by construction
Memory overhead low (plain text + ops) higher (ids + tombstones), mitigable
Offline / P2P awkward without server natural
Who uses it Google Docs, Etherpad Figma, Yjs, Automerge, Apple Notes

For a Google-Docs-style system with an always-available central backend, I choose server-authoritative OT as the primary engine: it is memory-lean, the central sequencer we already need makes the transform tractable, and it is exactly what Google ships. I would note that a modern greenfield build often picks a mature CRDT library (Yjs) instead, trading a little metadata overhead for far simpler correctness and free offline/P2P - a legitimate and increasingly common choice. The interview answer is: both converge; OT centralizes the hard work in a transform function against a server order, CRDT bakes convergence into the data type. Pick OT with a central server for lean memory, CRDT when you need decentralization or heavy offline.

2. Real-time sync protocol over WebSocket

Naive approach: HTTP POST each edit, GET to poll for others’ edits.

Where it breaks:

  • Latency and load are unwinnable together, same as any polling design: poll slowly and remote edits lag seconds (collaboration feels dead); poll fast and 1M sessions hammer the backend with mostly-empty requests. And HTTP request/response cannot push - the server knows an op exists but cannot tell a client until it next asks.
  • No ordering channel. Edits need a running revision number handshake; stateless HTTP has nowhere to hold it cheaply.

The answer: a WebSocket carrying a revision-numbered op protocol. One persistent full-duplex connection per editing session. The protocol is built around a client-tracked “known server revision” and the OT invariant that a client may have at most a bounded set of un-acked local ops in flight.

The canonical client-server OT sync loop (Google Docs / ProseMirror model):

Client state:
  - local replica of the doc
  - `synced_rev` : the highest server revision the client has incorporated
  - `pending`    : ops applied locally but not yet acked by the server
  - `sending`    : the single op-batch currently in flight (in-order, one at a time)

On local edit e:
  apply e to local replica immediately        # instant echo
  append e to `pending`
  if nothing in flight: send next batch

On sending a batch:
  send { type: OP, base_rev: synced_rev, ops: [...] }   # "these are based on rev N"

Server, on receiving { base_rev: N, ops }:
  transform `ops` against every committed op in (N, current_rev]   # OT catch-up
  assign them revisions current_rev+1 ...
  append to op log (durable), update current_rev
  ACK the sender: { type: ACK, rev: new_rev }
  BROADCAST to other clients: { type: OP, rev, ops(transformed), site }

Client, on ACK:
  move acked ops out of `pending`; advance `synced_rev`; send next batch if any

Client, on receiving a remote OP at rev R:
  transform incoming ops against my `pending` (ops the server had not seen from me)
  apply to local replica
  synced_rev = R

The rule that keeps it sane: a client sends one in-flight batch at a time and waits for the ack before sending the next, so the server never has to transform against an unbounded client backlog, and the client always knows exactly which of its ops are un-acked. Local echo is instant because the client applies first and reconciles later; if the server’s transform changes the op, the client’s incoming-remote path fixes up the local replica.

Connection routing. The gateway is a thin, stateless WebSocket terminator. On open, it reads doc_id from the session, looks up the Doc-Session Registry (doc_id -> collaboration server), and pins this socket’s op traffic to that one authoritative server (spinning the session up if the doc is newly opened). Every editor of a given doc thus funnels to the same sequencer - which is what makes a single global revision order possible. The registry entry has a heartbeat TTL so a crashed session server’s docs get re-homed on the next client reconnect.

Reconnection and offline. The client persists synced_rev and its pending queue locally (IndexedDB). On reconnect it sends base_rev = synced_rev plus all pending ops; the server transforms them against everything committed since, commits them, and streams back the missed ops. Because the op log is the durable source of truth, a client that was offline for an hour just replays from its revision - bounded, incremental, no full re-download. This is the same machinery as the online path, just with a bigger catch-up gap.

3. Cursor and selection presence

Naive approach: store each user’s cursor position in the doc row, broadcast on every move via the same durable op pipeline.

Where it breaks:

  • Cursor moves are a firehose - every arrow key, every mouse drag emits a position. Routing that through the durable op log (persist + version) is absurd; you would 10x your write volume storing data nobody ever needs again.
  • Positions go stale under edits. A cursor at “index 12” is meaningless the instant someone inserts text before it - it must ride the same transform as ops, or it jumps to the wrong place.

The fix: ephemeral, throttled presence on a side channel, with positions expressed against the OT/CRDT model, not raw indices.

  • Never persist cursors. They are transient signaling, exactly like typing indicators in chat. They flow over the WebSocket as PRESENCE frames, are held in the session server’s in-memory presence map, and are dropped when the user disconnects (TTL-based, self-healing - a crashed client’s cursor vanishes on timeout).
  • Throttle/coalesce. The client sends at most ~10-20 cursor updates/sec (coalesce rapid moves), so a fast typist does not flood peers. Only the latest position matters; drop intermediate ones.
  • Express position transform-stably. In OT, a cursor is a position that gets transformed by the same functions as ops: when a remote insert lands before a cursor, shift the cursor right. In CRDT, even cleaner - anchor the cursor to a character id (“just after id X”), so it moves correctly no matter what edits happen around it, with no transform needed.
  • Fan-out is O(editors in this doc), a tiny set (~50 max), and only to clients currently in that doc. The session server already has all their sockets, so it broadcasts directly.
// client -> server, throttled
{ "type": "PRESENCE", "doc_id": "d9", "cursor": {"anchor": "id:7.3|siteA",
  "head": "id:9.1|siteA"}, "user": "A", "color": "#4285F4" }

// server -> other clients (in-memory relay, never logged)
{ "type": "PRESENCE", "user": "A", "color": "#4285F4",
  "anchor": "id:7.3|siteA", "head": "id:9.1|siteA" }

Selections are just cursors with an anchor and a head (both anchored to character ids/positions), rendered as a colored highlight with the user’s name label. The whole subsystem stays entirely off the durable path, which keeps the storage layer seeing only real edits.

4. Versioning and history

Naive approach: snapshot the full document on every save.

Where it breaks:

  • Storage explodes. Saving a 40KB doc on every keystroke-batch is gigabytes per active doc per day of near-duplicate blobs. And you still cannot answer “who changed this word” without a diff.
  • No fine-grained restore or attribution. Full snapshots lose the operational granularity that makes “revert just this change” or “show me Bob’s edits” possible.

The fix: the op log IS the history; snapshots are periodic checkpoints for fast reads.

  • Append-only op log per document is the source of truth. Every committed op (with its revision, site/user, and Lamport/logical timestamp) is durably appended. The full document at any revision R is reconstructable by replaying ops 1..R - so history is inherent, not a separate store. Attribution (“who wrote this”) falls out because each op carries its author.
  • Periodic snapshots for performance and compaction. Replaying millions of ops to open a doc is slow, so every N ops (or T seconds) the session server writes a snapshot: the materialized document state at that revision. Opening a doc = load nearest snapshot + replay the tail of ops after it. Old ops before a snapshot can be compacted/GC’d from hot storage (kept in cold storage or summarized), bounding the log.
  • Named versions and auto-versions. Google Docs groups edits into “versions” (a burst of edits by a user in a time window becomes one history entry). We build these lazily: the Version History Service reads snapshot boundaries + op ranges and presents human-meaningful checkpoints, coalescing tiny edits. A “named version” is just a labelled, pinned snapshot that is never GC’d.
  • Restore = a new op. Reverting to version V does not rewrite history; it computes a diff from current state back to V’s state and applies it as new forward ops. History stays immutable and append-only, and the restore itself is auditable.
Reconstruct doc at revision R:
   snapshot_S = latest snapshot with rev <= R
   doc = snapshot_S.state
   for op in oplog where snapshot_S.rev < op.rev <= R:  apply(doc, op)
   return doc

Compaction:
   after writing snapshot at rev K:  archive/GC ops with rev < K (keep pinned versions)

This gives unlimited history conceptually, bounded storage practically, per-op attribution, and cheap doc-open - all from one append-only log plus periodic checkpoints.

API Design & Data Schema

Most traffic is WebSocket frames (ops, acks, presence); REST handles bootstrap, history, and document CRUD.

REST (bootstrap, CRUD, history)

POST /api/v1/docs
  -> create a document.  Body: { "title": "..." }
  Response 201: { "doc_id": "d9", "rev": 0 }

GET  /api/v1/docs/{doc_id}
  -> open: returns nearest snapshot + tail ops + the WSS endpoint to dial.
  Response 200: { "snapshot": {...}, "rev": 812, "tail_ops": [...],
                  "ws_url": "wss://collab-edge.docs.net/ws?doc=d9", "token": "..." }

GET  /api/v1/docs/{doc_id}/history
  -> list versions (coalesced checkpoints).
  Response 200: { "versions": [ {"rev": 800, "label": "Auto", "by": "A",
                  "at": "..."}, ... ] }

GET  /api/v1/docs/{doc_id}/at/{rev}
  -> materialized document at a past revision (snapshot + replay).

POST /api/v1/docs/{doc_id}/restore   { "to_rev": 800 }
  -> apply a revert as new forward ops. Response: { "new_rev": 815 }

WebSocket frames (the real-time path)

// client -> server: submit an edit batch based on a known revision
{ "type": "OP", "doc_id": "d9", "base_rev": 812,
  "ops": [ {"t":"ins","pos":42,"chars":"Hello"},
           {"t":"del","pos":10,"len":3},
           {"t":"fmt","from":0,"to":5,"attr":{"bold":true}} ] }

// server -> sender: acknowledge with the assigned revision
{ "type": "ACK", "base_rev": 812, "rev": 815 }

// server -> other clients: a committed, transformed op (server-initiated push)
{ "type": "OP", "rev": 815, "site": "A",
  "ops": [ {"t":"ins","pos":47,"chars":"Hello"} ] }

// presence (ephemeral, throttled, never persisted)
{ "type": "PRESENCE", "user": "A", "color": "#4285F4",
  "anchor": "id:7.3|siteA", "head": "id:9.1|siteA" }

// health
{ "type": "HEARTBEAT" }   // client, every ~20s; refreshes session TTL

Note the op frame carries base_rev: the server transforms the batch against everything committed after that revision. History reads are cursor-based on revision number (stable, monotonic), never offset-based.

Data store choices

1. Op Log - append-only distributed log (Kafka-style log, or a wide-column store keyed by doc). The access pattern is “append an op” and “range-scan ops for one doc by revision,” at hundreds of thousands of writes/sec, no joins, no cross-doc transactions. That is a sequential append log, sharded by doc_id.

Table/Topic: op_log
  doc_id       STRING   PARTITION KEY            -- all ops of a doc co-located, ordered
  rev          BIGINT   CLUSTERING KEY ASC       -- server-assigned global order per doc
  op_id        UUID                              -- idempotency / dedup key
  site_id      STRING                            -- which client/site
  user_id      BIGINT                            -- for attribution
  lamport_ts   BIGINT                            -- logical clock (causal ordering)
  type         TINYINT                           -- ins / del / fmt
  payload      BLOB                              -- chars/len/attrs (or CRDT ids)
  created_at   TIMESTAMP
  Shard by: doc_id   -> one doc's op history is a single ordered partition
  Read: "ops for d9 where rev > 800" = single-partition range scan (tail replay)

2. Document Store - wide-column / blob (Bigtable / Cassandra / S3 for snapshots), keyed by doc. Holds the latest materialized state and periodic snapshots.

Table: doc_snapshots
  doc_id       STRING   PARTITION KEY
  rev          BIGINT   CLUSTERING KEY DESC      -- latest snapshot first
  state        BLOB                              -- materialized doc at this rev
  pinned       BOOL                              -- named version -> never GC
  created_at   TIMESTAMP
  Read: "latest snapshot for d9" = first row of the partition
  GC: delete unpinned snapshots + ops below the newest checkpoint

3. Doc-Session Registry + Presence - Redis (in-memory, replicated). sess:{doc_id} -> collaboration_server_id with heartbeat TTL (routes all a doc’s editors to one sequencer), and in-memory presence maps on the session server itself (cursors never leave memory). Small, hot, ephemeral, rebuildable - the perfect in-memory fit.

Why NoSQL/log-structured over SQL here: the workload is per-document partitioned, append-heavy, transaction-free across docs, and needs horizontal scale to billions of docs and hundreds of thousands of appends/sec, plus an in-memory routing tier. The one place strong consistency matters - the per-document revision order - is provided not by the database but by the single authoritative session server that sequences ops for that doc. That is the elegant part: we get the strong ordering we need from application-level single-writer-per-doc, so the storage layer can be a scalable append log rather than an ACID relational DB whose single-writer ceiling we would otherwise crash into.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. A hot document with many concurrent editors (breaks first). One doc’s session server must serialize every op for that doc (single-writer-per-doc is what gives us the global revision order). A 50-editor doc in a heavy burst funnels all ops through one server. Fix: keep the per-doc session in memory and single-threaded per doc so ordering is trivial and lock-free; the op payloads are tiny and OT transforms are cheap, so one core handles thousands of ops/sec/doc easily. Cap live editors (~50-100, as Google does). The single-writer is a feature - it is the sequencer - not a bug; we only need it to keep up with one document’s human typists, which it trivially does.

2. Millions of documents, each needing a session. You cannot keep 1B docs live. Fix: sessions are spun up on first open and torn down when the last editor leaves (idle eviction). Only actively-edited docs (~1M) hold a live session; the rest are just data at rest. The Doc-Session Registry maps active docs to their servers; a doc with no session is loaded from snapshot on next open.

3. Op write throughput (~600K/sec peak). Fix: shard the op log by doc_id so writes spread across the partition space; no global counter (revisions are per-doc). Batching helps - a client sends an op batch, not one frame per character. Replication factor 3 for durability; persist-then-ack so a committed op survives node loss.

4. Reconstructing a doc from a long op history (slow open). Fix: periodic snapshots so open = nearest snapshot + short tail replay, and compaction to GC pre-snapshot ops. Bounds both open latency and log storage.

5. Presence / cursor firehose. Fix: keep cursors entirely off the durable path - in-memory relay only, throttled to ~10-20 updates/sec/client, TTL-expired on disconnect. Fan-out is only to the tiny set of editors in that one doc.

6. Session server crash = the doc’s live session is lost. All editors of that doc disconnect at once. Fix: because every committed op is in the durable op log, recovery is clean: clients auto-reconnect (jittered backoff), the registry re-homes the doc to a new session server, which rebuilds in-memory state from the latest snapshot + tail ops, and each client replays its un-acked pending from its synced_rev. Nothing acked is lost; at worst editors re-sync a few hundred ms of work. Stale registry entries expire by TTL.

7. Shard keys, stated plainly. Op Log -> doc_id (a doc’s ordered history is one partition; enables single-partition tail replay and a per-doc revision counter). Document/Snapshot Store -> doc_id. Registry/Presence -> doc_id. Sharding by user_id or by time would scatter one document’s ops across partitions and destroy the single-partition ordered replay that the whole convergence model depends on - doc_id is the only correct key because the document is the unit of consistency.

8. Duplicate / replayed ops on reconnect. Fix: each op carries a unique op_id; the session server dedups by (site_id, op_id) so a client that resends after a flaky ack does not double-apply. Combined with OT’s transform-against-committed, this makes reconnection idempotent.

9. Single points of failure. Gateways are stateless and interchangeable. Session servers are recoverable from the durable log (state is in the op log, not the box). Op log, document store, and registry are all replicated. No single box whose loss loses data - at worst a doc’s editors reconnect and its session rebuilds from storage.

10. Multi-region latency for globally-distributed co-editors. Fix: a document has a home region that owns its session and op log (co-editors route there so the single sequencer stays authoritative). Two editors in that region get sub-100ms sync; a distant editor pays cross-region RTT but still converges correctly - correctness never depends on latency, only the feel does. Snapshots replicate cross-region for disaster recovery and fast read-only opens. If a doc’s editors are permanently split across regions, you either migrate its home region to the majority or accept the WAN hop; you do not split the sequencer, because two sequencers means two revision orders means divergence.

Wrap-Up

The trade-offs that define this design:

  • Convergence algorithm over locking. We never lock the document (that would kill real-time collaboration); instead we let everyone edit freely and reconcile with OT (or a CRDT). We chose server-authoritative OT for lean memory and because the central sequencer we already run makes the transform tractable - accepting fiddly transform correctness as the cost, and noting a mature CRDT (Yjs) is the modern alternative when decentralization or heavy offline matters.
  • Single authoritative sequencer per document. Strong per-doc revision ordering comes from application-level single-writer-per-doc, not from the database - which lets storage be a scalable append log instead of an ACID DB we would outgrow. The trade is that a hot doc funnels through one server, which is fine because it only has to keep up with one document’s humans.
  • Instant local echo, reconcile later. The client applies its own keystroke immediately and transforms server truth into its local view afterward - trading a moment of local-vs-server divergence for a zero-latency typing feel that never blocks on the network.
  • The op log is the source of truth; snapshots are just fast-read checkpoints. History, attribution, restore, and crash recovery all fall out of one append-only per-doc log, with periodic snapshots and compaction bounding storage and open latency.
  • Ephemeral presence off the durable path. Cursors and selections are throttled, in-memory-only, TTL-expiring, and anchored to positions/ids so they transform correctly - keeping the firehose entirely away from storage.

One-line summary: a per-document collaboration session where a single authoritative server sequences every edit into a global revision order, clients apply their own keystrokes instantly and reconcile via operational transformation so all replicas converge to identical text, edits stream over WebSocket with a revision-numbered ack protocol, cursors ride an ephemeral in-memory presence channel, and an append-only op log with periodic snapshots delivers durable versioning, attribution, and clean crash recovery - all sharded by document because the document is the unit of consistency.