People hear “build Slack” and reach for a chat toy: a messages table, a WebSocket, insert on send, push to whoever is connected. That works for a demo and falls apart the instant you write down the real numbers. The largest org has 100K members. A single busy channel in that org - #general, #incidents, #announcements - can have tens of thousands of people watching it live. When someone posts, who do you deliver to, and how? Do you write a copy of that message into 50,000 inboxes (fan-out on write) and drown on a viral post, or do you store it once and make every reader come find it (fan-out on read) and drown on the read side instead? That single question - how a message gets from one sender to N readers - is the spine of the whole design, and Slack’s answer is different from Twitter’s or Facebook’s because Slack’s unit of delivery is the channel, not the follower graph.

Then it gets harder. Messages are not fire-and-forget; they persist forever and are searchable, so search across billions of messages is a first-class feature, not a bolt-on. Threads mean a message can have its own sub-conversation that must not spam the main channel. DMs are just channels with two members and stricter privacy. Files are large blobs that must not touch the message path. And all of it has to feel instant for 10M people connected at once.

Let me do it properly.

Functional Requirements (FR)

In scope:

  • Channels (public and private). A workspace has channels; users join them; a message posted to a channel is delivered to every member, in order, in real time, and is stored durably so someone who was offline sees it on reconnect. Public channels are discoverable and joinable by anyone in the org; private channels are invite-only and invisible to non-members.
  • Direct messages (DMs) and group DMs. A 1:1 or small-group conversation. Mechanically these are just channels with a fixed, small membership and no discoverability - so we build them on the same primitive as channels, not as a separate system.
  • Threads. Any message can be the root of a thread. Replies attach to that root and are shown in a side panel, not the main channel timeline, unless the replier chooses “also send to channel.” Threads must not flood the parent channel.
  • Presence and typing indicators. Who is online, who is typing. Ephemeral, best-effort, high-volume, low-value-per-event - handled on a cheap side channel, never persisted.
  • Search across all messages. Full-text search over every message the user is allowed to see (their channels, their DMs), with filters (in:channel, from:user, before/after date, has:file). Results ranked, permission-scoped, fast.
  • File sharing. Upload a file, share it into a channel or DM, others download it. Files are large binary blobs; the message just references them.
  • Read state and unread counts. Per user, per channel: what is the last message I have seen, how many unread, how many mentions of me.
  • Mentions and notifications. @user, @channel, @here drive push notifications and unread badges.

Explicitly out of scope (state this to control scope):

  • Voice/video huddles and calls (a separate real-time media system - WebRTC/SFU - not the messaging path).
  • The full org/identity/SSO stack. We assume an authenticated request carries a resolved user_id and org_id.
  • Apps, bots, slash commands, and the integration platform. They post through the same message API, so they are just clients to us.
  • Message editing/deletion history and compliance export (real, but a data-retention layer on top of the store we design).
  • Emoji reactions counting - mentioned briefly, not deep-dived.

The decision that drives everything: a message is delivered to a channel’s members, and channel membership is bounded and known. Unlike a social feed where a post fans out to an unbounded, ever-changing follower set, a Slack message goes to a specific channel whose member list we can enumerate. That bound is what lets us make a clean fan-out choice per channel instead of one global policy.

Non-Functional Requirements (NFR)

  • Scale: 100K orgs. Largest org 100K members. 10M concurrent users globally (holding live WebSocket connections). Assume a heavy write mix for a productivity tool during business hours across time zones.
  • Latency: message send-to-deliver must feel instant - p99 under ~200ms from sender pressing enter to the message appearing on other online members’ screens. Search p99 under ~500ms. Channel history load on open under ~300ms.
  • Availability: 99.99%+. Slack being down stops a company from working. The connection layer especially must degrade gracefully - a gateway node dying should reconnect its clients elsewhere within a second or two.
  • Consistency: strong ordering within a single channel is required - everyone must see the messages of one channel in the same order. Across channels, no ordering guarantee is needed. Read-your-writes for the sender (you always see your own message immediately). Presence and typing are best-effort/eventual. This “strong per channel, independent across channels” property is the sharding gift - the channel is the natural unit of consistency.
  • Durability: messages are permanent and must never be lost once acknowledged. A send is only ACKed after it is durably persisted and sequenced. Files are equally durable (object storage with replication). Presence/typing are throwaway.

Back-of-the-Envelope Estimation (BoE)

Let me ground it in numbers.

Users and connections:

Orgs:                    100,000
Largest org:             100,000 members
Total registered users:  ~50,000,000  (assume ~500 avg org size)
Peak concurrent users:   10,000,000   (holding live WebSocket connections)

Message write rate:

Assume an active user sends ~40 messages / 8-hour workday
  = 40 / 28,800s ≈ 0.0014 msg/user/sec
Of 10M concurrent users, assume ~30% are "active senders" at peak = 3M
  3,000,000 * 0.0014 ≈ 4,200 msg/sec baseline
Budget spikes (start of day, incidents, big launches) at 5-10x:
  peak writes ≈ 40,000 messages/sec

Message read / delivery rate (the real firehose):

Each message posted to a channel of M members must reach M readers.
Average channel size is small (~10), but the tail is enormous:
  a 50,000-member #general post = one write, 50,000 deliveries.

Model average delivery fan-out ~ 100 recipients/message (small channels
dominate volume; huge channels dominate per-event cost).
  40,000 msg/sec * 100 recipients ≈ 4,000,000 deliveries/sec at peak

Only deliveries to *currently connected* users go over WebSocket
(~say 40% online for a given message's members) ≈ 1.6M live pushes/sec.
The rest are picked up on reconnect / history fetch.

That 1.6M live pushes/sec, driven by fan-out, is the load that decides the delivery architecture.

Storage:

Per message record:
  msg_id (8B) + channel_id (8B) + user_id (8B) + ts (8B)
  + thread_root_id (8B) + body (avg ~200 bytes) + metadata (~50B)
  ≈ 300 bytes stored, call it ~500 bytes with indexes/overhead.

Messages/day: 40,000/sec is peak; assume ~8,000/sec daily average
  8,000 * 86,400 ≈ 700M messages/day
  700M * 500 bytes ≈ 350 GB/day of message data
  Per year ≈ ~125 TB/year, and it is never deleted -> multi-PB over time.

Messages are permanent and grow without bound. That kills any “keep it all in one hot database” idea and forces sharding plus tiered storage (hot recent, cold archive).

Search index:

Full-text index is typically 30-100% of raw text size.
Index only the body text (~200 bytes/msg):
  700M msgs/day * ~200 bytes ≈ 140 GB/day of text
  Index it -> ~100-150 GB/day of index growth
  Over a year -> tens of TB of index, sharded per org.

File storage:

Say 5% of messages carry a file, avg 1 MB:
  700M * 0.05 * 1 MB ≈ 35 TB/day of file blobs
Files dwarf message text in bytes -> object storage (S3-class),
never the message database, and served via CDN.

Connection memory:

10M WebSocket connections. Each connection holds ~10-50 KB of
kernel + app state (buffers, subscription list).
  10M * ~30 KB ≈ 300 GB of connection state across the gateway fleet.
A single box holds ~100K-500K connections -> 20-100 gateway nodes.

The numbers say three separate scaling problems: millions of live connections (gateway fleet), fan-out delivery at millions/sec (routing + inboxes), and petabytes of permanent, searchable messages (sharded store + search cluster). We design each explicitly.

High-Level Design (HLD)

Slack is not one service; it is a set of loosely coupled subsystems joined by a message bus. A client holds a persistent WebSocket to a gateway for real-time push, but sends messages over plain HTTP to a stateless API tier. The API persists and sequences the message, then publishes it to a fan-out/router layer that figures out who should get it and pushes to their gateways. Search, files, and presence are separate planes.

   ┌─────────────────────────────────────────────────────────────┐
   │  Clients (desktop / web / mobile)                            │
   │   - HTTP POST to send a message                              │
   │   - persistent WebSocket to receive pushes                   │
   └───────┬───────────────────────────────────┬─────────────────┘
           │ (A) send: HTTP POST /messages       │ (B) receive: WSS
           ▼                                     ▼
   ┌───────────────────┐                 ┌───────────────────────┐
   │  API / Message     │                 │  WebSocket Gateway     │
   │  Service (stateless)│                 │  fleet (stateful:      │
   │  - authz on channel │                 │  holds live conns,     │
   │  - assign seq no    │                 │  maps user->conn)      │
   │  - persist message  │                 └──────────▲────────────┘
   └───┬─────────┬───────┘                            │ push events
       │ write   │ publish                            │
       ▼         ▼                                    │
 ┌──────────┐  ┌──────────────────┐        ┌──────────┴───────────┐
 │ Message   │  │  Fan-out Router   │──────▶│  Presence / Routing   │
 │ Store     │  │  (per-channel)    │       │  registry: user->     │
 │ (sharded  │  │  decides delivery │       │  which gateway node   │
 │ by channel│  │  push vs inbox    │       │  (Redis)              │
 │ , Cassandra│  └────────┬─────────┘       └──────────────────────┘
 │ / Kafka log)│           │ index event
 └─────┬─────┘            ▼
       │           ┌──────────────────┐     ┌──────────────────────┐
       │           │  Search Indexer   │────▶│  Search Cluster       │
       │           │  (async consumer) │     │  (Elasticsearch,      │
       │           └──────────────────┘     │  sharded per org)     │
       │                                     └──────────────────────┘
       ▼
 ┌───────────────┐   Files path (separate):
 │ Read state /  │   Client --presigned PUT--> Object Store (S3) --CDN--> readers
 │ unread counts │   message just carries file_id + metadata
 └───────────────┘

Request flow - sending a message (the hot path):

  1. Client sends POST /channels/{id}/messages over HTTP to the stateless API tier (not over the WebSocket - the socket is for receiving; sending over HTTP is simpler to load-balance, retry, and authorize).
  2. API authorizes: is this user a member of this channel? (cached membership check.)
  3. API assigns a per-channel monotonic sequence number and persists the message durably to the message store. Only after the durable write does it ACK the sender. This gives strong per-channel ordering and read-your-writes.
  4. API publishes a message_created event to the fan-out router for that channel.
  5. The fan-out router looks up the channel’s members, and for each member consults the routing registry (which gateway node holds that user’s live connection, if any). For online members it pushes the event to the right gateway, which writes it down the WebSocket. For offline members it updates their unread state / inbox pointer so they get it on reconnect.
  6. In parallel, an async search indexer consumes the same event stream and indexes the message body into the search cluster. A file, if any, was already uploaded straight to object storage; the message just references its file_id.

Receiving: the client’s WebSocket to its gateway delivers the pushed event. On reconnect or channel-open, the client instead calls GET /channels/{id}/messages?after={last_seq} to backfill anything it missed while offline. The socket is a live tap; the store is the source of truth.

The key split: the write path (persist + sequence) is separate from the delivery path (fan-out + push), joined by an event stream. That decoupling lets the durable store and the real-time layer scale and fail independently - a slow gateway never blocks a durable send, and a search outage never blocks delivery.

Component Deep Dive

Four hard parts: (1) fan-out - getting one message to N channel members without dying on huge channels, (2) the connection layer - 10M live sockets and routing a push to the right one, (3) message storage and ordering - permanent, per-channel-ordered, petabyte-scale, and (4) search across billions of permission-scoped messages. Threads and DMs fall out of the channel primitive; I cover them inside fan-out. Each part goes naive -> where it breaks -> the fix.

1. Fan-out: One Message to N Members

This is the spine. When a message hits a channel, every member needs it. How?

Approach A: Fan-out on write (push a copy into every member’s inbox)

On send, look up the channel’s members and write the message into a per-user “inbox” table for each one. Reading a channel is then a cheap read of your own inbox.

on_send(channel, msg):
    for member in members(channel):        # <-- this loop is the problem
        inbox[member].append(msg_ref)

Fast reads, dead simple mental model. Where it breaks: the write amplification on large channels is catastrophic. A single post to a 50,000-member #general becomes 50,000 inbox writes. At 40,000 messages/sec with an average fan-out of 100 and a fat tail of huge channels, you are doing millions to tens of millions of inbox writes per second, and a viral message in the biggest org spikes 50K writes for one send. Worse, most of those inboxes belong to people who are offline or who have that noisy channel muted and will never look - you did enormous work for nothing. And every new member joining a busy channel would need backfill writes. Pure fan-out-on-write is how you melt the write tier on exactly the channels that matter most.

Approach B: Fan-out on read (store once, readers come get it)

Store the message exactly once, in the channel’s own log. Readers query the channel log for messages after the last one they saw.

on_send(channel, msg):
    channel_log[channel].append(msg)       # one write, done

on_open(user, channel):
    return channel_log[channel].after(user.last_seq[channel])

Where it breaks: one write is great, but now a user with 200 channels who opens the app must query 200 channel logs to compute unread state, and a hot channel is read by 50,000 people simultaneously - the read amplification and the “compute my unreads across all my channels” fan-in become the bottleneck. Pure fan-out-on-read makes app-open and unread-badge computation expensive, and it does nothing to push a live message to a connected user.

Approach C: The hybrid Slack actually needs - store once, route live, pull on open

The winning design separates durable storage from live delivery and picks per-channel:

  • Always fan-out on write for the durable store? No - store once. The message is written a single time to the channel’s log (fan-out on read for storage). This keeps writes at ~40K/sec, not millions.
  • Fan-out live only to connected members. The router pushes the message over WebSocket only to members who currently hold a live connection. Offline members get nothing pushed; they will pull on reconnect. This bounds live work to the online subset (~40%), not the whole membership.
  • For the fat tail (huge channels), do not even push per-member - use channel subscriptions on the gateway. Instead of the router computing “50,000 members, look up each one’s gateway, push 50,000 times,” each gateway that has any connected member of a channel subscribes to that channel’s event stream once. When a message is published, the gateway receives it once and locally forwards it to the (say 800) connected sockets it holds for that channel. This turns router-side fan-out from O(members) into O(gateways-with-a-member) - a few dozen - and pushes the per-socket fan-out to the edge where the connections already live.
Router publishes msg -> channel topic  (one publish)
      │
      ├─▶ Gateway-7  (has 800 connected members of this channel)
      │        └─ writes to 800 local sockets
      ├─▶ Gateway-3  (has 400 connected members)
      │        └─ writes to 400 local sockets
      └─▶ Gateway-9  (has 1200 connected members)
               └─ writes to 1200 local sockets
   Router did ~3 publishes, not 50,000 point pushes.
  • Unread counts without fanning out writes. Do not maintain a per-user copy of every message. Instead store, per user per channel, a single cursor: last_read_seq. Unread count = channel.latest_seq - user.last_read_seq, computed cheaply from the channel’s current sequence number and the user’s cursor. Mentions get a small dedicated counter incremented only for @mentioned users (that narrow fan-out-on-write is affordable because mentions are rare relative to messages).

Threads ride this exactly: a reply is a normal message in the channel log with a thread_root_id set. It is delivered live only to people viewing that thread (subscribed to the thread) plus the root author and thread participants - not broadcast to the whole channel timeline - unless also_send_to_channel is set. The main channel timeline query filters out thread_root_id != null replies so threads do not flood it. One storage model, two views.

DMs are channels with exactly 2 (or a few) members and no discoverability. Same log, same sequencing, same delivery. We special-case only privacy (never listed, membership immutable) and notification defaults - not the machinery.

Verdict: store once per channel (bounded writes), push live only to connected members via per-channel gateway subscriptions (bounded live fan-out to a few gateways), and derive unread from a cursor (no per-message user copies). Reserve narrow fan-out-on-write for mentions only.

2. The Connection Layer: 10M Live Sockets

Delivery needs to find the socket. How do you hold 10M WebSockets and route a push to the right one?

Naive: one server holds all connections and a map of user -> socket

Where it breaks: no single box holds 10M sockets (memory, file descriptors, CPU for TLS and framing all cap out around a few hundred thousand connections). And if that box dies, every user drops at once. You must have a fleet, which immediately raises the real question: when the router wants to push to user X, which of the 50 gateway nodes holds user X’s socket?

Better: a stateful gateway fleet + a routing registry

  • Gateway fleet. Dozens of gateway nodes, each terminating ~100K-500K WebSocket connections. Clients connect through a load balancer that supports long-lived connections; a connection sticks to its gateway for its lifetime. 10M / ~250K ≈ 40 nodes, run more for headroom and failure domains.
  • Routing registry. When a client connects, the gateway records user_id -> gateway_node_id (and conn_id) in a fast shared registry (Redis), and records which channels that user is subscribed to. When the router needs to deliver, it looks up the member’s gateway from the registry and pushes there. With the channel-subscription optimization from part 1, the router mostly publishes to channel topics and gateways self-subscribe, so the registry is consulted more for targeted events (DMs, mentions, presence) than for every channel message.
  • Presence lives naturally here: a user is “online” iff they hold a live connection, which the gateway already knows. Presence changes and typing indicators are published on a separate, lossy, low-priority channel - they must never compete with message delivery, and dropping a typing event is harmless.
Client --WSS--> [LB] --> Gateway-12
   on connect:  registry.set(user:U -> {node: 12, conn: c81})
                gateway subscribes to topics for U's active channels
   on message to U: router publishes to channel topic ->
                    Gateway-12 receives once, writes U's socket
   on disconnect:   registry.del(user:U); mark presence offline (after grace)
  • Reconnect and missed messages. WebSockets drop constantly (laptops sleep, phones switch networks, gateways get redeployed). We never rely on the socket for durability. On reconnect the client sends its per-channel last_seq cursors and pulls anything newer via GET /messages?after=. The socket is a live accelerator; the durable store plus cursors is the correctness guarantee. This is why sending over HTTP and receiving over WS is clean - the two paths fail independently.
  • Gateway death. If a gateway node dies, its ~250K clients detect the dropped socket and reconnect (with jittered backoff to avoid a thundering herd) to a new node via the LB, re-register in the registry, and backfill via cursors. No message is lost because none of them lived only in the socket.

Verdict: a horizontally scaled stateful gateway fleet, a Redis routing registry mapping user -> node and user -> channels, channel-topic subscriptions to minimize router work, and cursor-based backfill so a dropped socket is always recoverable from the durable store.

3. Message Storage and Ordering: Permanent, Ordered, Petabyte-Scale

Messages are permanent, per-channel-ordered, and grow to petabytes. Where do they live?

Naive: one big SQL table, messages(id, channel_id, user_id, ts, body), autoincrement id

Where it breaks: at ~700M rows/day and multi-PB total, a single SQL instance cannot hold it, and a global autoincrement id is a single hot sequence that serializes every write in the whole product. Ordering by wall-clock ts is wrong too - two messages in the same channel can land in the same millisecond, and clocks skew across app servers, so ts does not give a total order within a channel. Cross-channel range scans and the ever-growing index make reads slower over time. A single relational table is the wrong shape for an append-heavy, per-key-ordered, unbounded log.

Better: an append-only log sharded by channel, with a per-channel sequence

The access pattern is: append to a channel, and read a contiguous range of a channel’s recent messages. That is a partitioned commit log, not a relational table.

  • Shard by channel_id. All of one channel’s messages live on one shard, so per-channel ordering is a local property and reads are a single-shard range scan. channel_id is high-cardinality and uniformly hashable, so load spreads evenly; even the largest org’s busiest channel is one channel’s worth of writes, which one shard handles.
  • Per-channel sequence number for ordering. The API assigns a monotonic seq per channel at write time (an atomic increment on the channel’s counter, or the offset the append lands at in the channel’s log). Ordering within a channel is then defined by seq, not by wall-clock time - deterministic and total, immune to clock skew. Cross-channel there is no order and none is needed.
  • Store choice: a wide-column store (Cassandra-class) keyed by channel. Partition key = channel_id, clustering key = seq (descending, so “latest N” is the front of the partition). This gives O(1) partition location, ordered clustering for range reads, linear write scalability, and tunable replication for durability. It is built for exactly this append-and-range-scan-by-key pattern.
Table: messages
  PARTITION KEY: channel_id
  CLUSTERING KEY: seq DESC
  columns: msg_id, user_id, ts, thread_root_id, body, file_id, edited_at
Read latest 50:  SELECT * FROM messages
                 WHERE channel_id = ? ORDER BY seq DESC LIMIT 50
Backfill:        WHERE channel_id = ? AND seq > ? (client's last_seq)
  • Durability and ACK. The send is ACKed only after the write is committed to a quorum of replicas. That is the “never lose an acknowledged message” guarantee. The event that drives fan-out and indexing is emitted after the durable write, so nothing is delivered that is not stored.
  • Tiered storage. Recent messages (last weeks/months) are the hot 99% of reads - keep them on fast storage / cache. Old messages are rarely read - age them to cheaper cold storage, still queryable but slower. This keeps the hot path fast while permanent history stays affordable at petabyte scale.
  • A durable event log (Kafka) in front for fan-out and indexing. The API appends each message to a per-channel-partitioned Kafka topic as the ordering + distribution backbone; the message store, fan-out router, and search indexer are all consumers. Kafka’s per-partition ordering is the per-channel ordering, and its durability means a consumer (like search) that falls behind or restarts just replays from its offset without losing anything.

Verdict: an append-only, channel-sharded commit log (Kafka for distribution + Cassandra for the queryable store) with a per-channel seq for total in-channel order, quorum-durable writes before ACK, and hot/cold tiering for permanent history.

4. Search Across Billions of Permission-Scoped Messages

Search is first-class: full-text over every message a user is allowed to see, fast, filtered, ranked.

Naive: SELECT ... WHERE body LIKE '%term%' on the message store

Where it breaks: a LIKE '%term%' scan cannot use an index, so it is a full scan of a petabyte-scale store per query - seconds to minutes, and it melts the store that is also serving live reads. Relational/wide-column stores are built for key lookups, not inverted full-text search, ranking, or fuzzy matching. You need a purpose-built inverted index.

Better: an async inverted-index cluster, sharded per org, with permission filtering

  • Dedicated search engine (Elasticsearch-class). The search indexer consumes the message event stream (the same Kafka log, a separate consumer group) and writes each message into an inverted index: tokenized body, plus filterable fields channel_id, user_id, ts, has_file, thread_root_id. Indexing is fully async - it must never be on the send hot path. A message is searchable a second or two after it is sent, which is fine; being sent must not wait on being indexed.
  • Shard the index by org_id. A user only ever searches within their own org, so partitioning by org keeps every query on a bounded slice (even the 100K-member org is one org’s index), avoids cross-org scans entirely, and gives natural tenant isolation. Within a large org, sub-shard by channel or by time.
  • Permission scoping is the hard correctness part. A user must never see search hits from private channels or DMs they are not in. Two layers:
    1. Query-time filter: resolve the set of channels the user can read (their memberships, cheaply cached) and constrain the search to channel_id IN (that set). For a user in hundreds of channels this is a large but bounded filter, and it is the authoritative gate.
    2. Never index what leaks: DMs and private channels are indexed with an explicit ACL field so the filter is enforceable, and results are always re-checked against live membership before return, so a just-removed member cannot see fresh hits.
  • Ranking. Combine text relevance (BM25) with recency and social signals (messages in channels you are active in, from people you talk to, rank higher). Slack search leans heavily on recency - a message from this week usually beats a perfect keyword match from three years ago.
Send -> Kafka(msg) --consumer--> Indexer -> ES index (sharded by org_id)
Query: GET /search?q=deploy failed&in=#ops&from=@ana&after=2026-07-01
   1. resolve searchable channel set for user (cached memberships)
   2. ES query: full_text(q) AND channel_id IN {set} AND filters
   3. rank by relevance + recency, re-check ACL, return top N

Verdict: an async, per-org-sharded inverted index fed off the durable event log, with query-time channel-membership filtering as the authoritative permission gate and recency-weighted ranking. Search scales independently of both send and delivery, and can fall behind or rebuild by replaying Kafka without touching the live path.

API Design & Data Schema

Client-facing API

POST /v1/channels/{channel_id}/messages          # send a message
  Body: { client_msg_id (idempotency key), body, thread_root_id?, file_ids?[] }
  201: { msg_id, seq, ts, channel_id }
  403: not a member of channel

GET  /v1/channels/{channel_id}/messages           # history / backfill
  Query: ?before_seq= | ?after_seq= | ?limit=50
  200: { messages: [...], has_more: bool }

GET  /v1/channels/{channel_id}/threads/{root_id}  # a thread's replies
  200: { root: {...}, replies: [...] }

POST /v1/channels/{channel_id}/read               # advance read cursor
  Body: { last_read_seq }
  200: { unread: 0, mention_count: 0 }

GET  /v1/me/unreads                                # unread badges across channels
  200: { channels: [ {channel_id, unread, mention_count} ] }

WSS  /v1/connect                                   # open the live event socket
  server -> client events:
     { type: "message", channel_id, msg_id, seq, user_id, body, thread_root_id? }
     { type: "typing",  channel_id, user_id }        # ephemeral, lossy
     { type: "presence", user_id, status }           # ephemeral

# Files (kept off the message path)
POST /v1/files/upload-url                          # get a presigned PUT URL
  Body: { filename, content_type, size }
  200: { file_id, upload_url }                      # client PUTs bytes to object store
GET  /v1/files/{file_id}                            # 302 -> CDN/presigned GET URL

# Search
GET  /v1/search?q=...&in=&from=&before=&after=&has=file
  200: { hits: [ {msg_id, channel_id, seq, snippet, score} ], total }

Design notes:

  • client_msg_id makes send idempotent: a retried POST after a flaky network does not create a duplicate; the API dedupes on it. Essential because clients retry aggressively.
  • Sends are HTTP, live events are WSS - two independent paths, so a socket drop never risks a lost send and an HTTP retry never double-delivers.
  • Files never traverse the message API. The client gets a presigned URL, uploads bytes straight to object storage, then references file_id in a message. This keeps large blobs off every hot path and lets a CDN serve downloads.

Data schemas

Messages (wide-column, Cassandra-class) - sharded by channel:

messages
  channel_id     BIGINT     PARTITION KEY
  seq            BIGINT     CLUSTERING KEY (DESC)   -- per-channel total order
  msg_id         UUID
  user_id        BIGINT
  ts             TIMESTAMP                          -- display only, NOT ordering
  thread_root_id BIGINT     NULL                    -- set => this is a thread reply
  reply_count    INT                                -- denormalized on the root
  body           TEXT
  file_ids       LIST<UUID>
  edited_at      TIMESTAMP  NULL
  -- read pattern: latest N, and range after client's last_seq, both single-partition

Why NoSQL here: append-heavy, keyed strictly by channel_id, ordered by seq, no cross-partition joins, unbounded growth, needs linear write scale and tunable replication. That is the wide-column sweet spot. A relational table would choke on the volume, the global sequence, and the ever-growing index.

Channel membership + read state (SQL) - the source of truth for authz:

channels
  channel_id   BIGINT PK
  org_id       BIGINT     INDEX (org_id)
  type         ENUM       -- public | private | dm | group_dm
  name         TEXT
  latest_seq   BIGINT     -- current head of the channel log (for unread math)

channel_members
  channel_id   BIGINT
  user_id      BIGINT
  last_read_seq BIGINT    -- the cursor: unread = channel.latest_seq - this
  mention_count INT       -- narrow fan-out-on-write increments this only for @user
  role         ENUM
  PRIMARY KEY (channel_id, user_id)
  INDEX (user_id)         -- "what channels is this user in" (drives search scoping)

Membership and read state are small, relational, correctness-critical, and read on every send (authz) and every unread computation - a natural SQL fit, heavily cached. Unread is derived arithmetic (latest_seq - last_read_seq), not a stored per-message copy, which is what kills the fan-out-on-write pressure.

Search index (Elasticsearch-class) - sharded by org:

index: messages_{org_id}
  doc: { msg_id, channel_id, user_id, ts, thread_root_id,
         body (analyzed/tokenized), has_file, acl_type }
  filters: channel_id, user_id, ts range, has_file
  ranking: BM25(body) blended with recency + activity signals

Files: metadata row (file_id, uploader, size, content_type, storage_key, created_at) in SQL; bytes in object storage (S3-class) with replication, served via CDN. Never in the message store.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Fan-out on a huge channel (breaks first, and worst). A post to a 50K-member channel naively means 50K writes or 50K point-pushes. Fix: store once (per-channel log, not per-user copies) so writes stay ~40K/sec; push live only to connected members via per-channel gateway subscriptions so the router does O(gateways-with-a-member) publishes (a few dozen), not O(members); derive unread from a cursor so there is no per-message user copy at all. Reserve narrow fan-out-on-write for @mentions only, which are rare.

2. The 10M-connection layer as a SPOF / capacity wall. No single node holds the sockets, and a node’s death drops its clients. Fix: a horizontally scaled stateful gateway fleet (~40+ nodes at ~250K conns each), a Redis routing registry (user -> node, user -> channels), and cursor-based reconnect so a dropped socket is always recoverable from the durable store. Jittered reconnect backoff prevents a thundering herd on gateway death or redeploy.

3. Message store write/scan limits at petabyte scale. A single SQL table and a global autoincrement serialize writes and cannot hold the volume. Fix: shard the message log by channel_id with a per-channel seq for ordering, on a wide-column store built for append + range-by-key, fronted by a per-channel-partitioned Kafka log as the ordering + distribution backbone. Quorum-durable write before ACK. Hot/cold tiering keeps recent messages fast and permanent history cheap.

Shard key choice: channel_id for the message log (every read/write is scoped to one channel, and in-channel order is a per-channel invariant), and org_id for the search index (every query is scoped to one org). Sharding messages by user would split a channel’s timeline across shards and destroy the single-partition ordered read; sharding search by anything but org would force cross-tenant scans.

4. Search melting the primary store. LIKE scans over petabytes are impossible and would starve live reads. Fix: a separate async inverted-index cluster (Elasticsearch), fed off the Kafka log by its own consumer group, sharded per org, with query-time channel-membership filtering as the authoritative permission gate. Indexing is off the send path; search scales and fails independently and can rebuild by replaying Kafka.

5. Files clogging the message path. 35 TB/day of blobs cannot flow through the API or the message store. Fix: presigned direct-to-object-storage uploads; messages carry only a file_id; CDN serves downloads. Large binary is completely off every message hot path.

6. Hot channel = hot partition. The busiest #general in the biggest org concentrates reads on one partition. Fix: cache the hot tail (last N messages of hot channels) in an in-memory cache so the 50K live readers hit cache, not the store; the gateway-subscription model already collapses their live delivery into a handful of gateway pushes. Read replicas of the message store absorb history reads. If a single channel ever truly exceeds one partition’s write capacity, sub-partition its log by time window.

7. Presence/typing firehose starving message delivery. Typing events from 10M users can outnumber messages and are worthless if delivered late. Fix: put presence/typing on a separate, lossy, low-priority channel with its own rate limits and aggressive coalescing (batch presence deltas, debounce typing). They must never share a queue with durable messages. Dropping a typing indicator is invisible; delaying a message is not.

8. Mention/notification fan-out and the notification service. @channel on a 50K-member channel could trigger 50K push notifications. Fix: mentions increment a small per-member counter (narrow, affordable); push notifications go async through a dedicated notification service that respects per-user do-not-disturb, mute, and dedup rules, and rate-limits @channel/@here blasts. It consumes the event log, so it never blocks send or delivery.

9. Thread replies leaking into the main channel. A busy thread must not spam everyone in the channel. Fix: replies are stored with thread_root_id and delivered live only to thread subscribers (root author, prior repliers, anyone viewing the thread); the main timeline query filters thread_root_id IS NULL. One store, two delivery scopes.

Wrap-Up

The trade-offs that define this design:

  • Store once, deliver live only to the connected - hybrid over pure fan-out. Fan-out-on-write melts on huge channels; fan-out-on-read makes app-open expensive. Slack’s answer is to store the message a single time per channel, push live only to currently-connected members (and via per-channel gateway subscriptions, so the router does a few publishes not 50,000), and derive unread counts from a cursor instead of per-user copies. Narrow fan-out-on-write is reserved for the rare case that needs it: mentions.
  • Separate the write path from the delivery path, joined by a durable log. Persist-and-sequence (HTTP, quorum-durable, ACK after write) is decoupled from fan-out and push (WebSocket, best-effort, recoverable via cursors) through Kafka. A slow gateway never blocks a durable send; a search outage never blocks delivery; a dropped socket is always backfillable from the store.
  • Channel as the unit of everything. Ordering, sharding, consistency, and delivery are all per-channel. Strong order within a channel, independence across channels, channel_id as the shard key - one bounded, enumerable unit makes the whole system partition cleanly, and DMs and threads are just channels wearing different privacy and view rules.
  • Search as an independent async plane. An inverted index sharded per org, fed off the event log, with membership filtering as the authoritative permission gate - so search scales, fails, and rebuilds without ever touching the send or delivery hot paths.
  • Push the fan-out to the edge. Gateways that already hold a channel’s connections do the per-socket fan-out locally, turning router work from O(members) into O(gateways). The connections are already there; use them.

One-line summary: persist each message once to a channel-sharded, per-channel-sequenced durable log; fan it out live only to connected members through a stateful WebSocket gateway fleet using per-channel subscriptions; derive unread from cursors; and index asynchronously into a per-org search cluster - so channels, DMs, threads, search, and files all hold up at 10M concurrent users and 100K-member orgs without any single path melting under the others’ load.