Design Gmail (Email Service Backend) - System Design

Email looks like a solved problem. It is 40 years old, the protocol fits on a napkin, and every framework ships a send_mail() in three lines. So when the interviewer says “design Gmail,” the temptation is to draw a box called “mail server” and move on. That box is where the interview goes to die.

The real problem is that email is not one system. It is at least four, wired together and each hard at a different scale. You have to receive mail from the entire internet over SMTP, a protocol you do not control and cannot rate-limit at the source, from senders who range from your own users to botnets. You have to store it durably forever, because people treat their inbox as a filing cabinet and never delete anything. You have to make every user’s mailbox fully searchable in milliseconds, which at billions of messages means an inverted index the size of a small country. And you have to filter spam and malware on the way in, at line rate, before it ever touches a mailbox. 1.8B users, billions of messages a day, attachments up to 25MB, and a storage bill that only grows. Let me build it properly, one hard part at a time.

The one framing decision that drives everything: email is asymmetric and archival. Unlike chat, a message does not need to arrive in 200ms - SMTP is store-and-forward, seconds of latency are fine. But unlike chat, nothing is ever deleted, everything must be searchable years later, and the ingestion path is exposed to a hostile internet. So the design pressure is not real-time push; it is durable storage at petabyte scale, search over that storage, and a hardened front door.

Functional Requirements (FR)

In scope:

• Receive inbound email over SMTP. Accept mail from any mail server on the internet addressed to one of our users, run it through spam/malware filtering, and deposit it in the recipient’s mailbox.
• Send outbound email over SMTP. A user composes a message; we relay it to the recipient’s mail server (which we do not control), handling retries, bounces, and delivery status.
• Store mailboxes durably. Every message, read or unread, kept effectively forever with folders/labels (Inbox, Sent, Spam, Trash, custom labels), flags (read/unread, starred), and threading (group a reply chain into one conversation).
• Full-text search per user. “Find every email from alice@ that mentions invoice, last quarter, with a PDF attachment” returning in well under a second, scoped strictly to that one user’s mailbox.
• Attachments up to 25MB per message, stored efficiently (deduplicated), scanned for malware, downloadable on demand.
• Spam and malware filtering on inbound mail: classify, quarantine to Spam, or reject at SMTP time.
• Read path (webmail / IMAP / API). List a folder, open a thread, mark read/unread, move/label, delete.

Explicitly out of scope (say it so you own the scope):

• The webmail frontend / rich client UI. We are designing the backend and its APIs, not React.
• Calendar, Contacts, Chat, Drive. Adjacent products, separate systems.
• End-to-end encryption (S/MIME, PGP). We store mail encrypted at rest, but we are not designing client-side E2E key exchange. Note: real Gmail can read message bodies to build the search index; E2E would break search, which is a deliberate product trade-off.
• The exact ML model for spam. We design the pipeline (where classification runs, how signals flow, how feedback loops back), not the neural net.

The decision that drives everything: this is a storage-and-search system with a hostile write path, not a low-latency messaging system. Reads and writes are both heavy, storage grows without bound, and the SMTP front door faces the open internet. That shapes every component below.

Non-Functional Requirements (NFR)

• Scale: ~1.8B users, ~300B message-deliveries/day inbound (most of it spam that we reject or quarantine), ~1.5B legitimate inbound/day, ~1B outbound/day. Average mailbox tens of thousands of messages; heavy users in the millions.
• Latency: SMTP receive can take seconds - it is store-and-forward, senders retry. Target: accept-or-reject decision within a few seconds so the sending server is not left hanging. Search and mailbox reads are the latency-sensitive path: p99 under 300ms for a folder listing or a search query. Sending (enqueue) is instant to the user; actual relay is async.
• Availability: 99.99%+ on receive and read. If we cannot accept a message, SMTP says “try again later” (a 4xx) and the sender retries for days, so a brief outage does not lose mail - but a long one strands it. Read availability is what users feel; the inbox must load.
• Consistency: a delivered message must appear in the mailbox and be searchable within seconds (eventual consistency on the search index is acceptable, a few seconds of lag is fine). Flags (read/unread) and label changes must be read-your-writes for the user who made them, across their devices. No message ever silently lost.
• Durability: 11 nines for stored mail. Once we accept a message at SMTP (return 250 OK), we own it forever and must never lose it. This is the hardest guarantee in the system - we take responsibility from a sender who then forgets the message ever existed. Multi-replica, multi-region, checksummed.
• Security: hostile ingestion. Spam, phishing, malware attachments, spoofing (SPF/DKIM/DMARC checks), and volumetric abuse are the normal case, not the exception.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Inbound receive rate (writes):

Legit inbound:      1.5B messages / day
= 1,500,000,000 / 86,400 s ≈ 17,400 msg/sec average
Peak (3x)          ≈ 52,000 msg/sec

But raw SMTP connection attempts (mostly spam/bots) are ~10x that:
~15B connection attempts/day ≈ 175,000/sec average, ~500,000/sec peak.
Most are rejected at the SMTP edge before a message body is even accepted.

The edge has to survive half a million SMTP conversations a second at peak, the vast majority of which it will reject. That, not the legit 52K/sec, sizes the receiving tier.

Outbound send rate:

1B outbound / day ≈ 11,600 msg/sec average, ~35,000/sec peak.
Each may relay to a remote server that is slow, greylisting, or down -> async queue with retries.

Storage (the dominant cost):

Average message size, including headers and small inline content, call it ~50KB. Attachments are separate and larger. Per message metadata (for indexing/listing) is ~1KB.

Legit stored/day = 1.5B inbound + 1B outbound ≈ 2.5B messages/day
Bodies:      2.5B * 50KB   ≈ 125 TB/day of message bodies
Per year:    125 TB * 365  ≈ 45 PB/year of new mail bodies
With replication factor 3:  ≈ 135 PB/year on disk.

Attachments dominate and are why dedup matters:

Say 20% of messages carry an attachment, average 1MB (many are tiny, some are 25MB):
2.5B * 20% * 1MB ≈ 500 TB/day of attachments
Per year:         ≈ 180 PB/year raw, but a huge fraction are duplicates
(the same PDF/newsletter image mailed to millions).
Content-addressed dedup can cut this 3-5x -> store once, reference N times.

So we are adding hundreds of petabytes a year, forever, and it never shrinks because nobody deletes email. Storage growth is the defining cost of this system.

Search index size:

Index the text of 2.5B new messages/day. An inverted index is roughly
30-50% of the source text size for email-shaped content.
Text portion ≈ 2.5B * 10KB (text only) = 25 TB/day of text
Index ≈ 10 TB/day added, per-user, sharded.
Cumulative index runs into multiple PB - but it is per-user shardable,
so no single index is huge; it is the aggregate that is large.

Read / search QPS:

1.8B users, say 20% active daily, each doing ~30 mailbox reads
(folder loads, thread opens) + ~3 searches:
360M users * 33 ops ≈ 12B read ops/day ≈ 140,000 ops/sec average
Peak ≈ 420,000 read ops/sec.
Of these, searches ≈ 40,000/sec peak.

Bandwidth: dominated by attachment download from the blob store, served via CDN. Message-body reads are small (~50KB) but frequent; ~140K reads/sec * 50KB ≈ 7 GB/sec of body reads, cacheable.

The takeaways: the SMTP edge must survive ~500K hostile conversations/sec; storage grows ~100s of PB/year and never shrinks, making attachment dedup and tiered storage mandatory; search is a per-user-sharded index in the multi-PB range; and read QPS peaks near half a million/sec. Storage and search, not latency, are the design drivers.

High-Level Design (HLD)

The system splits into four planes: an ingestion plane (hardened SMTP edge + spam pipeline), a storage plane (metadata store + body blob store + attachment store, all durable), a search plane (per-user inverted index built async), and a delivery/read plane (outbound relay + the read API that webmail/IMAP hit). A durable message queue sits between ingestion and storage so a spike of inbound mail never blocks on disk writes, and the search index is built by consuming that same stream.

   Internet mail senders            Our users (webmail / IMAP / API)
          │ SMTP                              │ HTTPS / IMAP
          ▼                                   ▼
  ┌───────────────────┐              ┌──────────────────────┐
  │  SMTP Edge (MX)    │              │   Read API / IMAP     │
  │  + SPF/DKIM/DMARC  │              │   Gateway (stateless) │
  │  + connection      │              └───────┬──────────────┘
  │    throttling      │                      │ read
  └─────────┬──────────┘             ┌─────────┼───────────────┐
            │ accepted RAW mail      │         │               │
            ▼                        ▼         ▼               ▼
  ┌───────────────────┐      ┌────────────┐ ┌────────────┐ ┌────────────┐
  │  Spam / Malware    │      │ Metadata   │ │  Body Blob │ │  Search    │
  │  Filter Pipeline   │      │ Store      │ │  Store     │ │  Index     │
  │  (score, verdict)  │      │ (per-user, │ │ (immutable,│ │ (per-user  │
  └─────────┬──────────┘      │  sharded)  │ │  dedup)    │ │  shards)   │
            │ verdict + mail          ▲         ▲               ▲
            ▼                         │         │               │
   ┌─────────────────────────────────┴─────────┴───────────────┴──────┐
   │        Durable Ingest Queue (Kafka) - "a message arrived"          │
   └───────┬───────────────────────────────┬───────────────────────────┘
           │ Delivery Worker                │ Indexer Worker
           │ (writes metadata + body,       │ (parses text, updates
           │  applies labels)               │  that user's index)
           ▼                                ▼
   ┌────────────────┐              ┌──────────────────┐
   │ Attachment      │             │  Outbound Relay   │  (send path)
   │ Store (CAS,     │             │  Queue + MTA pool │  -> remote MX
   │ malware-scanned)│             │  retries, bounces │
   └────────────────┘             └──────────────────┘

Inbound receive flow (someone emails our user bob@):

1. The sender’s mail server does a DNS MX lookup for our domain, gets our SMTP Edge IPs, and opens an SMTP conversation.
2. The edge does cheap rejects first: is the source IP on a blocklist? Is the connection rate from this IP abusive? Does the sender domain’s SPF record authorize this IP? Most spam dies here with a 550 reject or a 421 throttle - we never accept the body.
3. For a surviving conversation, the edge accepts the message body (DATA), verifies DKIM signature and DMARC alignment, and writes the raw message once to a temporary durable spool. It returns 250 OK only after the raw mail is safely persisted - this is the durability handoff point.
4. The edge publishes an event to the Ingest Queue: “raw message X for user bob is spooled.”
5. The Spam/Malware Filter consumes it, scores the message (content signals, sender reputation, URL/attachment analysis), and attaches a verdict: inbox, spam, or reject-bounce.
6. The Delivery Worker consumes the scored message: it stores the body in the Body Blob Store, extracts attachments to the Attachment Store (content-addressed, dedup, malware-scanned), writes the metadata row into bob’s shard of the Metadata Store (with the label Inbox or Spam, thread id, flags), and emits an “indexed?” event.
7. The Indexer Worker consumes the same stream, parses the message text, and updates bob’s search index shard. Within a few seconds the message is searchable.
8. If bob has a live IMAP/webmail session, a lightweight notification pushes “new mail” so the client refreshes. Otherwise it is simply there next time bob loads the inbox.

Read flow (bob opens his inbox / searches):

1. bob’s client hits the Read API (HTTPS) or IMAP Gateway.
2. A folder listing is a query on bob’s shard of the Metadata Store: “messages where label=Inbox order by date desc limit 50” - metadata only, fast.
3. Opening a message fetches the body from the Body Blob Store (cached); attachments stream from the Attachment Store via CDN on demand.
4. A search hits bob’s search index shard, gets back matching message ids, then hydrates those ids from the Metadata Store for display.

Send flow (bob sends mail):

1. bob POSTs the composed message to the Read/Send API. We authenticate bob, run outbound spam checks (do not let a compromised account blast spam), DKIM-sign the message, write it to bob’s Sent folder (metadata + body + index), and enqueue it to the Outbound Relay Queue. bob’s UI shows “sent” immediately.
2. An MTA (mail transfer agent) worker dequeues it, does an MX lookup for each recipient domain, opens SMTP to the remote server, and delivers. If the remote is down or greylisting (4xx), it retries with backoff over hours/days. A hard failure (5xx) generates a bounce back to bob.

The key insight: a durable queue decouples the hostile, bursty ingestion front from the storage and index backends, and the message is persisted (spooled) before we ever say 250 OK, so accepting responsibility for a message and safely storing it are the same atomic moment.

Component Deep Dive

The hard parts, naive-first then evolved: (1) the SMTP receive edge under a hostile firehose, (2) storing mail durably and cheaply forever, (3) per-user full-text search, (4) the spam filtering pipeline. Attachments fold into (2).

1. Receiving mail over SMTP under a hostile firehose

Naive approach: one SMTP server that accepts everything, then sorts it out. Listen on port 25, accept every connection, take the DATA, drop it in the right mailbox.

Where it breaks:

• 90%+ of inbound is spam/bots. At ~500K connection attempts/sec peak, a single accept-everything server spends all its resources accepting garbage it will immediately throw away. You cannot afford to read the body of every spam message; you have to reject as early and as cheaply as possible.
• SMTP is synchronous and slow per conversation. A single SMTP exchange (HELO/MAIL FROM/RCPT TO/DATA/.) can take seconds, especially with a slow or deliberately-stalling (slowloris-style) sender holding the connection open. One server holding thousands of stalled SMTP conversations starves.
• Accepting before persisting loses mail. If the server says 250 OK and then crashes before writing to disk, the sender has moved on and the message is gone - violating durability.

First evolution: a horizontally scaled edge tier behind DNS MX + anycast, rejecting in layers. Multiple MX records and anycast IPs spread the ~500K/sec across many stateless edge nodes. Crucially, each node rejects in cheapest-first order so expensive work only runs on mail likely to be legit:

Layer 0 (connection):   IP reputation / RBL blocklist, per-IP rate limit.
                        -> reject with 421/550 before HELO. Kills most botnets. ~cents of CPU.
Layer 1 (envelope):     SPF check on MAIL FROM domain (is this IP authorized to send for it?).
                        RCPT TO: is this even a real user? Reject unknown recipients (no backscatter).
Layer 2 (greylisting):  First-time (IP, from, to) triple? Return "451 try later."
                        Real senders retry and pass; most spam bots never retry. Cheap, brutal filter.
Layer 3 (data):         Only now accept the body. DKIM verify, DMARC alignment. Spool to durable store.

Each layer is strictly cheaper than the next and discards the bulk of traffic, so the expensive body-accept and content scan run on a small fraction. This is the whole game at the edge: push rejects as far left (as cheap) as possible.

The durability handoff. The edge does not write to the final mailbox - it writes the raw message to a replicated durable spool (write to N replicas / append to a replicated log) and only returns 250 OK after that write is acknowledged by a quorum. If the edge crashes after 250 but before the spool write, it never sent 250 in the first place - the sender’s SMTP client gets a broken connection, treats it as a soft failure, and retries. At-least-once from the sender + dedup by Message-ID on our side = no loss, no dupes.

Edge node pseudo-flow for an accepted body:
  raw = read DATA until "\r\n.\r\n"
  spool_id = durable_spool.append(raw)      # replicated, quorum-acked
  ingest_queue.publish({spool_id, envelope, dkim_result})
  return "250 OK: queued as " + spool_id     # <-- only now do we take responsibility

Slowloris / connection exhaustion is handled by aggressive per-connection timeouts, a cap on concurrent connections per source IP, and event-driven IO (one process handling many connections, not a thread per connection) so a stalled sender ties up a file descriptor, not a whole thread. The edge is thin and stateless; state lives in the spool and queue.

2. Storing mail durably and cheaply, forever

Naive approach: one big relational table messages(user_id, subject, body, ...) on a beefy SQL database.

Where it breaks:

• Volume. We add ~45PB/year of bodies. No single SQL database, and no reasonable cluster of them, holds that or sustains 50K+ writes/sec of large blobs. The body of an email (50KB, sometimes megabytes with inline content) does not belong in a relational row at all.
• Access patterns are split. Listing a folder needs metadata (from, subject, date, flags, labels) for 50 messages - small, structured, queried and sorted constantly. Reading one message needs its full body - large, fetched rarely, never sorted or filtered. Jamming both in one row means every folder listing drags megabytes of bodies through the database for no reason.
• Attachments. The same 25MB PDF newsletter mailed to 5M users stored 5M times is 125TB of pure duplication.

The fix: separate metadata, bodies, and attachments into three stores, each matched to its access pattern.

(a) Metadata Store - NoSQL wide-column, sharded by user. The dominant query is “list messages in user U’s folder F, newest first” and “get flags for these ids.” That is a partition-scoped, sort-by-date, no-joins workload at massive write and read volume - a wide-column store (Bigtable / Cassandra / ScyllaDB) is the exact fit.

Table: mailbox_metadata
  user_id      BIGINT   PARTITION KEY        -- a user's whole mailbox is co-located
  label        STRING   } CLUSTERING          -- Inbox, Sent, Spam, Trash, custom
  received_at  BIGINT   } (desc)              -- sort within a label by time
  message_id   BIGINT                          -- global, time-sortable (Snowflake)
  thread_id    BIGINT                          -- groups a reply chain
  from_addr    STRING
  subject      STRING
  snippet      STRING    -- first ~100 chars for the list preview
  flags        INT       -- bitfield: read, starred, has_attachment, ...
  body_ref     STRING    -- pointer into the Body Blob Store
  attach_refs  LIST<STRING>  -- content hashes into the Attachment Store
  Shard by: user_id  ->  one user's mailbox is one partition, all reads are single-partition.
  Read pattern: "label=Inbox, received_at desc, limit 50" = single-partition range scan.

Sharding by user_id is the single most important decision: every read a user does touches exactly one shard. No scatter-gather for a folder listing or a search. A user with millions of messages is the one hot partition; split such power-users across sub-partitions by (user_id, label) if needed.

(b) Body Blob Store - immutable content-addressed object store. Message bodies are written once and never modified. Store each body as an immutable object keyed by its content hash in an object store (like S3/GCS-class storage or an internal equivalent). body_ref in the metadata row points to it. Immutability means aggressive caching (a body never changes) and free deduplication of identical bodies.

(c) Attachment Store - content-addressed with dedup (the big win). Hash each attachment (SHA-256); store the bytes once under that hash; the metadata row holds only the hash reference plus a per-user encryption key wrapping. Ten million copies of the same newsletter image collapse to one stored object with ten million tiny references.

put_attachment(bytes):
    h = sha256(bytes)
    if not attach_store.exists(h):        # dedup: only the first copy is stored
        scan_for_malware(bytes)           # scan once, at store time
        attach_store.put(h, encrypt(bytes))
    return h                               # metadata just references h

This is where the storage estimate drops 3-5x. It also means malware scanning runs once per unique attachment, not once per delivery.

(d) Tiered storage for the archival tail. Mail from last week is read constantly; mail from 2019 almost never. Keep recent bodies/attachments on fast SSD-backed storage; age older objects to cheaper cold/archival tiers automatically. Metadata stays hot (you still list and search old mail), but the bodies and attachments of old mail move to cold storage, cutting cost on the petabytes that dominate the bill. A read of an ancient message pays a small cold-fetch latency, which is fine because it is rare.

3. Per-user full-text search

Naive approach: SELECT * FROM messages WHERE user_id=U AND body LIKE '%invoice%'.

Where it breaks:

• A LIKE '%term%' scan reads every message in the user’s mailbox for every query. A user with 500K messages pays a 500K-row scan per search. At 40K searches/sec, and with multi-term queries, this is a non-starter.
• No relevance, no phrase/field/date query, no attachment-type filter. Users expect “from:alice invoice has:attachment after:2026/01” to work, which LIKE cannot express.
• One global index is wrong. You might reach for a single giant Elasticsearch cluster indexing all mail. But search is always scoped to one user - there is no cross-user search. A single global index means every query filters a planet-sized index down to one user, and one noisy-neighbor mailbox degrades everyone.

The fix: an inverted index, sharded by user, built asynchronously off the ingest stream.

Inverted index basics. For each user, maintain a map term -> sorted list of message_ids containing it. A query “invoice AND from:alice” intersects the postings list for invoice with the postings list for the field-term from:alice. Intersection of two sorted id-lists is fast and returns only matching ids - no full scan.

Per-user inverted index (conceptually):
  "invoice"      -> [msg_1004, msg_2251, msg_8890, ...]
  "from:alice"   -> [msg_2251, msg_2252, msg_9001, ...]
  "has:pdf"      -> [msg_2251, msg_5567, ...]
  "date:2026-Q2" -> [msg_2251, ...]
Query "invoice from:alice has:pdf in Q2" = intersect all four postings -> [msg_2251]
Then hydrate msg_2251 from the Metadata Store for display.

Shard the index by user_id, co-located with (or parallel to) the metadata shard. Each user’s index is small and self-contained; a query touches exactly one index shard. This is the same “single-partition per user” principle as the metadata store, and it means search scales by adding shards, not by growing one monster index. Field-terms (from:, subject:, has:attachment, label:, coarse date: buckets) are indexed alongside body terms so structured queries are just more postings-list intersections.

Build it asynchronously. The Indexer Worker consumes the ingest stream: when a message lands, it tokenizes the subject/body/from/attachment-types, and appends the message_id to the relevant postings lists in that user’s index shard. This is off the critical delivery path - a message is stored immediately and becomes searchable a few seconds later. Eventual consistency on search is explicitly acceptable per the NFR.

Handle updates and deletes. Email is mostly append-only, which is friendly to inverted indexes (indexes hate in-place updates). Flag changes (read/starred) live in metadata, not the index, so they never touch it. A delete (message moved to Trash then purged) marks the id as deleted in a small per-user tombstone/deletion bitmap; queries filter tombstoned ids out, and background compaction eventually rewrites the postings lists to drop them - the standard “soft-delete then compact” pattern that avoids expensive live edits to postings lists.

Why not just one big Elasticsearch. Per-user sharding beats a global index because: (1) queries are single-user, so a global index wastes work filtering to one user; (2) isolation - one huge mailbox cannot slow another user’s search; (3) you can tier a user’s cold index with their cold mail. The engine can still be Lucene/Elasticsearch under the hood; the point is the sharding key is the user, not one flat index of the world.

4. Spam and malware filtering pipeline

Naive approach: a static blocklist of bad words and bad sender IPs, checked inline at receive.

Where it breaks:

• Spam adapts hourly. A static word/IP list is stale the moment you ship it; spammers rotate IPs, obfuscate words (“v1agr@”), and A/B test against your filter.
• Inline blocks the receive path. Running a heavy content classifier synchronously inside the SMTP conversation adds latency to every accept and couples ingestion throughput to model inference speed.
• No feedback loop. When a user hits “Report Spam” or “Not Spam,” a static list learns nothing. The single best spam signal - human labels from 1.8B users - goes unused.

The fix: a layered pipeline - cheap synchronous signals at the SMTP edge, expensive async classification off the queue, and a feedback loop from user actions.

Edge (synchronous, cheap - reject before accepting body):
   IP reputation (RBL), SPF/DKIM/DMARC pass/fail, connection-rate abuse.
   -> Hard spam sources rejected at SMTP with 550. Never enters the system.

Async (off the ingest queue, per accepted message):
   Sender reputation:   domain/IP history, DMARC alignment, is this a known bulk sender?
   Content classifier:  ML model over subject/body/URLs/structure -> spam score 0..1.
   URL analysis:        expand + check links against phishing/malware feeds.
   Attachment analysis: sandbox/scan for malware (once per unique content hash).
   -> combine into a verdict: inbox | spam-folder | reject-bounce | quarantine.

Feedback loop (continuous):
   User clicks "Report Spam" / "Not Spam" -> labeled training example.
   Aggregate across 1.8B users -> retrain the classifier -> updated model deployed.
   A message many users report is retroactively pushed to Spam for others (fingerprint match).

Why the split matters. The edge stays fast and cheap - it only does deterministic checks (IP, SPF/DKIM) that reject the volumetric bulk without reading content. The expensive, model-heavy content analysis runs asynchronously after 250 OK, so it never adds latency to the SMTP conversation or caps ingestion throughput. A message is accepted (we owe the sender no more) before it is classified; classification decides which folder it lands in, which can take a couple of seconds, which is fine.

The feedback loop is the actual moat. With 1.8B users, “Report Spam” clicks are a firehose of fresh labels. Fingerprinting a reported message (hash of normalized content) lets you catch the same spam campaign hitting other users within minutes, and the aggregate labels retrain the classifier continuously. Malware scanning is content-addressed like attachments - scan each unique payload once, cache the verdict by hash.

API Design & Data Schema

Two API surfaces: SMTP (machine-to-machine, inbound receive and outbound relay, a protocol we implement but do not design) and a REST/IMAP read-write API for clients.

SMTP (the mail protocols, summarized)

Inbound (a remote server delivering to us):
  MAIL FROM:<[email protected]>          -> SPF check on other.com vs source IP
  RCPT TO:<[email protected]>            -> validate bob exists; reject unknown (550) 
  DATA ... <message> ... .            -> accept body, DKIM verify, spool durably
  250 OK: queued as <spool_id>         -> ONLY after durable spool write

Outbound (us delivering to a remote server): our MTA acts as the SMTP client,
  DKIM-signs, connects to the recipient domain's MX, retries on 4xx, bounces on 5xx.

REST API (webmail / mobile / third-party clients)

GET  /api/v1/folders/{label}/messages?cursor=<message_id>&limit=50
  -> paginated list of message metadata in a folder, newest first.
  200: { "messages": [{message_id, thread_id, from, subject, snippet, flags, received_at}], "next_cursor": "..." }

GET  /api/v1/messages/{message_id}
  -> full message: headers, body (fetched from Body Blob Store), attachment refs.
  200: { "headers": {...}, "body_html": "...", "attachments": [{name, size, content_hash, mime}] }

GET  /api/v1/attachments/{content_hash}
  -> streams the attachment bytes (redirect to a signed CDN URL). Malware verdict enforced.

POST /api/v1/messages/send
  body: { "to": [...], "cc": [...], "subject": "...", "body_html": "...", "attachment_hashes": [...] }
  -> outbound spam check, DKIM-sign, write to Sent, enqueue to relay. 202 Accepted.

PATCH /api/v1/messages/{message_id}
  body: { "flags": {"read": true, "starred": false}, "add_labels": ["Work"], "remove_labels": ["Inbox"] }
  -> update metadata only (flags/labels). Read-your-writes for this user across devices.

GET  /api/v1/search?q=<query>&cursor=<...>&limit=50
  -> full-text search over THIS user's index shard. q supports from:/subject:/has:/after:/before:.
  200: { "message_ids": [...], "next_cursor": "..." }   (client then hydrates via /messages)

DELETE /api/v1/messages/{message_id}   -> move to Trash (soft delete); purge after 30 days.

Pagination is cursor-based on message_id (time-sortable), never offset - new mail arrives constantly, so offsets would skip or repeat rows. IMAP is offered as a second protocol surface over the same stores for legacy clients.

Data store choices, stated plainly

Why NoSQL + object store over one SQL database: the workload is partition-scoped per user, write-heavy, join-free, and demands petabyte horizontal scale plus an object tier for immutable blobs. SQL’s joins and cross-row ACID transactions are features we do not need here, and its single-writer scaling ceiling is exactly the wall we would hit at 50K writes/sec of 50KB blobs. There is no financial ledger in email that would justify strong relational consistency; per-user read-your-writes on flags is easily met with a per-partition write ordering, not global transactions.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. The SMTP edge under ~500K hostile conversations/sec (breaks first). An accept-everything server drowns in spam. Fix: horizontally scaled, stateless edge nodes behind DNS MX + anycast; cheapest-first layered rejection (IP reputation -> SPF -> greylisting -> body accept) so expensive work runs only on likely-legit mail; event-driven IO and per-IP connection caps to survive slowloris.

2. Durability handoff - never lose an accepted message. A crash after 250 OK but before storage loses mail. Fix: write the raw message to a quorum-replicated spool before returning 250. If we crash first, we never sent 250, and the sender retries (SMTP is store-and-forward). Dedup by Message-ID.

3. Storage growth, hundreds of PB/year, never shrinks (the defining cost). Fix: split metadata (small, hot) from bodies/attachments (large, blob store); content-address attachments to dedup 3-5x; immutable bodies cached forever; tier cold mail (old bodies/attachments) to cheap archival storage while keeping metadata hot for listing/search.

4. Search over billions of messages. LIKE scans and a single global index both collapse. Fix: inverted index sharded by user_id, built async off the ingest stream. Every query is single-shard, single-user - no scatter-gather, noisy-neighbor isolated. Soft-delete via tombstones + background compaction.

5. Metadata hot partition - the power user with millions of messages. Fix: the shard key is user_id, so one huge mailbox is one hot partition. Split such users by (user_id, label) or a secondary partition dimension so a single mailbox’s writes/reads spread across nodes.

6. Duplicate attachments - the same file mailed to millions. Fix: content-addressed storage keyed by SHA-256; store the bytes once, reference N times; malware-scan once per unique hash. This is the single biggest storage saving.

7. Spam classification blocking ingestion. Heavy ML inline caps throughput. Fix: cheap deterministic checks synchronous at the edge; expensive content/URL/attachment analysis async off the queue after 250 OK. Classification picks the folder; it does not gate acceptance. Feedback loop from 1.8B users’ “Report Spam” retrains continuously and fingerprint-matches live campaigns.

8. Outbound relay to servers we do not control. Remote MX can be down, slow, or greylisting. Fix: async relay queue with exponential backoff retries over hours/days; hard 5xx generates a bounce; per-destination-domain rate limiting so we look like a good sender (and are not throttled by the remote). Outbound spam checks stop a hijacked account from blasting.

9. Shard keys, stated plainly. Metadata -> user_id (a mailbox is one partition, single-shard reads). Search index -> user_id (single-user queries). Attachments/bodies -> content hash (dedup + immutable). Queue -> partition by recipient user so a user’s mail stays ordered. Sharding by time would be the classic mistake: it piles all current writes onto the “now” shard, a permanent hot spot, and scatters a user’s mailbox across every shard.

10. Single points of failure. Edge nodes are stateless and interchangeable; the spool, queue, metadata store, blob store, and index are all replicated (RF 3+). Delivery and indexer workers are stateless consumers - a dead worker’s partitions are reassigned and reprocessed from the durable queue. No single box whose loss stops mail flowing.

11. Multi-region for a global user base and disaster durability. Fix: pin a user’s mailbox to a home region (data residency + latency), replicate metadata and blobs cross-region asynchronously for the 11-nines durability guarantee. The SMTP edge is anycast-global so senders hit the nearest node; accepted mail is forwarded to the recipient’s home region via the durable queue. Reads serve from the home region; a regional outage fails over to a replica.

Wrap-Up

The trade-offs that define this design:

• Durability over latency, everywhere. Email is store-and-forward, so we spend seconds we do not have in chat on persisting-before-acknowledging: we return 250 OK only after a quorum-replicated spool write, taking permanent ownership of a message at the exact moment it is safely stored. Search and delivery are allowed to lag a few seconds.
• Split storage by access pattern. Metadata (small, hot, sorted, per-user-sharded), bodies (immutable, cached, tiered), and attachments (content-addressed, deduped, scanned once) are three different stores because they are three different workloads. Cramming them into one SQL table breaks on volume, listing cost, and duplication.
• Everything shards by user. Metadata and the search index both partition on user_id, so every read and every search a user does touches exactly one shard - no scatter-gather - and one huge mailbox cannot degrade anyone else. Search is per-user because there is no cross-user search to serve.
• Reject cheap and early at the front door; classify rich and late off the queue. The hostile ~500K/sec is beaten by cheapest-first layered rejection (IP/SPF/greylist) that discards the bulk before reading a body, while the expensive ML spam and malware analysis runs asynchronously so it never caps ingestion, fed by a 1.8B-user feedback loop.
• Content-addressing is the storage lever. Deduping attachments and immutable bodies by content hash is what turns an impossible petabyte bill into a merely enormous one, and it makes malware scanning once-per-unique-file for free.

One-line summary: a storage-and-search system with a hardened SMTP front door, where mail is durably spooled before it is acknowledged, split into per-user-sharded metadata plus content-addressed deduped bodies and attachments on tiered object storage, made searchable through a per-user inverted index built asynchronously off a durable ingest queue, and defended by cheapest-first edge rejection plus async ML spam filtering trained on a 1.8B-user feedback loop - all engineered to store hundreds of petabytes a year forever without ever losing a message.