Facebook Marketplace looks like “Craigslist with photos,” and if you design it that way you will fail the interview. The naive read is a CRUD app: a listings table, a search box, a chat window. But every one of those three has a scale trap hiding in it. The listing store is easy until you remember that “500M active listings” means half a billion rows that are constantly created, edited, marked sold, and expired, each with several photos that dwarf the metadata. The search box is easy until you realize the query is not “find listings matching iphone” but “find listings matching iphone, priced under 30000, in category Electronics, within 10km of where I am standing, sorted by relevance and recency” - a keyword search and a geospatial search and a set of filters, all at once, over 500M documents, at ~150k queries a second. And the chat window is easy until you count the fraud: Marketplace is one of the most heavily-abused surfaces on the internet, so a scammer posting a fake listing and a stolen-goods reshipping scheme are not edge cases, they are the main event.

So the spine of this design is not “store listings and let people search them.” It is three coupled hard problems - a write-once-read-many listing+media store, a combined geo-plus-keyword-plus-filter search index that stays fresh as listings churn, and a real-time messaging plane - wrapped in an asynchronous fraud-detection pipeline that scores every listing and every conversation without slowing the happy path. Get the search index and the fraud pipeline right and the rest is plumbing. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Post a listing. A seller creates a listing with a title, description, price, category, location (derived from their profile or a picked point), and 1-10 photos. They can edit it, mark it sold, or delete it. Listings expire after a set period (say 30 days) unless renewed.
  • Search nearby. A buyer searches by free-text keyword and filters by category, price range, and distance radius, and gets back relevant, nearby, in-stock listings sorted by a blend of relevance, distance, and recency. The default is “near me.”
  • Browse a category. With no keyword, a buyer browses a category (Electronics, Furniture, Vehicles) within a radius, paginated.
  • View a listing. Full detail: photos, price, description, seller’s public profile, distance from the buyer.
  • Message the seller. A buyer opens a conversation from a listing and exchanges messages with the seller in near real time. Messages are durable and tied to the listing.
  • Fraud detection. Every new/edited listing is scored for fraud (scam patterns, prohibited items, stolen goods, duplicate spam); risky listings are blocked, shadow-limited, or queued for review. Abusive message behavior is detected and rate-limited.

Explicitly out of scope (state this to own the scope):

  • Payments and checkout. Marketplace is largely a discovery-and-contact product; the actual transaction often happens in person with cash. Payment rails, escrow, and shipping logistics are a separate system.
  • The relevance/ranking ML model itself. We build the serving path for ranked search (index, filters, blended sort, features fed in) but do not design the learning-to-rank model. We treat the ranking score as a feature we can compute.
  • The social graph and identity. We consume “user_id, name, profile, rough home location” from the core Facebook identity service. We do not design signup, friend graph, or auth.
  • News Feed / ads integration. Injecting Marketplace items into Feed and ad auctions are separate surfaces.
  • Recommendations (“For you”). A personalized recommender is its own system; here the entry point is search and category browse.

The one decision that drives everything: this is a read-heavy discovery system whose defining query is a simultaneous geo + keyword + structured-filter search over a large, churning corpus, and whose defining risk is fraud - so the two components that get real engineering are the search index and the async fraud pipeline, while listing storage and messaging are solved with well-understood sharded stores.

Non-Functional Requirements (NFR)

  • Scale: ~1B monthly Marketplace users, ~250M daily active. 500M active listings at any time (older ones expired/archived). ~16M new/edited listings per day. Read-to-write ratio is enormous - hundreds of searches and views per listing created.
  • Latency: search results in under ~300ms p99; listing detail in under ~150ms; message send-to-deliver in under ~1s. Photo first-byte from CDN in tens of ms.
  • Availability: 99.95%+ on the read path (search, browse, view). Posting a listing and messaging can tolerate slightly lower availability and can degrade (queue the write, retry) without data loss.
  • Consistency: eventual consistency on the search path is fine. A new listing appearing in search a few seconds after it is posted is acceptable. Listing detail should be read-your-writes for the seller (they see their own edit immediately). Messaging is eventually consistent but ordered per conversation. The one thing that must be strongly correct is money-adjacent state, and we have punted payments, so nothing on the hot path needs a distributed transaction.
  • Durability: listings, photos, and messages are durable and replicated - losing a listing or a conversation is unacceptable. Search-index documents are derived and rebuildable from the source of truth, so the index can be non-durable/reconstructable.
  • Freshness: a posted listing should be searchable within a few seconds; a “sold” or “deleted” listing should stop appearing in search within a few seconds (stale “still available” results waste buyers’ time and, for scams, are actively harmful).

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they pick the architecture.

Listing writes (small):

500M active listings, ~30-day average lifetime.
New/edited listings ≈ 500M / 30 ≈ 16.6M writes/day
= 16.6M / 86,400 ≈ 190 writes/sec average
Peak (say 3x, evening) ≈ 600 writes/sec.

That write rate is trivial - a single well-sharded database handles it. Writes are not the problem here; the corpus size and the read fan-out are.

Search + browse reads (the load that matters):

250M DAU, each doing ~20 search/browse/view actions per active day
= 5B read actions/day
= 5,000,000,000 / 86,400 ≈ 58,000 reads/sec average
Peak (2.5x) ≈ 150,000 reads/sec.
Of these, roughly half are geo+keyword searches (the expensive query),
the rest are listing-detail views (cheap point reads).
So ~75k expensive search QPS at peak.

75k complex geo+text queries/sec against 500M documents is the number that dictates the search tier: it must be a horizontally-sharded inverted-index engine with heavy caching, not a database WHERE clause.

Photos (the storage firehose):

16.6M new listings/day * ~4 photos avg = ~66M photo uploads/day.
Post-compression avg ~300KB (plus a couple of resized variants ~ +150KB)
Per photo stored ≈ 450KB across variants.
Daily photo bytes ≈ 66M * 450KB ≈ 30 TB/day of new media.

Active photo footprint:
500M listings * 4 photos * 450KB ≈ 900 TB live in object storage.
Add archived/expired retained a while: multi-petabyte object store.

Listing metadata storage:

Per listing ≈ 2KB (title, desc, price, category, geo, seller_id, status, ts).
500M * 2KB = 1 TB active metadata. Trivial for a sharded store;
with history/archive and indexes call it a few TB.

Messages:

Say 100M messages/day (buyers contacting sellers, back-and-forth).
= 100M / 86,400 ≈ 1,160 msg/sec average, peak ~5,000/sec.
Per message ≈ 300B stored.
Yearly message bytes ≈ 100M * 365 * 300B ≈ 11 TB/year. Durable, sharded.

Cache + bandwidth:

Search cache: cache the hot (region, category, keyword, filter) result
sets. A large fraction of searches are popular repeats
("sofa near me", "iphone"), so a ~500GB-1TB result cache absorbs
a big chunk of the 75k search QPS.

Photo egress: served entirely from CDN. If avg listing view pulls
~1.5MB of images and there are ~2.5B views/day, that is ~3.75 PB/day
of image egress - all CDN, essentially none hitting origin.

The takeaways that shape the design: writes are tiny (~600/sec) but the corpus is 500M documents searched at ~75k complex QPS; photos are a multi-petabyte object-store + CDN problem, never touching the app servers’ write path; messages are a modest but durable sharded store; and search results and images are cache/CDN-dominated. The search index and the media pipeline, not the listing database, are the drivers.

High-Level Design (HLD)

The system splits into planes: a write plane (listing service + media upload) that is the source of truth, a search plane (a geo+text inverted index kept fresh from listing changes) that serves discovery, a media plane (object store + CDN + async processing), a messaging plane (persistent connections + durable per-conversation store), and a fraud plane (an async scoring pipeline fed by every listing and message event). Changes to listings flow through a change stream that fans out to the search index and the fraud pipeline, so those stay decoupled from the write path.

                         ┌───────────────┐
   Buyers / Sellers ───▶ │  API Gateway  │ ── auth, rate-limit, routing
                         └──────┬────────┘
        ┌───────────────┬───────┼──────────┬───────────────┐
        ▼               ▼       ▼          ▼               ▼
 ┌────────────┐  ┌────────────┐ │  ┌──────────────┐ ┌──────────────┐
 │  Listing    │  │  Search    │ │  │  Messaging   │ │  Media       │
 │  Service    │  │  Service   │ │  │  Service     │ │  Upload Svc  │
 │ (write/read)│  │ (read only)│ │  │ (WS + store) │ │ (presign)    │
 └─────┬───────┘  └─────┬──────┘ │  └──────┬───────┘ └──────┬───────┘
       │ write          │ query  │         │ msgs          │ presigned
       ▼                ▼        │         ▼               ▼
 ┌────────────┐   ┌────────────┐ │  ┌──────────────┐ ┌──────────────┐
 │ Listing DB  │   │ Search      │ │  │ Message Store│ │ Object Store │
 │ (sharded    │   │ Index       │ │  │ (Cassandra,  │ │ (S3-like) +  │
 │  by list_id)│   │ (Lucene/ES, │ │  │ by convo_id) │ │  CDN         │
 └─────┬──────┘    │ region-     │ │  └──────────────┘ └──────┬───────┘
       │ CDC       │ sharded)    │ │                          │ event
       ▼           └─────▲──────┘ │                           ▼
 ┌──────────────────────┐│        │                  ┌──────────────┐
 │ Change stream (Kafka) ├┼────────┼─────────────────▶│ Media proc:  │
 │ topic: listing_events ││        │                  │ resize/thumb │
 └──────────┬───────────┘│        │                  │ /phash       │
            │ consume     │ index  │                  └──────────────┘
            ▼             │ upsert │
 ┌──────────────────────┐ │        │
 │  Fraud Pipeline       │─┘        │
 │ (rules + ML scoring,  │◀─────────┘  message events
 │  image hash, velocity)│
 │  -> block/shadow/queue│
 └──────────────────────┘

Post-a-listing flow:

  1. Seller fills the form; the client first calls the Media Upload Service to get presigned URLs and uploads each photo directly to the object store (bytes never pass through app servers). Upload triggers async media processing (resize, thumbnails, perceptual hash).
  2. Client POST /listings with metadata + the uploaded photo keys. The Listing Service validates, writes the row to the Listing DB (sharded by listing_id), status = pending_review or active.
  3. The write emits a listing_created event to the Kafka change stream (via CDC/outbox so the event is never lost).
  4. Two consumers act on it independently: the Search indexer upserts the listing into the search index (making it searchable within seconds), and the Fraud pipeline scores it. If fraud scores it risky, it flips status to blocked/shadow and re-emits an event that removes/limits it in the index.

Search flow:

  1. Buyer GET /search?q=sofa&lat&lng&radius=10km&category&price_max. The API Gateway routes to the Search Service.
  2. Search Service picks the index shard(s) for the buyer’s region (geo-sharded), runs the combined query (geo_distance filter + keyword match + structured filters), applies the blended ranking (relevance x distance x recency), and returns a page of listing summaries (title, price, thumbnail URL, distance).
  3. Popular (region, keyword, filter) result pages are served from a result cache; thumbnails come from the CDN. Detail views are point reads from the Listing DB (cached).

Messaging flow: buyer opens a chat on a listing; the Messaging Service holds a WebSocket, writes each message durably to the per-conversation store, and delivers to the other party’s connection (pub/sub routing across the WS fleet, like any chat system). Message events also feed the fraud pipeline for abuse detection.

The key insight: the source-of-truth listing write is small and simple; everything expensive (search freshness, fraud scoring, media processing) hangs off an async change stream so it never blocks or slows the post, and the read path is served by a purpose-built geo+text index and a CDN, not by the write database.

Component Deep Dive

The hard parts, naive-first then evolved: (1) geo + keyword + filter search at 500M listings, the core; (2) photo upload and serving; (3) buyer-seller messaging; (4) the fraud pipeline.

1. Geographic + keyword + filter search over 500M listings (the core)

The query is “keyword sofa, category Furniture, price under 20000, within 10km of (lat,lng), best first.” Three different kinds of predicate at once.

Naive approach: one SQL query with a bounding box. Store listings in Postgres, and:

SELECT * FROM listings
WHERE category = 'Furniture' AND price <= 20000 AND status = 'active'
  AND lat BETWEEN :minLat AND :maxLat
  AND lng BETWEEN :minLng AND :maxLng
  AND to_tsvector(title||description) @@ to_tsquery('sofa')
ORDER BY <some rank>
LIMIT 40;

Where it breaks:

  • No single index serves all three predicates. A B-tree on (category, price), a GiST index on geo, and a full-text (GIN) index on the text are three separate indexes; the planner picks one and post-filters the rest, scanning far more rows than the 40 returned. At 500M rows and 75k QPS this melts.
  • Bounding box is not distance. A lat/lng box is a rectangle, not a circle; you still compute haversine per row to filter to a true radius, and boxes distort badly near the poles and the antimeridian.
  • Ranking needs a scan. A relevance-x-distance-x-recency sort over the whole candidate set means computing a score per matching row before you can LIMIT, so you cannot stop early.
  • Text search in a relational DB is a bolt-on. GIN full-text works for small corpora; it is not a competitive scoring engine at half a billion docs.

First evolution: a geohash/S2 prefix index to make geo cheap. Encode each listing’s location as a geohash (or S2 cell id) and index the prefix. A radius query maps to “the cell covering the point plus its neighbor cells at the right precision,” which becomes a set of prefix-range lookups instead of a full box scan, and you only haversine the small candidate set. This fixes geo but you still have to intersect it with keyword and filters, and you are still bolting text search onto a store that is not built for it.

The answer: a dedicated inverted-index search engine (Lucene / Elasticsearch / a Lucene-based service) with geo, filters, and text as first-class, geo-sharded, kept fresh by the change stream. Each listing is a document with an inverted index over its tokens, a geo field (indexed as a BKD-tree / geohash grid so geo_distance is native), and doc-values for structured filters (category, price, ts, status) and for sort. The engine evaluates all predicates against posting lists and skips non-matching docs cheaply, so it touches a tiny fraction of the corpus per query.

Document (search index):
  listing_id, title_tokens[], desc_tokens[], category, price,
  location (geo_point), geohash_cell, created_at, status,
  seller_id, rank_features{popularity, seller_score, ...},
  thumb_url

Query pipeline:
  1. filter clause  : status=active AND category=Furniture
                      AND price<=20000 AND geo_distance(loc, point) <= 10km
                      (cheap, cacheable bitset, no scoring)
  2. must clause     : match(title/desc, "sofa")   (scored, BM25)
  3. sort/rank       : blended = w1*text_score + w2*(1/dist) + w3*recency
  4. paginate        : top-K with search_after (cursor), not deep OFFSET

Why sharding is by geography, not by listing_id. A buyer’s search is always local (“near me”), so if the index is sharded by region, a query fans out to only the one or few shards covering the buyer’s radius instead of every shard. We shard by a coarse geo cell (for example S2 level ~4-6, roughly metro-to-region sized) so each shard holds the listings for a region. Cross-region searches (buyer near a shard boundary) hit 2-4 neighbor shards and merge. This keeps query fan-out bounded and lets hot regions (a dense metro) get more replicas.

ApproachGeoTextFilters500M docs @ 75k QPS
SQL bounding box + GINbox, post-filterbolt-onseparate indexesscans too much, melts
Geohash prefix indexgoodstill separateseparategeo fixed, text still weak
Inverted engine, geo-shardednative geo_distancenative BM25doc-value bitsetsfan-out bounded, caches well

Freshness. The indexer consumes listing_created / updated / sold / deleted / blocked events off the change stream and upserts/deletes the document, so a new listing is searchable in seconds and a sold/blocked one drops out in seconds. The index is derived and fully rebuildable by replaying listings from the source DB, so an index shard loss is a rebuild, not data loss.

Ranking without a full scan. The blended score uses the engine’s native scoring plus doc-values for distance and recency; top-K with a heap and search_after cursors avoids deep-offset pagination (which forces the engine to skip and discard millions of docs). Relevance features (seller reputation, listing popularity) are precomputed and stored on the doc, not computed at query time.

The elegant part: we stop trying to make one database serve three predicate types, and instead use an inverted-index engine where geo, text, and filters are all native, sharded by region so local searches stay local, and kept fresh by the async change stream rather than by writing search-side on the hot path.

2. Photo upload and serving

Each listing is mostly photos, and photos are 1000x the size of the metadata.

Naive approach: multipart-upload the images to the Listing Service, which writes them to the DB (or a mounted disk) alongside the row. The app server receives the bytes and persists them.

Where it breaks:

  • App servers become byte pumps. 30TB/day of uploads flowing through stateless app servers wastes their CPU/network on copying blobs and couples the write path’s latency to image size and the user’s upload speed.
  • Databases hate blobs. Storing 900TB of images as DB rows/BLOBs bloats the store, kills backup/restore times, and wrecks cache locality for the tiny metadata you actually query.
  • Serving from origin does not scale. Serving 3.75PB/day of image egress from your app tier is impossible; images must come from a CDN.

The answer: direct-to-object-store upload via presigned URLs, async processing, and CDN serving.

  • Presigned upload. Client asks the Media Upload Service for a presigned PUT URL per photo; the client uploads directly to the object store (S3-like). Bytes never touch app servers. The service returns opaque object keys the client attaches to the listing.
  • Async processing on upload event. Each object write fires an event; a media-processing consumer generates resized variants and thumbnails (so the search grid loads a 20KB thumb, not a 4MB original), strips EXIF/GPS metadata (privacy), and computes a perceptual hash (pHash) of each image for the fraud pipeline (duplicate-detection and known-scam-image matching).
  • CDN in front. Processed variants are served through a CDN with long TTLs and content-addressed URLs (/media/{hash}/{variant}.jpg), so 99%+ of image egress is edge-cached and origin sees almost nothing.
  • Validation and safety. Processing also runs NSFW/prohibited-content image classification (feeding the fraud plane) before the listing is allowed to go fully active.
Client -> MediaUpload: request 4 presigned PUT URLs
Client -> ObjectStore: PUT each photo directly (no app server in path)
ObjectStore -> event -> MediaProc:
    resize -> {thumb 320px, card 640px, full 1080px}
    strip EXIF/GPS; compute pHash; classify content
    write variants back to ObjectStore; emit media_ready{listing_id, keys}
Client -> Listing: POST listing with photo keys
Serve: browser -> CDN -> (miss) -> ObjectStore origin

The lesson: large binary media never flows through your application or database tier - it goes straight to object storage via presigned URLs, is processed asynchronously into cache-friendly variants, and is served from a CDN, leaving your services to handle only the small metadata.

3. Buyer-seller messaging

A buyer taps “Message seller” and they chat about the sofa.

Naive approach: a messages table and the client polls it. POST /messages inserts a row; the recipient’s app polls GET /messages?since=... every few seconds.

Where it breaks:

  • Polling is wasteful and laggy. Millions of open chats polling every few seconds is a constant load whether or not anything changed, and still delivers messages seconds late.
  • A single messages table is a hot, unsharded blob. Ordered reads of a conversation and high write volume on one relational table do not scale, and cross-partition sorting for “my conversations, newest first” gets ugly.
  • No delivery/read state, no presence. Buyers want “seen,” “delivered,” and “seller is typing,” which polling handles poorly.

The answer: persistent connections for delivery, a per-conversation durable store for history, routed through a pub/sub bus across the WS fleet. This is the well-trodden chat pattern, adapted:

  • WebSocket delivery. Each active client holds a WebSocket to a Messaging Service edge server. Sending a message writes it durably, then publishes it to the recipient’s channel; the bus routes it to whatever WS server holds the recipient (a connection registry user_id -> ws_server, TTL-heartbeated), which pushes it down the socket. Offline recipients get a push notification.
  • Durable per-conversation store, keyed by conversation_id. A conversation is (listing_id, buyer_id, seller_id). Messages are stored in a wide-column store (Cassandra-style) partitioned by conversation_id and clustered by message timestamp/sequence, so “load this conversation, newest first” is a single-partition ordered read. A separate inbox table per user lists their conversations sorted by last-activity for the conversation list.
  • Ordering. Per-conversation monotonic sequence numbers (assigned by the write) give a total order within a conversation; that is all the ordering a chat needs. Cross-conversation ordering does not matter.
  • State. Delivered/read receipts and typing indicators are lightweight events over the same socket; read state is stored per (conversation, user).
Conversation key: convo_id = hash(listing_id, buyer_id)
Send:
  1. Messaging Svc: append msg to messages[convo_id] (seq = next(convo))
  2. update inbox[buyer], inbox[seller] last_activity
  3. publish channel:{recipient} -> bus -> recipient's WS server -> socket
  4. if recipient offline -> push notification
Load: single-partition read messages[convo_id] ORDER BY seq DESC LIMIT 50

The lesson: messaging reuses the standard chat architecture - persistent sockets plus a recipient-keyed pub/sub bus for live delivery, and a per-conversation-sharded durable store for ordered history - because buyer-seller chat is chat, and the only Marketplace-specific twist is that the conversation is anchored to a listing.

4. The fraud pipeline (the risk that defines the product)

Marketplace is a magnet for scams: fake listings, prohibited items, stolen goods, phishing in messages, price-too-good bait, duplicate spam across accounts. Fraud is not a feature to bolt on; it is a core plane.

Naive approach: synchronous rule checks at post time, plus human review of reports. When a listing is posted, run a few if rules inline and block if any fire; otherwise publish it and rely on users reporting bad ones.

Where it breaks:

  • Synchronous scoring slows the post and cannot be deep. Rich checks (image similarity against known-scam images, cross-account velocity, text classification, seller-history models) are too slow to run inline on the post request. Doing them synchronously either times out the user or forces shallow checks.
  • Report-and-review is too late. By the time users report a scam listing, buyers have been defrauded. Reactive-only moderation loses.
  • Rules alone are brittle. Scammers adapt; static rules get evaded. You need models and signals that update.

The answer: an asynchronous, multi-stage scoring pipeline fed by the change stream, that can block, shadow-limit, or queue for human review, with a fast synchronous pre-gate for the obvious cases.

  • Fast synchronous pre-gate. At post time, cheap deterministic checks run inline: banned keywords, prohibited categories (weapons, drugs), obviously malformed price, an account under a hard block. These reject the blatant cases immediately. Everything else is admitted as pending/active-limited and scored asynchronously.
  • Async multi-signal scoring. The listing_created event drives a scoring service that combines:
    • Content signals: text classifier over title/description (scam phrasing, off-platform-payment lures), image classifier (prohibited/NSFW), and perceptual-hash matching of the photos against a store of known-scam and stock/duplicate images (a listing whose “own” photo is a stock image or matches 50 other listings is a strong scam signal).
    • Behavioral / velocity signals: how many listings this seller posted in the last hour, account age, prior takedowns, device/IP reputation, whether many accounts post the same image or text (a cross-account duplicate-detection join on pHash and text-shingles).
    • Graph signals: clusters of accounts sharing devices, IPs, or images (fraud rings), computed in a slower batch layer.
  • Actions by score band. High score -> block (status flips, the change stream removes it from the search index). Medium -> shadow-limit (visible to the seller, suppressed or down-ranked in search, sometimes region-limited) and queue for human review. Low -> allow. The action is applied by writing listing status and letting the same change-stream mechanism update the index, so enforcement reuses the freshness path.
  • Message-abuse detection. Message events feed a lighter classifier for phishing/off-platform-payment scams and spam velocity; abusive senders get rate-limited or blocked, protecting buyers mid-conversation.
  • Feedback loop. User reports and human-review decisions are labeled data that retrains the models; confirmed-scam images/text are added to the pHash/shingle blocklists so the next identical attempt is caught by the cheap pre-gate.
post -> sync pre-gate (banned words, prohibited cat, hard-blocked acct)
     -> if pass: status=active-limited, emit listing_created
listing_created -> Fraud scoring:
     content(text+image classifiers) + pHash match(known-scam/dup)
     + velocity(seller rate, acct age, device/IP rep)
     + ring signals(batch)
     -> score
        high   -> status=blocked   -> change stream removes from index
        medium -> status=shadow    -> down-rank + human review queue
        low    -> status=active
report/human-decision -> labels -> retrain + extend blocklists

The point: fraud scoring is asynchronous and multi-signal so it can be deep without slowing the post, with a cheap synchronous pre-gate for the blatant cases; enforcement is just a listing-status change that rides the same change stream the search index already consumes, so blocking a scam removes it from search within seconds using machinery we already built.

API Design & Data Schema

REST for listings/search, WebSocket for messaging, presigned URLs for media.

Listings

POST /api/v1/listings
  body: { title, description, price, currency, category,
          location:{lat,lng} | use_profile_location:true,
          photo_keys:["media/ab12...","media/cd34..."] }
  -> 201 { listing_id, status:"active" | "pending_review" }

PATCH /api/v1/listings/{listing_id}
  body: { price?, description?, status?:"sold"|"active", photo_keys? }
  -> 200 { listing_id, status }

DELETE /api/v1/listings/{listing_id}    -> 204
GET    /api/v1/listings/{listing_id}
  -> { listing_id, title, description, price, category, photos:[urls],
       seller:{id,name,rating}, location_approx, distance_m, status,
       created_at }

Search and browse

GET /api/v1/search
  query: q, lat, lng, radius_m=10000, category?, price_min?, price_max?,
         sort=relevance|distance|recent, cursor?, limit=40
  -> { results:[ { listing_id, title, price, thumb_url,
                   distance_m, created_at } ], next_cursor }

GET /api/v1/browse
  query: category, lat, lng, radius_m, cursor, limit   (no keyword)
  -> same shape, sorted by recency+distance

Media

POST /api/v1/media/presign
  body: { count:4, content_type:"image/jpeg" }
  -> { uploads:[ { put_url, object_key, expires_at } ] }
// client PUTs bytes directly to put_url (object store), then posts keys

Messaging

// WebSocket frames
{ "type":"SEND", "listing_id":"l_9", "to":"seller_7", "text":"still available?" }
{ "type":"MESSAGE", "convo_id":"c_5", "from":"buyer_3", "seq":42,
  "text":"yes, 15000", "ts":1752... }
{ "type":"READ", "convo_id":"c_5", "up_to_seq":42 }

GET /api/v1/conversations            -> user's inbox, newest first
GET /api/v1/conversations/{convo_id}/messages?before_seq=&limit=50

Data stores - the right tool per plane

1. Listing store - sharded relational or wide-column, by listing_id. The source of truth. Each listing is a self-contained record with no cross-listing transaction, so it shards cleanly on listing_id (hash). A relational store (sharded MySQL/Postgres) or a wide-column store both work; the deciding factor is that we need durable, point-read-friendly, easily-CDC’d rows, not joins. An outbox/CDC feed emits every change to Kafka.

Table: listings              (shard key = listing_id)
  listing_id     BIGINT   PK
  seller_id      BIGINT   INDEX
  title          TEXT
  description    TEXT
  price          BIGINT   -- minor units
  currency       CHAR(3)
  category       INT      INDEX
  lat, lng       DOUBLE
  geohash        CHAR(9)  INDEX     -- for source-side coarse geo
  photo_keys     JSON
  status         ENUM(active,pending,shadow,blocked,sold,expired,deleted)
  fraud_score    FLOAT
  created_at     TIMESTAMP
  expires_at     TIMESTAMP  INDEX   -- for TTL/expiry sweep
  Secondary access: "my listings" by seller_id (index or a by-seller table)

2. Search index - inverted-index engine (Lucene/Elasticsearch), geo-sharded, derived. Documents as in deep dive 1, sharded by coarse geo cell, replicated per region, rebuildable from the listing store. NoSQL/search-engine, not relational: we need native geo_distance + BM25 + doc-value filters at 75k QPS, which a relational store cannot give.

3. Object store + CDN - media. S3-like object storage for originals and variants (multi-petabyte, 11-nines durability), fronted by a CDN with content-addressed keys. Never in a database.

4. Message store - wide-column (Cassandra-style), by conversation_id. Ordered per-conversation writes and single-partition history reads; hard to beat with relational at this write volume and access pattern. Plus a per-user inbox table (partition = user_id) of conversations by last-activity.

Table: messages        (partition key = convo_id, cluster = seq DESC)
  convo_id   TEXT      PARTITION KEY
  seq        BIGINT    CLUSTERING KEY
  from_id    BIGINT
  text       TEXT
  ts         TIMESTAMP
Table: inbox           (partition key = user_id, cluster = last_activity DESC)
  user_id       BIGINT PARTITION KEY
  convo_id      TEXT   CLUSTERING KEY
  listing_id    BIGINT
  other_user_id BIGINT
  last_activity TIMESTAMP
  unread_count  INT

5. Fraud state + blocklists - KV + a batch/graph store. Seller velocity counters and device/IP reputation in a fast KV (Redis), known-scam pHashes and text shingles in a searchable blocklist, and fraud-ring graph computation in a batch layer (Spark/graph store).

6. Cache - Redis/memcached. Hot search result pages (keyed by normalized query+region+filters), hot listing details, and the connection registry for messaging.

Why the mixed SQL/NoSQL split: the listing store wants durable, point-addressable, CDC-friendly rows (relational or wide-column, sharded by id); search wants a specialized inverted+geo engine (not a general DB at all); messages want per-conversation ordered partitions (wide-column); media wants object storage; fraud counters want a fast KV. There is no single store that is good at all five, and crucially none of the hot paths needs a cross-entity transaction - so we pick the right specialized store per plane and glue them with an async change stream.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Search read load (breaks first). ~75k complex geo+text queries/sec against 500M docs is the dominant load. Fix: the geo-sharded inverted-index engine so each query hits only the shard(s) covering the buyer’s radius; replicate hot regions to add read capacity where the queries are; and a result cache for popular (region, keyword, filter) pages, which absorbs a large fraction of repeated searches (“sofa near me”).

2. Hot regions / hot searches. A dense metro or a viral item (a hot concert ticket resale, a popular game console at launch) concentrates queries on a few shards and a few keys. Fix: more replicas for hot geo shards (replicate, do not just re-shard, since the corpus is fixed but the read rate spikes), cache the hot result pages with short TTLs, and request-coalesce identical in-flight queries so a spike collapses into one backend call.

3. Index freshness lag. If the change-stream consumer falls behind, sold/blocked listings linger in search - actively harmful for scams. Fix: keep the indexer lightweight and horizontally scaled per shard; prioritize delete/block/sold events over create/update in the pipeline so takedowns are never delayed; monitor consumer lag and alert. Index writes are idempotent upserts keyed by listing_id, so replay is safe.

4. Photo storage and egress. Multi-petabyte media and multi-petabyte daily egress. Fix: direct-to-object-store uploads (app tier never touches bytes), CDN for all serving (origin sees <1%), aggressive variant sizing (thumbnails for the grid), and lifecycle policies moving expired-listing media to cold storage.

5. Write-path coupling. Doing search-indexing, fraud-scoring, and media-processing inline would make posting a listing slow and fragile. Fix: the async change stream (Kafka via outbox/CDC) decouples all derived work from the write; the post returns as soon as the durable row is written, and indexing/fraud/media happen downstream. A slow fraud model can never slow a post.

6. Fraud pipeline lag and evasion. Deep scoring is slow; scammers adapt. Fix: the synchronous pre-gate catches blatant cases instantly; async scoring handles the deep checks and applies enforcement via status change; a feedback loop turns confirmed scams into cheap-to-catch blocklist entries. Scale the scoring consumers independently; a scoring backlog delays enforcement, not posting, and shadow-limiting new sellers by default bounds the risk of the lag.

7. Messaging fan-out and connection scale. Millions of concurrent chats across a fleet where buyer and seller sit on different WS servers. Fix: recipient-keyed pub/sub bus plus a connection registry (user -> ws server, TTL-heartbeated); each WS server subscribes only to its users’ channels. Message store shards by conversation_id, spreading writes evenly. Offline delivery falls back to push notifications.

8. Hot keys - a viral listing. One listing (a rare item, a scam that went viral) gets a disproportionate share of detail views and messages. Fix: cache the listing detail aggressively (it is read-mostly), serve its photos from CDN, and for the messaging surge on one seller, rate-limit inbound contacts per listing and shard the seller’s inbox writes. The listing row itself is a cached point read, so read amplification lands on cache, not the DB.

9. Listing expiry and archival. 500M active listings with 16M/day churn need timely expiry so search does not fill with dead listings. Fix: an expiry sweep on expires_at (or a TTL) flips listings to expired and the change stream removes them from the index; expired media moves to cold storage. Archival keeps the active corpus at ~500M rather than growing unbounded.

10. Single points of failure and multi-region. Fix: every plane is replicated and, where derived (search index, caches), rebuildable from the source of truth. Deploy per geographic region and route users to their home region, which is natural here because search is inherently local - a buyer’s radius sits in one region’s shards, so the latency-critical search path stays local and cross-region traffic is rare. The listing store, message store, and object store are cross-region replicated for durability; loss of a search shard is a rebuild, not an outage, and reads fail over to a replica.

Wrap-Up

The trade-offs that define this design:

  • A specialized geo+text search engine over a database query. We refuse to make one relational store serve keyword, geo, and structured filters at once; instead a geo-sharded inverted-index engine makes all three native and keeps local searches local, at the cost of running a second, derived, eventually-consistent store fed by a change stream.
  • Async everything-derived over a fat write path. The listing write is tiny and simple; search indexing, fraud scoring, and media processing all hang off an async change stream, so posting is fast and a slow model or a lagging indexer never blocks a seller - at the cost of seconds of eventual consistency between posting and being searchable.
  • Media out of the application entirely. Photos go straight to object storage via presigned URLs and are served from a CDN, so the petabyte-scale media problem never touches app servers or the database - the app tier only ever handles the 2KB of metadata.
  • Fraud as a first-class asynchronous plane with a synchronous pre-gate. Blatant abuse is blocked inline; deep multi-signal scoring runs async and enforces by flipping listing status, reusing the same freshness path the search index already consumes - so catching a scam removes it from search in seconds without new machinery.
  • Right store per plane, glued by a change stream, no distributed transactions. Sharded listing store, inverted search index, wide-column messages, object-store media, KV fraud counters - each the right tool, none needing a cross-entity transaction, because we deliberately kept payments out of scope and made every hot path a single-entity read or write.

One-line summary: a read-heavy discovery system where tiny, durable listing writes fan out through an async change stream to a geo-sharded inverted-index search engine (native geo + keyword + filters, kept fresh in seconds), photos flow direct-to-object-store and serve from CDN, buyer-seller chat runs on the standard sockets-plus-pub/sub chat pattern, and an asynchronous multi-signal fraud pipeline scores every listing and message and enforces by a status change that rides the same freshness path - everything eventually consistent, nothing needing a distributed transaction.