An online auction looks like “a listing with a current price that goes up,” and if you model it that way you will lose the interview in the first five minutes. The trap is that an auction is not a CRUD row you update - it is a serialization problem wearing a shopping-site costume. The moment two people bid on the same item at the same instant, you have a race: which bid is higher, which one wins, what is the new current price, and did the loser get told they were outbid. Multiply that by a “hot” item - a rare sneaker drop, a graded Charizard, a concert ticket - where thousands of people pile in during the last ten seconds, and the naive “read price, compare, write price” pattern shreds itself into lost updates and double-wins. Then remember there are 100M active listings and each one has a hard deadline, and every one of those deadlines must fire close to on time, or the auction that “ended at 9:00:00” is still taking bids at 9:00:04 and someone is furious.

So the spine of this design is not “store items and let people bid.” It is three coupled hard problems - a strictly-serialized per-auction bid path that stays correct under thousands of concurrent bids, a deadline-firing mechanism that closes 100M auctions at their exact scheduled times, and a read/broadcast plane that shows millions of watchers the live current price without hammering the bid path. Get bid serialization and auction-close right and the rest is plumbing. Let me build it properly.

Functional Requirements (FR)

In scope:

  • List an item. A seller creates an auction listing: title, description, photos, category, starting price, optional reserve price, optional buy-it-now price, and an end time (duration). The listing goes live and is browsable/searchable.
  • Place a bid. A buyer bids an amount on an active auction. A bid is accepted only if it beats the current highest bid by at least the minimum increment (and clears the reserve to actually win). The system tracks the current highest bid, the highest bidder, and full bid history.
  • Proxy / automatic bidding. A buyer sets a maximum they are willing to pay; the system bids on their behalf up to that max, incrementing only as needed to stay in the lead. This is eBay’s core mechanic and it changes the concurrency story.
  • Watch an auction. Buyers watch an item and see the current price, time remaining, and bid count update live. Hot items have millions of concurrent watchers.
  • End the auction. At the scheduled end time the auction closes: no more bids accepted, the highest bid above reserve wins, winner and seller are notified, and an order is created.
  • Buy It Now. If set, a buyer can end the auction instantly by paying the fixed price (subject to no bids yet, or bids below the BIN threshold, per policy).
  • Anti-snipe (soft close). Optionally, a bid in the final seconds extends the end time by a short window so late bids do not silently steal the item at the buzzer.

Explicitly out of scope (state this to own the scope):

  • Payment capture, escrow, shipping, and dispute resolution. Winning creates an order; charging the card, holding funds, generating shipping labels, and handling returns are separate systems. We hand off to them.
  • Search relevance/ranking ML. We build the serving path for browse/search but do not design the learning-to-rank model.
  • Fraud/trust ML models. We note where fraud scoring hooks in (shill bidding, fake listings) but do not design the models.
  • Identity, auth, seller onboarding, and the social/feedback graph. Consumed from core services.
  • Recommendations and “similar items.” A separate personalization system.

The one decision that drives everything: an auction is a linearizable, single-writer state machine per item (bids must be totally ordered), with a hard real-time deadline, read by a massive fan-out audience - so the two things that get real engineering are the per-auction serialized bid path and the deadline scheduler, while listing storage, search, and media are solved with well-understood patterns.

Non-Functional Requirements (NFR)

  • Scale: 100M active listings at any time. Say ~200M registered active buyers, ~50M DAU. Most auctions get a handful of bids; a small fraction of “hot” items get thousands, concentrated in the final seconds.
  • Latency: placing a bid should feel instant - under ~200ms p99 for the accept/reject response. Live price updates to watchers within ~1-2s. Auction close must fire within ~1s of the scheduled end time (soft close makes small jitter tolerable, but seconds-late is unacceptable for the last-second race).
  • Availability: 99.95%+ on the read/browse path. The bid path must be highly available and correct - during a hot auction’s final seconds, a bidding outage is a business disaster.
  • Consistency: the bid path must be strongly consistent (linearizable) per auction. “Current highest bid” and “who is winning” cannot be eventually consistent - two people cannot both be told they are the high bidder. Across different auctions there is no shared state, so we only need strong consistency within one auction, which is what makes this scalable. Browse/search/watch can be eventually consistent (a watcher seeing the price 1s late is fine).
  • Durability: every bid is durable and auditable. Losing a bid, or the record of who bid what and when, is unacceptable - this is money-adjacent and legally sensitive (bid history is evidence in disputes). Auctions and their outcomes are durable and replicated.
  • Fairness / correctness at close: the winner must be deterministic and defensible: highest bid above reserve, ties broken by earliest timestamp. No lost bids, no double-wins, no accepting a bid after close.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they pick the architecture.

Listings:

100M active listings, average auction duration ~7 days.
New listings/day ≈ 100M / 7 ≈ 14.3M new listings/day
= 14.3M / 86,400 ≈ 165 listings/sec average, peak ~500/sec.
Listing metadata ≈ 3KB each (title, desc, prices, category, times, seller).
100M * 3KB = 300 GB active metadata. Trivial for a sharded store.

Listing creation is a tiny write rate. The item corpus itself is small in bytes. Not the problem.

Bids (the load that matters):

Assume avg ~10 bids per auction over its life (most get few, some get thousands).
14.3M auctions ending/day * 10 bids ≈ 143M bids/day
= 143M / 86,400 ≈ 1,650 bids/sec average.

But bids are NOT uniform. They bunch at the end, and hot items dominate.
A single hot auction can take THOUSANDS of bids in the final 10 seconds:
say 5,000 bids in 10s = 500 bids/sec ON ONE AUCTION KEY.
And thousands of hot auctions can close in the same evening window.
Peak aggregate bid rate ≈ 50,000-100,000 bids/sec across the fleet,
with dangerous per-key concentration on the hot items.

The absolute bid rate (~1.6k/sec average, ~100k/sec peak) is not scary as a total. The scary part is per-auction concentration: 500+ bids/sec landing on one key that must serialize them. That single number is what dictates the whole bid-path design.

Auction closings (the deadline firehose):

14.3M auctions end/day = 14.3M / 86,400 ≈ 165 closings/sec average.
But listings cluster on round end times (people list for "7 days" and
sellers favor Sunday evenings), so closings spike:
a popular evening minute might see 50,000-100,000 auctions all
scheduled to end within the same 60 seconds.
Each close = check reserve, pick winner, notify, create order.
So the scheduler must fire ~100k timers in a tight window, reliably.

Watch / read fan-out (the audience):

50M DAU * ~30 item-views/watch-refreshes per day ≈ 1.5B reads/day
= 1.5B / 86,400 ≈ 17,000 reads/sec average, peak ~50,000/sec.
A single hot auction can have MILLIONS of concurrent watchers,
each wanting live price updates. Pushing every price change to
2M watchers over a 10-second bid storm = 2M * (say 50 updates)
= 100M push messages for ONE item. This is a broadcast problem,
not a request/response problem.

Storage of bid history:

143M bids/day * ~200B per bid record ≈ 28.6 GB/day of bid history.
* 365 ≈ 10 TB/year, kept durably for audit/dispute. Sharded, cheap.

The takeaways that shape the design: total bid rate is modest but per-auction concurrency on hot keys (hundreds of bids/sec on one item) forces a single-writer serialization design; auction closings spike to ~100k in a minute so the scheduler is a real system, not a cron job; live price to millions of watchers is a broadcast/fan-out problem that must be decoupled from the bid path; and listing/media storage is trivial by comparison. The bid serializer, the close scheduler, and the price-broadcast plane are the drivers.

High-Level Design (HLD)

The system splits into planes: a listing plane (source-of-truth items and search), a bid plane (the linearizable per-auction bid processor - the heart), a scheduling plane (fires auction closes at their deadlines), a close/settlement plane (determines winner, creates orders), and a broadcast plane (pushes live price/state to watchers). Bids are routed to a single owner per auction so they serialize; everything read-heavy or fan-out is decoupled from that owner.

                          ┌───────────────┐
   Buyers / Sellers ────▶ │  API Gateway  │ ── auth, rate-limit, routing
                          └──────┬────────┘
      ┌──────────────┬──────────┼───────────┬────────────────┐
      ▼              ▼          ▼            ▼                ▼
┌───────────┐ ┌───────────┐ ┌────────────┐ ┌───────────┐ ┌────────────┐
│ Listing   │ │ Search    │ │  Bid       │ │ Watch/    │ │ Media      │
│ Service   │ │ Service   │ │  Service   │ │ Read Svc  │ │ Upload Svc │
│(write/read)│(read only) │ │ (router)   │ │ (WS/SSE)  │ │ (presign)  │
└────┬──────┘ └────┬──────┘ └─────┬──────┘ └─────┬─────┘ └─────┬──────┘
     │ CDC          │ query        │ route by      │ subscribe   │ direct
     ▼              ▼              ▼ auction_id     ▼             ▼
┌───────────┐ ┌───────────┐ ┌──────────────────┐ ┌──────────┐ ┌──────────┐
│ Listing DB│ │ Search    │ │  Auction Partition│ │ Pub/Sub  │ │ Object   │
│ (sharded  │ │ Index     │ │  (single writer   │ │ fan-out  │ │ Store +  │
│  by       │ │ (ES/      │ │   per auction_id) │ │ (price   │ │ CDN      │
│ auction_id)│ │  Lucene)  │ │  in-memory state  │ │ topics)  │ └──────────┘
└─────┬─────┘ └───────────┘ │  + WAL/durable log│ └────▲─────┘
      │ CDC                 └───────┬──────┬─────┘      │ publish
      ▼                             │ bids │ close      │
┌───────────────────────┐          │ log  │ event      │
│ Event stream (Kafka)   │◀─────────┘      │            │
│ bid_events / close_evt │                 ▼            │
└──────┬────────────────┘        ┌──────────────────┐   │
       │ consume                 │  Scheduler       │───┘
       ▼                         │ (timing wheel /  │
┌───────────────────────┐        │  time-bucketed   │
│ Bid History Store      │        │  close queue)    │
│ (append-only, sharded) │        └────────┬─────────┘
└───────────────────────┘                  │ at deadline
                                            ▼
                                 ┌──────────────────┐
                                 │ Close/Settlement │
                                 │ pick winner,     │
                                 │ create order,    │
                                 │ notify           │
                                 └──────────────────┘

List-an-item flow:

  1. Seller uploads photos via presigned URLs directly to the object store (bytes never touch app servers), then POST /listings with metadata and the end time.
  2. The Listing Service writes the item to the Listing DB (sharded by auction_id), and registers the auction’s end time with the Scheduler.
  3. A CDC/outbox event fans out to the Search indexer (so the item is browsable) and initializes the auction’s in-memory state in its Auction Partition (starting price, no bids yet).

Place-a-bid flow (the hot path):

  1. Buyer POST /auctions/{id}/bids {amount, max_proxy}. The API Gateway routes to the Bid Service, which routes the bid to the single partition that owns this auction_id (consistent hashing / partition assignment).
  2. That partition is the only writer for this auction. It processes bids for this key one at a time, in order: validate (auction active, amount >= current + increment), run proxy-bid resolution, update in-memory current price / high bidder, and append the accepted bid to a durable log (WAL) before acknowledging so the bid is never lost.
  3. It returns accept/reject to the bidder synchronously (sub-200ms), emits a bid_accepted event to Kafka (for history storage and analytics), and publishes the new price to the auction’s pub/sub topic for broadcast.
  4. Watchers subscribed to that topic get the new price pushed within ~1-2s. The bid path never talks to the millions of watchers directly.

Auction-close flow:

  1. The Scheduler fires the auction at its end time (within ~1s). It signals the owning Auction Partition to seal the auction: stop accepting bids, take the final state (highest bid, timestamp).
  2. The Close/Settlement service checks the reserve, picks the winner (highest bid above reserve; ties by earliest timestamp), writes the final result durably, creates an order (handed to the payment system, out of scope), and notifies winner and seller.
  3. If soft-close/anti-snipe is on, a bid inside the final window instead extends the deadline and reschedules the close.

The key insight: every auction is owned by exactly one writer that serializes its bids in memory with a durable log behind it, so correctness is a local single-key problem (trivially linearizable) rather than a distributed-lock problem; the deadline is a separate scheduling system; and the read/broadcast audience is fanned out through pub/sub so millions of watchers never touch the serialized bid path.

Component Deep Dive

The hard parts, naive-first then evolved: (1) serializing bids on a hot auction, the core; (2) proxy/automatic bidding; (3) closing 100M auctions at their deadlines; (4) broadcasting live price to millions of watchers.

1. Serializing bids on a hot auction (the core)

The operation is “if my bid beats the current highest by the increment, accept it and make me the new leader.” Read current, compare, write new. Two people do this at the same microsecond on the same item.

Naive approach: read-modify-write against a row in a database.

-- read
SELECT current_price, high_bidder FROM auctions WHERE id = :id;
-- app compares in memory: is :bid > current_price + increment?
-- write
UPDATE auctions SET current_price = :bid, high_bidder = :me WHERE id = :id;

Where it breaks:

  • Lost updates / double-win race. Two bidders both read current_price = 100, both decide their bid of 105 wins, both write. One overwrites the other; both were told they won. Classic non-atomic read-modify-write.
  • Row locking does not scale on a hot key. The obvious fix is a transaction with SELECT ... FOR UPDATE, serializing writers on the row. That is correct, but at 500 bids/sec on one row, every bid waits for a database round-trip while holding a lock, and the lock queue on that single hot row becomes the bottleneck - throughput collapses to how fast one row’s lock can cycle (a few thousand/sec at best, with brutal tail latency), and every other transaction touching that shard suffers.
  • Distributed lock (Redis/ZooKeeper) has the same shape plus failure modes. Grabbing a distributed lock per bid adds a network round-trip per bid, and lock-holder crashes mid-bid create lease-expiry correctness puzzles. You have moved the hot-row bottleneck onto a lock service.

The core problem: a hot auction is inherently a single serialization point, and forcing that serialization through a shared database row or a distributed lock pays a network round-trip (and lock contention) per bid on the one key that is hottest.

First evolution: atomic conditional update, push the compare into the store. Replace read-then-write with a single atomic compare-and-set:

UPDATE auctions
   SET current_price = :bid, high_bidder = :me, version = version + 1
 WHERE id = :id AND :bid >= current_price + increment AND status = 'active';
-- rows_affected = 1 -> accepted; 0 -> outbid, reject

This removes the lost-update bug (the DB serializes the conditional update, no double-win) and is a great answer for warm items. But on a hot key it is still one row that every bid contends on with a DB round-trip and row lock per bid. Correct, but the hot-key throughput ceiling remains.

The answer: a single in-memory writer per auction (the actor/partition model) with a durable append-only log. Assign each auction_id to exactly one owning process (an “auction partition”) via consistent hashing. All bids for that auction are routed to that owner. The owner holds the auction’s state in memory and processes its bids sequentially in a single-threaded event loop for that key:

AuctionPartition owns a set of auction_ids.
For each auction: in-memory state { current_price, high_bidder,
                                    high_proxy_max, increment, status,
                                    end_time, seq }

on Bid(auction_id, bidder, amount, proxy_max):
   a = state[auction_id]
   if a.status != active or now > a.end_time: reject("closed")
   resolve proxy logic (see part 2) -> new_price, new_leader, outbid_user
   if not valid: reject("too low")            # no DB round-trip
   a.seq += 1
   append {auction_id, seq, bidder, amount, ts} to DURABLE LOG   # fsync/quorum
   a.current_price = new_price; a.high_bidder = new_leader
   ack accept to bidder
   emit bid_accepted event; publish price to topic
   if outbid_user: notify "you were outbid"

Because one thread owns the key, bids are totally ordered with zero locking and zero per-bid network round-trip to a shared store - the compare-and-set is a memory operation. The only durability cost is appending the accepted bid to a log (batched/group-committed and replicated) before acking. This is the same trick a stock-exchange matching engine uses: serialize per-symbol in one place, in memory, with a log for durability and replay. A hot auction taking 500 bids/sec is trivial for an in-memory sequential loop; the durable log append (group-committed) is the only real cost, and it is sequential and cache-friendly.

Durability and failover. The append-only log per partition (think Kafka partition or Raft log) is the source of truth for bids. If the owning process crashes, a standby replays the log to rebuild the exact in-memory state and takes over ownership. Because state is a deterministic fold over an ordered bid log, recovery is exact - no bid lost, no double-processing (each bid has a monotonic seq; replay is idempotent). The auctions row in the Listing DB stores the materialized current price for reads and search, updated asynchronously from the log; it is a cache of the log’s fold, not the source of truth for bidding.

ApproachCorrect?Hot-key throughputPer-bid cost
Read-modify-write (no lock)No (lost updates)-broken
SELECT FOR UPDATE txnYeslow (lock cycle/row)DB round-trip + lock wait
Atomic conditional UPDATEYesmediumDB round-trip per bid
Single in-memory writer + logYesvery highmemory op + batched log append

Sharding the writers. With 100M auctions, no single process owns them all. Partition auction_id across many owner processes (say by hash(auction_id) % N). Each auction lives on exactly one owner at a time, so within an auction everything serializes, and across auctions there is no shared state, so it scales horizontally by adding partitions. Hot auctions do not need special sharding because a single auction’s bid rate (hundreds/sec) fits comfortably on one owner - the concurrency is bounded by human bidders on one item, not by the fleet.

The elegant part: we turn a scary distributed-concurrency problem into a boring local one - one in-memory single-threaded writer per auction, backed by a replicated append-only log - so bids on the hottest item serialize with zero locks and zero per-bid shared-store round-trips, exactly like a matching engine.

2. Proxy / automatic bidding

eBay’s real mechanic is not “enter a price,” it is “enter the maximum you would pay, and the system bids for you.” This changes the concurrency and the correctness story, so it deserves its own treatment.

Naive approach: store each user’s max, and on every new bid, loop both proxies against each other, incrementing back and forth until one exceeds the other’s max, writing each intermediate bid. Simulate the “bidding war” step by step.

Where it breaks:

  • A ladder of writes per bid. If A has max 100 and B enters max 200, naively you write 55, 60, 65, … up to 105 in a loop, dozens of bid rows for one action. Wasteful and slow, and on a hot key it multiplies the serialization load.
  • Ordering ambiguity if done outside the single writer. Resolving proxy wars needs the current authoritative state; doing it in the client or a separate service reintroduces the race.

The answer: resolve proxies as a single closed-form step inside the auction’s single writer. The state stores the current leader and the current leader’s hidden max proxy. When a new bid arrives with its own max, the owner computes the outcome directly, no loop:

State: current_price, leader, leader_max (hidden)
New bid from B with max_B (must be >= current_price + increment):

  if max_B > leader_max:
      # B outbids the incumbent A. New price = just enough to beat A's max.
      new_price  = min(max_B, leader_max + increment)
      new_leader = B ;  new leader_max = max_B
      outbid = A
  elif max_B <= leader_max:
      # A's proxy still wins; price rises to just beat B (or to B's max)
      new_price  = min(leader_max, max_B + increment)
      new_leader = A (unchanged) ;  leader_max stays
      B is immediately outbid (told so)
  # tie on max -> earliest bidder keeps lead (timestamp rule)

This is one arithmetic resolution, one accepted bid record (with the effective price), no ladder of writes. The hidden leader_max is stored server-side and never exposed (revealing it would let others snipe exactly one increment above). Every proxy resolution happens inside the single-writer loop, so it sees authoritative state and is automatically serialized - the same mechanism that makes plain bids correct makes proxy bids correct for free.

Why the single-writer model is what makes proxy bidding clean. Proxy resolution is a read-modify-write over (current_price, leader, leader_max). If that were spread across a shared row, every proxy bid would be a locked transaction. Because one in-memory owner holds all three fields and processes bids sequentially, the whole proxy war for an item collapses into a sequence of O(1) closed-form updates with no locking.

The lesson: proxy bidding is not a loop of incremental bids - it is a closed-form comparison of two hidden maximums, computed inside the auction’s single serialized writer, which keeps it both correct (no race on the leader/max) and cheap (one write per bid, not a ladder).

3. Closing 100M auctions at their exact deadlines

Every auction has an end time. At that instant it must stop taking bids and settle. There are 100M live auctions and closings spike to ~100k in a single minute.

Naive approach: a cron job that scans the table every minute for end_time <= now and closes them.

SELECT id FROM auctions WHERE status='active' AND end_time <= now();
-- close each

Where it breaks:

  • Scan of 100M rows on an interval. Even with an index on end_time, running this constantly and closing tens of thousands in a batch creates a lumpy, bursty load, and a full-table sweep is expensive.
  • Coarse timing. A per-minute cron closes auctions up to ~60s late. For a last-second bidding war, being 60s late means bids land after the intended end - unacceptable. You cannot make the cron per-second without hammering the DB every second.
  • Missed/duplicate fires on failure. If the cron host dies mid-batch, some auctions close twice or not at all; you need exactly-once close.
  • Thundering herd at round times. Everyone lists “7-day” auctions ending on the hour, so one minute holds 100k closings - a naive batch stalls.

The answer: a time-bucketed, sharded scheduler (a distributed timing wheel) that fires precise timers, decoupled from the item table. Instead of scanning, we schedule:

  • Bucket by time. When an auction is created, register (auction_id, end_time) into a scheduler keyed by a coarse time bucket (for example a per-second or per-minute bucket, sharded across scheduler nodes). This is a durable priority structure (a sorted set in Redis - ZADD closes score=end_epoch, or a Kafka-based delay queue, or a database of pending timers indexed by fire time), not the item table.
  • Timing wheel for precision. Each scheduler shard maintains an in-memory timing wheel for the near future (next few minutes at second granularity) hydrated from the durable store. Firing is O(1) per tick: advance the wheel, pop the due bucket, emit close events. This gives sub-second firing without any table scan.
  • Fan out closings. At fire time the scheduler emits a close_auction(auction_id) message; the owning Auction Partition seals the auction (stops accepting bids - it already knows end_time in its in-memory state as a second line of defense, rejecting any bid past it even if the close signal is delayed), and the Close/Settlement service settles it.
  • Exactly-once via idempotent close. Closing checks-and-sets status: active -> closing -> closed atomically (a conditional update on the auction record). A duplicate close signal is a no-op because the status already advanced. A missed fire is caught by a slow reconciliation sweep (a low-frequency background scan for status=active AND end_time < now - grace) that re-enqueues stragglers - a safety net, not the primary path.
  • Handle the round-time herd. Because closings are pre-registered in buckets, a 100k-in-one-minute spike is spread across scheduler shards and processed as a stream, not a single batch. Settlement work (winner selection, order creation, notifications) is pushed onto a queue and worked by an autoscaled consumer pool, so the spike smooths out.
create auction -> ZADD close_bucket[shard] score=end_epoch member=auction_id
scheduler tick (per second):
   due = pop all members with score <= now from this shard's near buckets
   for each: emit close_auction(auction_id)
Auction Partition on close_auction:
   CAS status active->closing (idempotent); take final state; stop bids
Close/Settlement (queue consumer):
   above reserve? winner = high_bidder ; ties -> earliest ts
   write result durably; create order; notify winner+seller
   CAS status closing->closed
reconciliation (every few min): status=active AND end_time<now-grace
   -> re-enqueue (missed-fire safety net)

Anti-snipe / soft close rides on the same mechanism: a bid inside the final N seconds tells the scheduler to ZADD a new later end_time for that auction (extend the timer) and updates the owner’s in-memory end_time. The close simply fires at the new time. Because the deadline lives in a reschedulable timer, extension is a cheap update, not a special case.

The point: auction close is a scheduling problem, not a scanning problem - pre-register each deadline in a sharded, durable, time-bucketed timer fed into a timing wheel for sub-second firing, make the close idempotent with a status CAS, and keep a slow reconciliation sweep as the only safety net, so 100k simultaneous closings become a smooth stream instead of a table-scan stampede.

4. Broadcasting live price to millions of watchers

A hot auction has millions of people staring at the current price as it climbs in the final seconds. Each price change must reach all of them within a second or two.

Naive approach: every watcher polls GET /auctions/{id} every second.

Where it breaks:

  • Polling melts the read path. 2M watchers polling one item every second is 2M reads/sec on one auction - a catastrophic hot key on the read tier, and it is mostly returning unchanged data.
  • Coupling watchers to the bid path. If watchers read the authoritative in-memory owner directly, millions of reads slam the one process that must stay free to serialize bids. The read audience must never touch the writer.
  • Latency. Polling delivers updates up to a poll-interval late and still costs the full request even when nothing changed.

The answer: push via a pub/sub fan-out tree, decoupled from the writer, with the current price cached at the edge.

  • The owner publishes, it does not serve watchers. Each time the auction’s price changes, the single-writer owner publishes one small message ({auction_id, price, high_bidder_masked, bid_count, time_left}) to a topic for that auction. That is one publish per price change, regardless of audience size.
  • Fan-out layer. A pub/sub / fan-out fleet (WebSocket or SSE gateways) subscribes to auction topics. Watchers hold a persistent connection to a gateway; the gateway subscribes once to the item’s topic on behalf of all its connected watchers for that item and pushes each update down every socket. Fan-out is a tree: one publish -> a few gateway nodes -> millions of sockets. The owner’s cost is O(1) per price change; the fan-out cost is spread across the gateway fleet.
  • Snapshot on join, deltas after. When a watcher opens the item, it gets the current snapshot from a read cache (Redis/CDN-cached current state, updated from the price stream), then receives live deltas over the socket. New watchers never hit the owner - they hit the cache.
  • Coalescing under a storm. In the final-second storm, prices can change 500 times/sec. Pushing 500 distinct updates/sec to 2M watchers is 1B messages/sec of pointless churn. The fan-out layer coalesces: it pushes at most, say, a few updates/sec per item (the latest price wins), because a human cannot perceive 500 updates/sec anyway. This caps broadcast volume no matter how hot the bidding gets.
  • Watch-count and browse reads are served entirely from the read cache and search index, never from the writer.
owner: on price change -> publish topic:auction_{id} {price, count, t_left}
gateways subscribed to topic:auction_{id}:
   coalesce to <= K updates/sec (latest wins)
   push to all local watcher sockets
new watcher: GET snapshot from read cache (not the owner) -> then live deltas
read cache updated asynchronously from the price stream

The lesson: the millions-of-watchers audience is a broadcast problem solved by pub/sub fan-out and edge caching, completely decoupled from the single bid writer - the owner publishes one message per price change, a fan-out tree spreads it, and coalescing caps the volume so a 500-bid/sec storm never becomes a billion-message/sec push.

API Design & Data Schema

REST for listings and bids, WebSocket/SSE for live price, presigned URLs for media.

Listings

POST /api/v1/listings
  body: { title, description, category, photo_keys:[...],
          start_price, reserve_price?, buy_now_price?,
          duration_days | end_time, increment_rule? }
  -> 201 { auction_id, status:"active", end_time }

GET  /api/v1/listings/{auction_id}
  -> { auction_id, title, description, photos:[urls], category,
       current_price, bid_count, high_bidder_masked, reserve_met:bool,
       buy_now_price?, end_time, time_left_ms, status, seller:{id,rating} }

PATCH /api/v1/listings/{auction_id}   // seller edits (only if no bids / policy)
DELETE /api/v1/listings/{auction_id}  // cancel (policy-gated)

Bidding

POST /api/v1/auctions/{auction_id}/bids
  body: { max_amount, idempotency_key }   // max_amount = proxy maximum
  -> 200 { accepted:true,  current_price, are_you_high_bidder:true }
  -> 200 { accepted:false, reason:"below_min_increment"|"auction_closed"
                                  |"outbid_by_proxy",
           current_price, min_next_bid }

POST /api/v1/auctions/{auction_id}/buy-now
  body: { idempotency_key }
  -> 200 { won:true, order_id }  |  409 { reason:"already_bid_over_threshold" }

GET  /api/v1/auctions/{auction_id}/bids     // bid history (masked bidders)
  -> { bids:[ { seq, amount, bidder_masked, ts } ], count }

The idempotency_key makes bid submission safe to retry: the owner dedupes by (auction_id, key) so a client retry after a timeout does not place two bids.

Live updates and media

// WebSocket / SSE
SUBSCRIBE  auction_{auction_id}
<- { type:"PRICE", auction_id, price, bid_count, time_left_ms }
<- { type:"CLOSED", auction_id, winner_masked, final_price }

POST /api/v1/media/presign  -> { uploads:[{ put_url, object_key }] }
// client PUTs photos directly to object store, then posts keys with listing

Data stores - the right tool per plane

1. Auction bid log - append-only, partitioned by auction_id (the source of truth for bids). Every accepted bid is an immutable ordered record. This is a replicated log (Kafka-partition / Raft-log style), the durable backing for the single-writer owners, and it is naturally append-only and shardable by auction_id. NoSQL/log, not relational: we need ordered, high-throughput, replayable appends, not joins.

Log: bids  (partition = auction_id, ordered by seq)
  auction_id   BIGINT
  seq          BIGINT     -- monotonic per auction, total order
  bidder_id    BIGINT
  max_amount   BIGINT     -- proxy max (server-secret)
  effective    BIGINT     -- resulting current_price after this bid
  ts           TIMESTAMP  -- tie-break key
  idem_key     STRING     -- dedupe

2. Auction/listing store - sharded relational or wide-column, by auction_id. The materialized current state for reads/search and the item metadata. current_price / high_bidder here are a cache of the bid log’s fold, updated asynchronously; status transitions (active/closing/closed) use conditional updates (CAS) for idempotent close. Relational is fine (sharded MySQL/Postgres): single-entity records, point reads, CDC-friendly.

Table: auctions           (shard key = auction_id)
  auction_id     BIGINT PK
  seller_id      BIGINT INDEX
  title, description, category, photo_keys(JSON)
  start_price    BIGINT
  reserve_price  BIGINT NULL       -- hidden
  buy_now_price  BIGINT NULL
  increment_rule INT
  current_price  BIGINT            -- materialized from log (read cache)
  high_bidder    BIGINT NULL
  leader_max     BIGINT NULL       -- hidden proxy max of leader
  bid_count      INT
  status         ENUM(active,closing,closed,cancelled)
  end_time       TIMESTAMP INDEX   -- also lives in scheduler + owner memory
  created_at     TIMESTAMP

3. Scheduler store - sorted timer set + timing wheel. A durable sorted structure keyed by fire time (Redis sorted set per shard, or a DB pending_closes(fire_time INDEX, auction_id)), hydrated into an in-memory timing wheel. Not the auctions table - a purpose-built timer index so firing is O(1), not a scan.

4. Search index - inverted-index engine (ES/Lucene), derived. For browse/search by keyword, category, price, “ending soon.” Sharded and rebuildable from the auctions store via CDC. NoSQL/search engine: native text + filter + sort (“ending soonest”) at scale.

5. Object store + CDN - media. Photos go direct to object storage via presigned URLs and serve from CDN. Never in a database.

6. Cache + connection registry - Redis. Read-cache of auction snapshots (for watcher joins and browse), fan-out topic subscriptions, and idempotency-key dedupe. The connection/gateway registry for the WS fan-out fleet.

Why the mixed split: bids want an ordered, replayable, append-only log (source of truth), the auction/item state wants a point-addressable CDC-friendly row (relational), deadlines want a time-indexed timer (sorted set / timing wheel), search wants an inverted index, media wants object storage, and live price wants pub/sub + cache. There is no single store good at all of these, and crucially the only place that needs strong consistency - the per-auction bid decision - is handled by the single-writer + log, so nothing needs a cross-auction distributed transaction.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Hot-key bid concurrency (breaks first). Hundreds of bids/sec on one auction in the final seconds is the defining load, and a shared-row or distributed-lock approach collapses on that key. Fix: the single in-memory writer per auction_id processes that key’s bids sequentially with no lock and no per-bid shared-store round-trip; durability is a batched, group-committed append to the auction’s replicated log. One item’s bid rate is human-bounded (hundreds/sec), which one owner handles easily.

2. Bid-log durability latency. Fsync-per-bid on the log would cap throughput and add tail latency. Fix: group commit - batch many accepted bids into one durable, quorum-replicated write; ack after the batch is durable. A hot auction’s bids naturally batch, so the amortized durability cost per bid is tiny while still never losing a bid.

3. Auction-close thundering herd. ~100k auctions scheduled to end in the same minute (round-time listing behavior). Fix: pre-registered, sharded time buckets + timing wheel so firing is O(1) and spread across scheduler shards; settlement pushed to an autoscaled queue consumer so winner-selection/order-creation/notifications smooth out over seconds rather than stalling in one batch. Idempotent status CAS makes retries safe.

4. Watcher read fan-out on a hot item. Millions of watchers wanting live price would melt the read path if they polled or read the owner. Fix: pub/sub fan-out tree (owner publishes once per change), coalescing to a few updates/sec per item, snapshot-from-cache on join so new watchers never touch the writer. Browse/search reads served by the index and read cache.

5. Owner failover / hot-partition loss. The single writer for a set of hot auctions is a potential SPOF for those keys. Fix: each owner’s state is a deterministic fold over its replicated log; a standby replays the log and takes over on crash (leader election per partition, Raft-style). Recovery is exact - monotonic seq makes replay idempotent, so no bid is lost or double-counted. Ownership is reassigned by a coordinator; the auctions row is only a read cache, so a brief owner failover does not corrupt state.

6. Partition skew from too many hot auctions on one owner. If many hot items hash to the same owner, that process saturates. Fix: rebalance ownership by moving auction_ids across owners (consistent hashing with virtual nodes); because a single auction’s rate is bounded, the fix is spreading hot keys across owners, not sharding one key (which is unnecessary). Load-aware placement moves known-hot auctions (ending soon, high bid_count) onto lightly loaded owners.

7. Search / browse and “ending soon.” 100M listings, ~50k read QPS, plus the popular “ending soonest” sort concentrates reads on soon-to-close items. Fix: inverted-index engine sharded and replicated, result caching for popular queries and category pages, and a dedicated “ending soon” cache refreshed frequently since it is a hot, predictable query.

8. Notifications spike at close. 100k closings/min each generate winner + seller + outbid notifications. Fix: notifications are async off the close event through a queue and a notification service; they can lag seconds without harming correctness. The authoritative outcome is written durably before any notification is sent.

9. Media storage and egress. Item photos across 100M listings. Fix: direct-to-object-store uploads (app tier never touches bytes), CDN for all serving, thumbnail variants for browse grids, and lifecycle policies moving ended-auction media to cold storage.

10. Single points of failure and multi-region. Fix: every plane is replicated; the bid log and auctions store are quorum-replicated, the search index and caches are rebuildable/derived. For the strongly-consistent bid path, an auction is owned in one region at a time (its log has a single leader) to preserve linearizability - you cannot cheaply have two regions both accepting bids on the same item, so an auction is pinned to a home region with synchronous cross-AZ replication and cross-region failover, while read/browse/watch is served from every region off replicated caches and indexes.

Wrap-Up

The trade-offs that define this design:

  • A single in-memory writer per auction over a shared row or a distributed lock. We refuse to serialize hot-item bids through a contended database row or a per-bid lock service; instead each auction is owned by one process that orders its bids in memory with a replicated append-only log behind it - correctness becomes a trivial local single-key property, at the cost of running an owner/log tier and per-partition leader election and failover.
  • Bids as an ordered log, item state as a materialized cache. The bid log is the source of truth; the auctions row’s current_price is a fold of that log, updated asynchronously - so reads and search are eventually consistent while the bid decision stays linearizable, and recovery is an exact log replay.
  • Close as scheduling, not scanning. Deadlines are pre-registered in sharded, time-bucketed timers fed to a timing wheel for sub-second firing with idempotent status CAS, so 100k simultaneous closings are a smooth stream and a slow reconciliation sweep is the only safety net - at the cost of a dedicated scheduling subsystem instead of a cron line.
  • Watchers fanned out, never touching the writer. Live price reaches millions through a pub/sub fan-out tree with coalescing and edge-cached snapshots, so the single bid writer publishes one message per change and the broadcast volume is capped regardless of how hot the bidding gets - trading perfect per-tick fidelity (humans cannot see 500 updates/sec) for a bounded, survivable fan-out.
  • Strong consistency only where it is needed, and pinned to one region. The only linearizable point is the per-auction bid decision; everything else (browse, search, watch, notifications, media) is eventual and multi-region, and the consistent auction is home-pinned to one region because two regions cannot both own the same item’s serialization point without a distributed transaction we deliberately avoid.

One-line summary: an auction is a linearizable single-writer state machine per item - bids serialize in one in-memory owner backed by a replicated append-only log (with closed-form proxy resolution), deadlines fire from a sharded time-bucketed scheduler with idempotent settlement, and millions of watchers get live price through a coalescing pub/sub fan-out that never touches the writer - so the system stays correct on the hottest item and on-time at 100M deadlines while keeping everything read-heavy eventually consistent and horizontally scaled.