Design Coinbase (Crypto Exchange) - System Design

“Let users buy and sell crypto” hides two different, both-irreversible problems welded together. Inside the exchange, Coinbase is a matching engine: for a popular pair like BTC-USD it holds its own order book, matches your buy against someone’s sell by price-time priority, and the moment two orders cross, a trade has happened and cannot be un-happened. Outside the exchange, Coinbase is a custodian on a blockchain: it holds your coins, watches public chains for your deposits, and signs withdrawals that broadcast to Bitcoin or Ethereum where a confirmed transaction is permanent - no chargeback, no reversal, no support ticket that claws it back. So the core invariant is brutal on both sides: never let a user trade or withdraw a balance they do not have, never double-credit a deposit or double-broadcast a withdrawal, and never let the internal ledger disagree with what is actually on-chain.

The second thing that reshapes the design is the clock, but not the way a stock exchange’s does. Crypto never closes. There is no 9:30 open to size against; instead the load is driven by price volatility. A sudden 15% move in BTC at 3am on a Sunday spikes order placement, cancels, and balance reads 10x in minutes, and everyone hits the same few hot pairs at once. So the system must be correct under a burst that can arrive at any second, concentrated on a handful of order books. The engineering: a strongly-consistent balance ledger and a deterministic matching engine at the center, wrapped in an elastic edge that survives a 10x volatility spike, bridged to slow, irreversible public blockchains at the edges. Let me build it properly.

Functional Requirements (FR)

In scope:

• Buy / sell crypto (place an order). A user submits an order on a pair (e.g. BTC-USD): side (buy/sell), quantity, type (market or limit). The exchange reserves the funds, matches against the order book, and settles the trade into both parties’ balances.
• Order book matching for popular pairs. Coinbase runs its own matching engine with price-time priority for liquid pairs. This is in scope, unlike a broker that routes out.
• Wallet balances. Every user has a balance per asset (USD, BTC, ETH, …). Show available vs held (reserved by open orders), and the balance must be exactly correct at all times.
• Deposits (on-chain in). A user sends crypto from an external wallet to a Coinbase-controlled address. We detect the incoming transaction on-chain, wait for enough confirmations, and credit the balance.
• Withdrawals (on-chain out). A user withdraws crypto to an external address. We sign a transaction, broadcast it to the chain, and debit the balance - exactly once, because a broadcast confirmed transaction is irreversible.
• Order lifecycle. Orders move through open, partially filled, filled, canceled; users can cancel a resting order and see status and history.

Explicitly out of scope (name it so you own the scope):

• The blockchains themselves. I treat Bitcoin, Ethereum, etc. as external systems with a node/RPC interface I read from and broadcast to. I do not design consensus, mining, or the mempool.
• Fiat rails (ACH/wire/card). USD deposits and withdrawals go through banking partners; I treat fiat balance movements as an async input analogous to a crypto deposit and focus on the crypto and trading engine.
• Illiquid pairs / OTC / staking / DeFi / derivatives. I design spot trading on liquid pairs with a central limit order book. Thin pairs (RFQ/market-maker quotes), margin, and staking are named but not built.
• KYC/AML onboarding, tax reporting, fraud scoring. Consumed as gates around the flows; not designed here.

The decision that drives everything: Coinbase is a custodial exchange sitting between a strongly-consistent internal ledger and slow, irreversible public blockchains, so balances live in an ACID double-entry ledger, matching happens in a deterministic single-writer engine per pair, and both the deposit-credit and the withdrawal-broadcast are made exactly-once against chains that never forgive a double. Correctness comes from reserving before matching, sequencing the matching engine, and idempotency at every chain boundary - not from trusting the network.

Non-Functional Requirements (NFR)

• Scale: 100M users. The market runs 24/7, so there is a steady base load, but the sizing number is the volatility spike: a sharp price move drives order placement, cancels, and balance reads to ~10x the base, concentrated on the few hottest pairs (BTC-USD, ETH-USD).
• Latency: placing an order (validate + reserve + accept into the book) should return p99 under ~100ms - traders react to a fast-moving price. The match itself, on an in-memory engine, is microseconds; settlement into balances is asynchronous but fast. Balance and order-book reads should feel live, sub-second.
• Availability: 99.99% on the trade and balance path. The matching engine per pair is a single logical writer, so its failover must be fast and lossless. Deposits and withdrawals are async and tolerate seconds-to-minutes (they already wait on block confirmations), but must never lose or duplicate a movement.
• Consistency: strong for balances, holds, and the order book. A user must never trade or withdraw more than they hold, never have a deposit credited twice, never have a withdrawal broadcast twice. The order book must be a single, consistent, deterministic sequence of events. Market-data display (book depth, last price) can be eventually consistent, a few hundred ms stale; the ledger and the book cannot.
• Durability: every accepted order, every trade, every ledger entry, and every broadcast withdrawal survives any node loss. A lost “broadcast” record could mean coins that left our hot wallet on-chain that we have no memory of - an unaccounted, irreversible loss. Zero loss on money and chain paths.
• Auditability: every balance movement is reconstructable and must reconcile against on-chain reality. We can prove “the sum of all user balances of BTC equals the BTC we actually control on-chain” (proof of reserves) at any time.

The tension: the ledger and matching engine want strong consistency and determinism, which fight throughput and horizontal scaling; the blockchain bridges are slow, asynchronous, and irreversible; the market-data and balance-read edge wants massive elastic fan-out. So we split the system into a strongly-consistent core (ledger + per-pair matching engines), asynchronous chain-facing services (deposits, withdrawals) that reconcile against the network, and an elastic read/streaming layer, each scaled on its own axis.

Back-of-the-Envelope Estimation (BoE)

Real numbers, because they show the spike is on a few pairs, not spread out.

Users and orders:

100M registered users, assume ~5M daily active traders.
Assume an active trader places ~6 orders/day (crypto traders churn more,
and market/limit + cancels count).
=> 5M * 6 = 30M orders/day
24 hours = 86,400 sec
Base order rate = 30M / 86,400 ≈ 350 orders/sec

Volatility spike: a sharp BTC move can 10x the rate for ~10-20 min,
concentrated on the top pairs:
  10 * 350 ≈ 3,500 orders/sec peak, and ~70-80% of it lands on
  BTC-USD + ETH-USD. So one order book must absorb ~2,000-2,500
  orders+cancels/sec at peak. That single-book number sizes the engine.

Balance and order-book reads:

Traders and apps poll balances and book depth far more than they trade.
5M actives * 60 balance/book views/day = 300M reads/day
≈ 3,500 reads/sec base, ~35,000/sec in a spike.
Plus every book change streams to subscribers (market data, below).
Read:write ≈ 10:1, and reads spike with writes on the same hot pairs.

Market-data fan-out:

A hot book (BTC-USD) can change hundreds of times/sec during a spike.
We do NOT push every micro-change to every client. We publish a
conflated top-of-book + depth snapshot a few times/sec per pair, and a
trade tape. If 2M users are connected watching ~3 pairs each, the
interesting set is the union of ~a few hundred active pairs, conflated.
This is a fan-out problem for the streaming layer, not the ledger.

Storage per year:

Order row (pair, side, qty, type, prices, state, refs)  ≈ 400 bytes
Trades per order (~1-3 fills)                            ≈ 3 * 200 = 600 bytes
Ledger entries per trade (double-entry, ~6 legs)         ≈ 6 * 120 = 720 bytes
Order events (state transitions, ~4)                     ≈ 4 * 150 = 600 bytes
--------------------------------------------------------------------
≈ 2.3 KB per order, round to ~3 KB with indexes

30M orders/day * 3 KB ≈ 90 GB/day
* 365 days ≈ 33 TB/year (market never closes, so all 365)

Plus deposit/withdrawal records and per-block scan state, small by comparison. Retained for years (regulatory + reconciliation): low hundreds of TB over the window. The hot set (open orders + current balances + live books) is tiny; history archives to cold storage.

Balances / wallet working set:

100M users * (~30 assets, but most hold ~3) => sparse.
Say avg 3 asset balances * 100M = 300M balance rows * ~120 bytes
  ≈ 36 GB of balance rows.
Fits a sharded SQL cluster; hot accounts cache in memory.
On-chain: a hot wallet per major chain + a huge cold wallet;
deposit address book maps addresses -> user (100M+ addresses if HD-derived).

The headline: base throughput (~350 orders/sec) is trivial; the design is sized by a volatility spike that puts ~2,000-2,500 strongly-consistent order/cancel writes per second onto a single hot order book, alongside a ~35K/sec read spike - all while slow, irreversible blockchain deposits and withdrawals flow underneath. Everything below keeps the ledger and the hot book correct under that spike and the chain bridges exactly-once.

High-Level Design (HLD)

The spine: a stateless API / Order Gateway validates and dedupes, the Balance / Ledger Service reserves funds in one ACID transaction, orders flow to a per-pair Matching Engine (in-memory, single-writer, sequenced) that produces trades, a Settlement Service applies trades to the double-entry ledger, and two chain-facing services - Deposit Watcher and Withdrawal Service - bridge the ledger to public blockchains. A separate Market Data + Streaming layer publishes books and trades for reads.

                   ┌────────────────────────────────────────┐
                   │        Client (mobile / web / API)        │
                   └────┬───────────────┬──────────────┬─────┘
        place/cancel     │  balance/book │              │  deposit addr /
        (POST order)     │  reads (WS)   │              │  withdraw request
               ┌─────────▼─────────┐  ┌──▼───────────┐  │
               │  API/Order Gateway │  │ Market Data   │  │
               │ authN/Z, validate  │  │ Streaming API │  │
               │ idempotency, rate  │  │ (WS fan-out)  │  │
               └─────────┬─────────┘  └──▲───────────┘  │
                         │ new order       │ books/trades │
              ┌──────────▼───────────┐     │              │
              │ Balance / Ledger Svc  │     │              │
              │  ONE ACID txn:        │     │              │
              │  reserve funds        │─────┼─► reads: balances
              │  write order=ACCEPTED │     │              │
              │  + outbox event       │     │              │
              └──────────┬───────────┘     │              │
                         │ order.accepted (outbox->bus)     │
              ┌──────────▼───────────────┐ │              │
              │  Matching Engine (per pair)│ │              │
              │  in-memory book, single    │─┘ trades,      │
              │  writer, sequenced journal │   book deltas  │
              └──────────┬───────────────┘                 │
                         │ trade events (sequenced)          │
              ┌──────────▼───────────────┐                  │
              │  Settlement Service        │                  │
              │  trade -> double-entry     │                  │
              │  ledger, release holds     │                  │
              └──────────┬───────────────┘                  │
                         │ balance updates                    │
   ┌─────────────────────┼──────────────────────┐            │
   │                     │                        │            │
┌──▼───────────────┐  ┌──▼─────────────────┐  ┌──▼────────────▼───┐
│ Deposit Watcher   │  │ Withdrawal Service  │  │ Wallet / Key Mgmt  │
│ scan chains, N    │  │ sign, broadcast,    │  │ hot wallet (online)│
│ confirmations,    │  │ track confirmations,│  │ cold wallet (HSM,  │
│ credit ledger     │  │ debit ledger        │  │ offline, air-gap)  │
└──┬───────────────┘  └──┬─────────────────┘  └───────────────────┘
   │ read blocks          │ broadcast tx
┌──▼──────────────────────▼─────────────────────────────────────┐
│   Public Blockchains (Bitcoin, Ethereum, ...) - IRREVERSIBLE   │
│   full nodes / RPC, mempool, block confirmations               │
└────────────────────────────────────────────────────────────────┘

Synchronous path (what the user waits on, target p99 < 100ms):

1. Client calls POST /orders with {pair, side, qty, type, limit_price?} and a client Idempotency-Key.
2. The Gateway authenticates, validates (pair tradable? qty within limits? limit sane?), rate-limits, and checks the idempotency layer - a known key returns the stored response and does nothing else.
3. The Balance / Ledger Service runs one ACID transaction: for a buy on BTC-USD, guard usd_available >= qty * price_estimate + fee and move that from available to held; for a sell, hold the base asset (btc_available >= qty). Write the orders row ACCEPTED, insert an outbox event, commit.
4. Gateway returns 202 Accepted with order_id and state ACCEPTED. The order is now the matching engine’s problem; the client subscribes to updates over the WebSocket.

Asynchronous path (match, settle, and the chain bridges):

1. The Matching Engine for that pair consumes order.accepted in sequence, inserts the order into its in-memory book, and matches by price-time priority. Each match emits a trade event to a sequenced journal. No shared DB transaction is on the match hot path - the engine is a deterministic in-memory state machine.
2. The Settlement Service consumes trade events and, in an ACID transaction per trade, swaps assets between the two parties’ balances (double-entry), applies fees, releases leftover holds, and advances order state. Balances update; the client sees fills.
3. Deposits: the Deposit Watcher scans each chain’s new blocks for transactions to our addresses, waits N confirmations, then credits the user’s balance via a ledger transaction - idempotently keyed on the on-chain txid.
4. Withdrawals: the Withdrawal Service validates and holds the balance, has the Wallet service sign a transaction from the hot wallet, broadcasts it exactly once, tracks confirmations, then finalizes the debit. Cold wallet holds the bulk of funds offline and refills the hot wallet in batches.

The structural insight: there is no transaction that spans the internal ledger and the blockchain, so deposit-credit and withdrawal-broadcast are each an idempotent, recoverable saga keyed on an on-chain identity; and matching is deliberately pulled out of the database into a deterministic in-memory engine so it can be fast and sequenced, with settlement reconciling it back into the ACID ledger. The market-data and balance-read layer hangs off the side and scales independently.

Component Deep Dive

The hard parts, naive-first then evolved: (1) the balance ledger with holds so nothing is oversold, (2) the per-pair order book matching engine under a spike, (3) crediting on-chain deposits exactly once across confirmations and reorgs, (4) broadcasting on-chain withdrawals exactly once from hot/cold wallets.

1. Wallet balances and holds without overspending

The job: the instant a user places an order or requests a withdrawal, reserve exactly enough of the right asset so concurrent actions can never together exceed what the account holds. A user rapidly tapping “sell BTC” while also requesting a BTC withdrawal must not commit the same coins twice.

Naive approach: read balance, then act

def place_buy(user, pair, qty, price):
    if user.usd >= qty * price:       # read
        match_order(pair, qty)        # engine
        user.usd -= qty * price       # write, later

Where it breaks:

• Time-of-check to time-of-use race. Two orders read usd = 1000, both pass >= 800, both proceed. The account just committed $1600 with $1000. On a fast-moving market with rapid taps and API bots, this race fires constantly.
• Check-then-act-then-debit is three steps with crashes between. The order matches and a trade happens, the process dies before debiting: the user got BTC but still shows full USD, and can spend it again. Double spend by crash.
• A trade and a withdrawal race for the same asset. Balance passes the check for both; both proceed; the account goes negative in a custodial system, which means Coinbase is short real coins.
• Market orders have no fixed price. For a market buy qty * price is a guess; a naive exact debit is wrong the moment the price moves.

Read-modify-write with no hold prints spendable balance out of thin air, and in a custodial exchange that shortfall is real money.

Evolved approach: two-bucket balances + a conditional hold in one ACID transaction, with a collar for market orders

Split every asset balance into available and held. Placing an order or requesting a withdrawal does not debit - it holds. In one ACID transaction, guarded against races:

BEGIN;
  -- BUY BTC-USD: hold USD. The WHERE clause is the concurrency guard.
  UPDATE balances
     SET available = available - :hold_amount,
         held      = held      + :hold_amount,
         version   = version + 1
   WHERE user_id = :uid AND asset = 'USD'
     AND available >= :hold_amount;        -- 0 rows => insufficient => ROLLBACK

  INSERT INTO orders (order_id, user_id, pair, side, qty, type,
                      limit_price, hold_asset, hold_amount, state)
       VALUES (:oid, :uid, 'BTC-USD', 'BUY', :qty, :type, :lim,
               'USD', :hold_amount, 'ACCEPTED');

  INSERT INTO outbox (order_id, topic, payload)
       VALUES (:oid, 'order.accepted', ...);
COMMIT;

• The conditional UPDATE ... WHERE available >= amount is the guard. Row locks serialize concurrent actions on the same (user, asset); the second either still fits or fails and rolls back. No lost update, no over-hold, no external lock service.
• Sells hold the base asset. A sell of 0.5 BTC runs the same pattern on the BTC balance row: available >= 0.5. This is what stops a user selling coins they do not hold or selling the same coins twice.
• Withdrawals hold too. A withdrawal request holds the amount + network fee on the asset balance in the same conditional-update pattern, so a trade and a withdrawal cannot both commit the same coins.
• Market-order collar. For a market buy we do not know the fill price, so we hold a collar: hold = qty * (last_price * (1 + buffer)) + fee with a buffer for the worst plausible move. We hold slightly high; when trades settle at real prices, we release the difference back to available. Holding high and refunding is safe; holding low and overshooting means a negative balance.
• Every hold is released or converted, never leaked. Fill, cancel, reject, expire, withdrawal-confirmed or failed - each terminal path converts the hold into a real ledger movement or releases it. A recovery sweeper reconciles any order/withdrawal stuck with a hold but no active work.

Why this is right: the hold makes the balance committed the instant the action is accepted, so no concurrent order or withdrawal can double-commit it; the conditional update makes the race impossible at the row level; and the collar handles unknown market-fill prices by erring toward over-holding. In a custodial system the balance is the money, so we lock it before anything downstream can act on it.

2. The per-pair order book matching engine under a volatility spike

The job: match buys against sells on BTC-USD by price-time priority, correctly and deterministically, while a volatility spike throws ~2,000-2,500 orders and cancels per second at that one book. It must be consistent (one canonical book), fast (microsecond matches), and recoverable (survive a node crash without losing or reordering trades).

Naive approach: match in the database with row locks

def match_buy(pair, qty, price):
    with db.transaction():
        asks = db.query("SELECT * FROM orders WHERE pair=? AND side='SELL'"
                        " AND price <= ? ORDER BY price, created_at FOR UPDATE", pair, price)
        for a in asks:
            fill = min(qty, a.remaining); ...   # write each fill row

Where it breaks:

• The database becomes the bottleneck at the exact wrong moment. Every order takes a range lock over the resting side of the book and writes multiple rows. On the hottest pair during a spike that is thousands of overlapping transactions all contending on the same rows. Lock contention collapses throughput to a crawl when you need it most.
• Price-time priority is hard to keep under concurrency. Interleaved transactions can match a later order ahead of an earlier one at the same price. The fairness guarantee - the whole point of a limit order book - silently breaks.
• No canonical order of events. Two matches, a cancel, and a new order can commit in an order that is not reproducible, so the trade tape and the book are not deterministic and cannot be audited or replayed.
• Latency is milliseconds, not microseconds. Traders and market makers need matching far faster than a durable DB transaction per order allows.

A limit order book is a hot, contended, latency-critical single-sequence data structure. A general-purpose database is the wrong tool for the match itself.

Evolved approach: in-memory single-writer engine per pair, sequenced input journal, snapshot + replay recovery

One matching engine process owns one pair’s book, entirely in memory, as the single writer. No locks - there is only one writer, so there is no contention. The book is two price-indexed priority structures (bids descending, asks ascending), each price level an FIFO queue for time priority.

struct Book {  bids: map<price desc -> FIFO<order>>,  asks: map<price asc -> FIFO<order>>  }

def on_order(o):                       # single-threaded, deterministic
    book = o.side == BUY ? asks : bids
    while o.remaining > 0 and best(book) crosses o.price:
        resting = head(best_level(book))
        fill = min(o.remaining, resting.remaining)
        emit_trade(o, resting, fill, resting.price)   # taker pays maker's price
        o.remaining -= fill; resting.remaining -= fill
        if resting.remaining == 0: pop(resting)
    if o.remaining > 0 and o.type == LIMIT:
        insert(o.side == BUY ? bids : asks, o)         # rest on the book

• Sequenced input journal (the key to correctness and recovery). Every command into the engine - new order, cancel - is first appended to a durable, ordered log (a Kafka partition or Raft log dedicated to that pair) with a monotonic sequence number. The engine consumes the journal in sequence order. That single ordered input is what makes matching deterministic: given the same journal, the engine always produces the same trades in the same order. Price-time priority is guaranteed because “time” is just journal position.
• Trades out to a sequenced output. Each match emits a trade event with the engine’s own monotonic seq to a durable trade stream that Settlement, market data, and audit all consume. The book itself is never the source of truth for money; the trade stream is.
• Recovery = snapshot + replay. Periodically the engine writes a full book snapshot at sequence S. On crash, a standby loads the latest snapshot and replays journal entries after S to rebuild the exact book, then resumes. No trade is lost or reordered because the journal is the durable truth and replay is deterministic. Failover is a standby taking over the journal consumer position; because output trades carry seqs, downstream dedupes any replayed trade.
• Sharding by pair. Each pair is an independent engine, so the whole exchange scales by assigning pairs to engines/hosts. BTC-USD and ETH-USD run on separate, beefy hosts; thin pairs pack many-per-host. A single hot pair is one engine’s problem, sized for its own peak (~2,500 orders/sec is easy in memory, single-threaded).
• Backpressure and admission control. During a 10x spike the Gateway rate-limits per user and globally, and the journal absorbs bursts; the engine drains at its own steady rate. Because input is a durable journal, a burst queues rather than dropping orders.

Why this is right: pulling the book out of the database into a single-writer in-memory engine removes lock contention entirely and gets microsecond matches; the durable sequenced journal makes matching deterministic (so price-time priority holds and the tape is auditable) and makes crash recovery a snapshot-plus-replay with no lost or reordered trades; and per-pair sharding means a volatility spike concentrated on one book is one engine’s bounded problem. The database is used for durable balances and settlement, not for the match.

3. Crediting on-chain deposits exactly once (confirmations and reorgs)

The job: when a user sends BTC to their Coinbase deposit address, detect it on-chain, wait until it is safe, and credit their balance exactly once - despite the chain being asynchronous, deposits arriving without warning, and blocks occasionally getting reorged away.

Naive approach: on seeing the transaction, credit the balance

def on_new_tx(tx):
    if tx.to in our_addresses:
        user = owner_of(tx.to)
        credit(user, tx.asset, tx.amount)   # immediately

Where it breaks:

• Zero-confirmation deposits can be reversed. A transaction in the mempool or a freshly mined block can be dropped or replaced (double-spent) before it is buried. Credit on sight and a user deposits, trades or withdraws the credited funds, then the deposit vanishes in a reorg - Coinbase is short the coins.
• Duplicate detection double-credits. We rescan blocks, reconnect nodes, or see the same txid across a mempool sighting and a mined block. Credited each time, the user is credited twice for one deposit.
• Reorgs invalidate an already-credited block. A block at height H is orphaned; a transaction we saw and credited is no longer on the canonical chain. Without handling reorgs, the ledger keeps a credit the chain no longer backs.
• Address-to-user mapping at 100M users. Naively one shared address makes deposits ambiguous; you cannot tell whose money arrived.

Crediting on sight, without confirmations, dedup, or reorg handling, credits money the chain can still take back.

Evolved approach: per-user derived addresses + confirmation threshold + idempotent credit keyed on txid + reorg rollback

Give every user a unique deposit address per asset, derived from an HD wallet (BIP32) so we control the keys without storing 100M private keys - addresses derive deterministically from an xpub + index. address -> (user, asset) is a lookup table. Now every incoming transaction is unambiguously attributable.

Track each deposit through a confirmation state machine, keyed on the on-chain txid + output index (a utxo/txid:vout, or txid + log index on Ethereum):

def scan_block(block):
    for tx in block.txs:
        for out in tx.outputs:
            if out.address in our_addresses:
                dep_key = (tx.txid, out.index)
                if not deposits.exists(dep_key):                 # dedup guard
                    deposits.insert(dep_key, state='SEEN', user=owner(out.address),
                                    amount=out.value, block_height=block.height)

def on_new_head(height):
    for d in deposits.where(state='SEEN' or 'CONFIRMING'):
        confs = height - d.block_height + 1
        if reorged_out(d):                       # its block no longer canonical
            d.state = 'ORPHANED'                  # do NOT credit; if credited, reverse
        elif confs >= threshold(d.asset):        # e.g. BTC 3-6, ETH ~30-60
            credit_once(d)                        # idempotent on dep_key

def credit_once(d):
    with db.transaction():
        if d.state == 'CREDITED': return          # idempotent
        ledger_post([ entry(d.user, d.asset, +d.amount, kind='DEPOSIT', ref=d.txid) ])
        d.state = 'CREDITED'

• Confirmation threshold sized to reorg risk. Credit only after N confirmations - deep enough that a reorg is economically implausible. BTC ~3-6, Ethereum after finality (~2 epochs). Larger deposits can require more confirmations. Below threshold the deposit is visible as “pending” but not spendable.
• Idempotent credit keyed on the on-chain identity. The credit is guarded by dep_key and the state machine, so re-seeing a transaction (rescan, node reconnect, mempool-then-block) credits exactly once. The ledger_entries row carries the txid; a unique index on (txid, vout) is the hard backstop.
• Reorg handling. The watcher tracks each deposit’s including block. If that block is orphaned before threshold, the deposit goes ORPHANED and is never credited. In the rare case a reorg runs deeper than threshold after a credit, a compensating reversal entry claws the credit back (and if the funds were already withdrawn, it becomes a loss event for risk/ops, not a silent ledger corruption). Choosing a safe threshold makes this vanishingly rare by design.
• Scan reliability. The watcher tracks the last fully-scanned height per chain, processes blocks in order, and can re-scan a range safely because crediting is idempotent. Multiple redundant full nodes avoid a single node lying or lagging.

Why this is right: per-user derived addresses make every deposit attributable at 100M scale without holding 100M keys; the confirmation threshold turns a probabilistic, reorg-prone chain state into a safe “final enough” signal before money becomes spendable; idempotent credit keyed on the on-chain txid kills double-credit across rescans and reconnects; and explicit reorg rollback keeps the ledger honest to the canonical chain. The chain is treated as eventually-final input, and we only commit money once it is.

4. Broadcasting on-chain withdrawals exactly once (hot/cold wallets)

The job: when a user withdraws BTC to an external address, sign a transaction, broadcast it to the network exactly once, and debit the balance - even though the signer can crash mid-broadcast, the mempool is asynchronous, and a confirmed transaction is irreversible. A double-broadcast that both confirm sends the coins twice; a lost broadcast record loses track of coins that already left.

Naive approach: debit, sign, broadcast

def withdraw(user, asset, amount, dest):
    user.balance -= amount            # debit
    tx = wallet.sign(amount, dest)    # sign from hot wallet
    node.broadcast(tx)                # send to chain

Where it breaks:

• Crash between broadcast and record. The transaction hits the mempool, the process dies before persisting “broadcast”. On retry we sign and broadcast again; if the network accepts a second distinct transaction spending different UTXOs to the same dest, the user is paid twice and the coins are gone irreversibly.
• Ambiguous broadcast. The RPC call times out. Did the node accept it into the mempool or not? Rebroadcasting risks a double; not rebroadcasting risks a stuck withdrawal. The transport error alone cannot say.
• Debit-then-sign leaves money in limbo on failure. If signing fails after the debit, the user’s balance is gone but no coins were sent. Order matters.
• Hot wallet holds everything. Keeping all customer funds in an online, signing-capable wallet means one key compromise drains the exchange. Real exchanges keep the vast majority offline.

Sign-and-broadcast without a durable intent and idempotency, against an irreversible chain, double-pays or loses coins.

Evolved approach: hold -> durable intent -> deterministic single-broadcast -> confirm -> finalize, split hot/cold

Cold/hot split first. The cold wallet holds the large majority of funds, keys in offline HSMs / air-gapped signers, never touched by online systems. The hot wallet holds a small operational float (enough for normal daily withdrawals) and is the only thing that signs online. When the hot wallet runs low, a batched, reviewed transfer from cold refills it. A hot-wallet compromise caps the loss at the float, not the exchange.

Withdrawal as a durable, idempotent saga:

1. VALIDATE + HOLD.  In one ACID txn: hold amount+networkFee on the user's
   balance (conditional update, section 1); write withdrawal row PENDING with
   a client idempotency key (unique). Risk/AML checks; large ones queue for review.

2. BUILD + SIGN.  Wallet service selects inputs (UTXOs) or nonce (account model),
   builds the tx. The account-model NONCE or the specific chosen UTXOs make the
   signed tx DETERMINISTIC for this withdrawal_id: the same withdrawal always
   maps to the same tx identity, so a retry rebuilds the same transaction, not a
   new one. Persist state=SIGNED with the txid BEFORE broadcasting.

3. BROADCAST ONCE.  Broadcast the signed tx. Because state=SIGNED+txid is
   pre-written, recovery after a crash checks the chain for THAT txid rather than
   asking "did I send something?". Rebroadcasting the identical tx is a no-op on
   the network (same txid / same nonce), so broadcast is effectively idempotent.
   state=BROADCAST.

4. CONFIRM.  Watch the chain for the txid reaching N confirmations. state=CONFIRMED.

5. FINALIZE.  In one ACID txn: convert the hold into a real debit (double-entry:
   user balance down, hot-wallet control account down), release any leftover fee
   hold. state=COMPLETED.

• Deterministic transaction identity is the exactly-once trick. On an account-model chain (Ethereum) the nonce makes the transaction unique: two attempts for the same withdrawal reuse the same nonce, so at most one can ever confirm; a rebroadcast is the same transaction. On a UTXO chain (Bitcoin) we bind the withdrawal to a specific chosen set of inputs and persist them, so a rebuild reuses the same inputs and yields the same txid. Either way, “sign again” produces the same on-chain identity, never a second payout.
• Pre-write SIGNED with the txid before broadcasting. This is the same discipline as the routing state machine: the durable record of what we are about to put on-chain exists before the irreversible act, so recovery is deterministic. After any crash we look up the txid on-chain; if it is there, we move forward; if not, we rebroadcast the identical tx.
• Idempotency end to end. The client idempotency key (unique constraint) makes a retried request one withdrawal; the deterministic txid makes a retried broadcast one payout. Both layers guard the irreversible act.
• Batching and fee management. Many small withdrawals can be batched into one on-chain transaction (multiple outputs) to save fees during congestion; each user output still maps to its withdrawal_id for finalization. Fee estimation and replace-by-fee handle a stuck transaction - carefully, because RBF changes the txid and must be tracked as the same withdrawal_id.
• Recovery sweeper. A job scans withdrawals stuck in SIGNED or BROADCAST past a heartbeat, checks the chain by txid, and either advances them (found, confirming) or safely rebroadcasts the identical tx (not found). Nothing that left the hot wallet is ever untracked.

Why this is right: the cold/hot split caps the blast radius of a key compromise; holding before signing keeps the balance and the coins consistent on failure; pre-writing SIGNED+txid makes recovery deterministic against an irreversible chain; and a deterministic transaction identity (nonce or bound UTXOs) turns “broadcast, maybe retry” into exactly-once on a network that would otherwise happily pay twice. At-least-once plumbing over an irreversible chain becomes exactly-once withdrawal.

API Design & Data Schema

API

All mutating calls require an Idempotency-Key.

POST /api/v1/orders
Headers: Authorization: Bearer <token>
         Idempotency-Key: <client-generated UUID, reused on retries>
Body:
{
  "pair":        "BTC-USD",
  "side":        "BUY",              // BUY | SELL
  "quantity":    "0.5",             // string decimal to avoid float error
  "type":        "LIMIT",           // MARKET | LIMIT
  "limit_price": "62000.00"         // required for LIMIT
}
Response 202:
{
  "order_id":   "ord_9a1c...",
  "state":      "ACCEPTED",
  "pair":       "BTC-USD",
  "side":       "BUY",
  "quantity":   "0.5",
  "held":       "31031.00",         // USD held (limit*qty + fee), or collar for market
  "created_at": "2026-07-27T09:30:01Z"
}
Errors:
  400  invalid payload / bad pair / limit required
  402  insufficient balance
  409  idempotency key still in progress (retry shortly)
  422  pair halted / not tradable / below min size
  429  rate limited (tightened during volatility spikes)

Placing an order returns 202, not 201: we accepted and held funds; the fill arrives asynchronously from the matching engine. Reads, lifecycle, and the wallet flows:

GET  /api/v1/orders/{id}                  -> state, filled_qty, avg_fill_price
POST /api/v1/orders/{id}/cancel           -> best-effort cancel (may lose race to a fill)
GET  /api/v1/accounts/orders?status=&cursor= -> paginated history (read model)
GET  /api/v1/accounts/balances            -> per-asset available + held
GET  /api/v1/products/{pair}/book?level=2 -> order book depth (cached, conflated)
GET  /api/v1/products/{pair}/trades       -> recent trade tape
WS   /api/v1/stream                       -> subscribe pairs: book deltas, trades, order/balance updates

POST /api/v1/wallets/{asset}/address      -> get/derive a deposit address for the user
GET  /api/v1/deposits?cursor=             -> deposit history + pending confirmations
POST /api/v1/withdrawals                  -> {asset, amount, destination}; holds + starts saga
  Idempotency-Key required. Response 202 with withdrawal_id and state=PENDING.
GET  /api/v1/withdrawals/{id}             -> state (PENDING..BROADCAST..COMPLETED), txid

Data stores

Different access patterns, different stores. Be explicit.

1. Balances, orders, ledger, deposits, withdrawals - SQL, ACID, sharded by user_id. This is where strong consistency, the available >= amount hold guard, the two-bucket model, and the same-transaction outbox live. A relational database (PostgreSQL, or distributed SQL like CockroachDB/Spanner for HA) is the correct, non-negotiable choice. NoSQL’s eventual consistency would reintroduce the overspend and double-credit races we designed out. Shard by user_id so all of a user’s balances, orders, and ledger entries co-locate - every hold and settlement leg for a user is single-shard. (A trade touches two users on possibly different shards; Settlement handles that as two idempotent, per-user legs linked by trade_id, not one cross-shard lock.)

Table: balances                          -- sharded by user_id
  user_id     BIGINT
  asset       STRING    -- USD | BTC | ETH | ...
  available   NUMERIC   -- spendable (use exact decimal, never float)
  held        NUMERIC   -- reserved by open orders / withdrawals
  version     BIGINT    -- optimistic concurrency
  PRIMARY KEY (user_id, asset)
  Invariant: available >= 0 AND held >= 0

Table: orders                            -- sharded by user_id
  order_id      UUID    PRIMARY KEY
  user_id       BIGINT  INDEX
  pair          STRING  INDEX (user_id, created_at)
  side          CHAR    -- B | S
  qty           NUMERIC
  filled_qty    NUMERIC
  type          STRING  -- MARKET | LIMIT
  limit_price   NUMERIC NULL
  hold_asset    STRING
  hold_amount   NUMERIC
  state         STRING  -- ACCEPTED|OPEN|PARTIALLY_FILLED|FILLED|CANCELED|REJECTED
  version       BIGINT  -- optimistic guard for transitions
  seq           BIGINT  -- matching-engine sequence, once booked
  created_at    TIMESTAMP

Table: ledger_entries                    -- APPEND ONLY, immutable, double-entry
  entry_id    UUID    PRIMARY KEY
  user_id     BIGINT  INDEX
  asset       STRING
  amount      NUMERIC -- signed
  kind        STRING  -- HOLD | RELEASE | TRADE | FEE | DEPOSIT | WITHDRAWAL | ADJUST
  ref_type    STRING  -- ORDER | TRADE | DEPOSIT | WITHDRAWAL
  ref_id      STRING  INDEX  -- trade_id / txid / withdrawal_id
  created_at  TIMESTAMP
  Invariant: per trade_id, value-sum across both users' legs = 0 (fees to fee acct)

Table: trades                            -- from the engine, immutable tape
  trade_id    UUID    PRIMARY KEY
  pair        STRING  INDEX (pair, seq)
  seq         BIGINT  -- engine monotonic sequence
  price       NUMERIC
  qty         NUMERIC
  taker_order UUID
  maker_order UUID
  created_at  TIMESTAMP

Table: deposits                          -- idempotent on-chain-in
  txid        STRING
  vout        INT
  PRIMARY KEY (txid, vout)               -- hard double-credit backstop
  user_id     BIGINT  INDEX
  asset       STRING
  amount      NUMERIC
  block_height BIGINT
  confirmations INT
  state       STRING  -- SEEN | CONFIRMING | CREDITED | ORPHANED

Table: withdrawals                       -- idempotent on-chain-out
  withdrawal_id UUID  PRIMARY KEY
  idem_key      STRING UNIQUE            -- request-level dedup
  user_id       BIGINT INDEX
  asset         STRING
  amount        NUMERIC
  destination   STRING
  txid          STRING NULL INDEX        -- deterministic identity, pre-written
  nonce_or_inputs JSON                   -- bound for deterministic rebuild
  state         STRING  -- PENDING|SIGNED|BROADCAST|CONFIRMED|COMPLETED|FAILED
  created_at    TIMESTAMP

Table: idempotency_keys ( idem_key PK, ref_id, status, response, created_at )
Table: outbox          ( outbox_id PK, ref_id, topic, payload, sent INDEX )

2. Order book (live) - in-memory in the matching engine, journaled to a durable log. The book is not in SQL; it is the engine’s in-memory state, its truth being the sequenced input journal + trade output. Snapshots persist to object storage for fast recovery.

3. Order history / balance read model - NoSQL wide-column (Cassandra/DynamoDB), by user. Customer-facing order and deposit/withdrawal lists are read by user_id in reverse-chronological pages and tolerate eventual consistency. Built by a projector off the event bus so history reads never touch the transactional primary.

4. Market data (book depth, trade tape, tickers) - in-memory + Redis/CDN cache. Conflated snapshots per pair, fanned out over WebSockets; cache-friendly and eventually consistent by design.

5. Address book - address -> (user, asset), sharded KV. Potentially 100M+ HD-derived addresses; a simple high-throughput lookup for the Deposit Watcher.

Why SQL for the core, NoSQL/in-memory for the rest: the money path needs multi-row ACID, row-level locking for the hold guard, and same-transaction idempotency + outbox - exactly what relational databases exist for, and sharding by user_id keeps each user’s holds and settlement legs single-shard. The order book needs microsecond single-writer speed, which is in-memory, not SQL. Order history and market data are key-scoped, read-heavy, and eventual-consistency-tolerant, so they go to wide-column and cache tiers. Match the guarantee to the data: absolute for balances, deposits, withdrawals, and trades; relaxed for history and the moving book display.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix. Most failures here are correctness/availability under the volatility spike, not raw average throughput.

1. Overspend / double-commit of balances under concurrency (breaks first, worst impact). Rapid taps, API bots, or a trade racing a withdrawal spend the same coins twice, and in a custodial system that shortfall is real. Fix: two-bucket balances (available/held) with a conditional UPDATE ... WHERE available >= amount and row locks; a client Idempotency-Key with a DB unique-constraint backstop; sharding by user_id so the guard is a fast single-shard row lock.

2. Hot order book saturation during a volatility spike. ~70-80% of a 10x spike lands on BTC-USD + ETH-USD, so one book takes the punishment. Fix: in-memory single-writer engine per pair (no lock contention, microsecond matches), a durable sequenced journal that absorbs bursts while the engine drains steadily, admission control / per-user rate limits tightened during spikes, and sizing the hot-pair host for its own peak. A hot pair is one engine’s bounded problem, not a system-wide meltdown.

3. Double-credited deposits and reorgs. Rescans, node reconnects, or a reorg credit money the chain can take back. Fix: confirmation threshold before credit, idempotent credit keyed on (txid, vout) with a unique constraint, and explicit reorg rollback (orphaned-before-threshold is never credited; the rare deep reorg posts a compensating entry). Deposits are only spendable once final enough.

4. Double-broadcast or lost withdrawals against an irreversible chain. A crash or ambiguous RPC sends coins twice or loses track of coins that left. Fix: pre-write SIGNED + txid before broadcast, a deterministic transaction identity (account nonce or bound UTXOs) so a rebroadcast is the same on-chain payout, a recovery sweeper that checks the chain by txid, and hot/cold split so a key compromise caps loss at the hot float.

5. The dual-write problem (DB commit vs bus publish). Funds held but the match event lost leaves an order accepted that never books. Fix: the outbox pattern - order.accepted is inserted in the same ACID transaction as the hold; a relay publishes at-least-once; the engine dedupes on order_id; the sweeper re-drives orders stuck in any non-terminal state.

6. Matching-engine single point of failure. The per-pair engine is a single logical writer; if it dies, that market halts. Fix: snapshot + journal replay onto a hot standby that resumes from the journal position; because trades carry a monotonic seq, downstream dedupes any replayed trade. Failover is seconds, lossless, and deterministic.

7. Balance-read and market-data spike. ~35K/sec balance/book reads during a spike would hammer the core and the engine. Fix: separate read layer - balance read model projected off the bus, conflated book snapshots cached in Redis/CDN and streamed over WebSockets, subscription-scoped fan-out per pair. The transactional core and the engine never serve display reads.

8. Cross-shard trade settlement. A trade pairs two users who may live on different shards; a distributed lock on the hot path would be slow. Fix: no cross-shard transaction - Settlement applies the trade as two independent, idempotent per-user legs keyed on trade_id, each single-shard. The trade stream is the source of truth; a per-order integrity check reconciles both legs. Eventual within milliseconds, never inconsistent because each leg is idempotent and the trade is immutable.

9. Blockchain node dependency (lag, lies, outage). A single full node can lag, fork-follow wrongly, or go down. Fix: multiple redundant full nodes per chain, quorum on chain head, ordered block scanning with a persisted last-scanned height and safe idempotent re-scan; withdrawals queue and hold (state persists) during a node outage rather than dropping.

10. Proof-of-reserves drift / durability. Internal balances must always match on-chain holdings, and no committed movement may be lost. Fix: continuous reconciliation - SUM(user balances of asset) + fee/ops accounts == on-chain controlled balance per asset; a suspense account keeps the ledger balanced while breaks are investigated; SQL with synchronous replication and automatic per-shard failover loses no committed transaction; the outbox, sweeper, and journal guarantee nothing in flight vanishes silently.

Shard keys, stated plainly: shard balances, orders, ledger_entries, deposits, withdrawals by user_id so every hold and each settlement leg is single-shard - no distributed transactions on the hot path. Shard the matching engine by pair so each book is an independent single-writer engine. The read model and market data partition by user_id and by pair respectively. Never shard the core by asset (BTC would hot-shard every trade) and never by time (concentrates writes on the newest partition). The event bus partitions by pair for per-book ordering into the engine and by user_id for per-user ledger ordering.

Wrap-Up

The trade-offs that define this design:

• Coinbase is a custodian between an ACID ledger and irreversible blockchains, so reserve locally, match deterministically, and bridge the chain exactly-once. The only strongly-consistent steps are the local hold and the ledger settlement; matching is a fast in-memory sequence; deposits and withdrawals are idempotent sagas keyed on on-chain identity, because a confirmed chain transaction cannot be rolled back.
• Hold before you act, always. Two-bucket balances plus a conditional update make overspending and overselling impossible under concurrency, including a trade racing a withdrawal. A conservative collar on market orders (hold high, refund the difference) handles the unknown fill price.
• Pull matching out of the database. An in-memory single-writer engine per pair, fed by a durable sequenced journal, gives microsecond matches, deterministic price-time priority, and snapshot-plus-replay recovery - and per-pair sharding makes a volatility spike on one book a bounded problem.
• Treat the chain as slow, final-eventually, and unforgiving. Credit deposits only after enough confirmations, idempotent on the txid, with reorg rollback; broadcast withdrawals exactly once via a deterministic transaction identity and a hot/cold wallet split that caps compromise loss.
• Prove it, do not trust it. An append-only double-entry ledger and immutable trade tape make every movement reconstructable; continuous proof-of-reserves reconciliation proves internal balances equal on-chain holdings.

One-line summary: a custodial crypto exchange holds funds in a strongly-consistent, user_id-sharded double-entry ledger the instant an order or withdrawal is accepted, matches popular pairs in a deterministic in-memory single-writer engine fed by a sequenced journal, credits on-chain deposits exactly once after confirmations with reorg rollback, and broadcasts withdrawals exactly once from a hot/cold wallet via a deterministic transaction identity - staying correct through a 10x volatility spike on a market that never closes.