“Let a user buy and sell stocks” sounds like a form that writes a row. Then you notice where the money and the shares actually live. Robinhood does not match your buy against someone’s sell - an exchange does that. Robinhood is the broker: the stateful intermediary that holds your cash and your positions, reserves your buying power the instant you place an order, forwards that order to a real exchange or market maker, tracks it through a lifecycle it does not fully control, and settles the fill back into your portfolio - all while showing you a portfolio value that moves with the market in real time. A fill at the exchange is irreversible. You cannot un-buy 100 shares of AAPL. So the broker’s core invariant is brutal: never let a user place an order they cannot cover, never lose or duplicate an order sent to the exchange, and never let the positions ledger disagree with reality.
The second thing that reshapes the design is the clock. This is not a smooth 24/7 load. The US market opens at 9:30 ET and the first few minutes are a wall of orders and a firehose of price ticks; the close is another. Overnight it is nearly idle. So the system is bursty by an order of magnitude, and the write path (orders) and the read path (portfolios repricing on every tick) spike together. The engineering is: a correct, strongly-consistent order-and-ledger core that cannot oversell buying power, wrapped in an elastic real-time layer that survives the 9:30 spike. Let me build it properly.
Functional Requirements (FR)
In scope:
- Place a buy/sell order. A user submits an order for a symbol: side (buy/sell), quantity, type (market or limit), and time-in-force. The system validates buying power (buys) or share availability (sells), reserves it, and forwards the order to an execution venue.
- Order lifecycle and history. Every order moves through a lifecycle - accepted, routed, working, partially filled, filled, canceled, rejected - and the user can see current status and full historical order list.
- Cancel / modify a working order. A user can cancel or replace an order that has not fully filled. The broker forwards the cancel to the venue; it is best-effort because a fill can race the cancel.
- Portfolio and positions. Show the user’s holdings: shares per symbol, average cost, cash balance, buying power, and total portfolio value marked to the live market price.
- Real-time prices. Show live (or near-live) prices for symbols the user holds or is watching, and reprice the portfolio as prices move.
- Settlement into the ledger. When a fill comes back from the venue, update cash and positions correctly (double-entry), including partial fills and fees.
Explicitly out of scope (name it so you own the scope):
- The matching engine. I route orders to an exchange / market maker as a black box with an order-entry API (FIX) and execution reports. I do not design price-time-priority matching - that is a separate problem.
- The market-data fan-out infrastructure. I consume a normalized price-tick stream as an input and design how the portfolio layer uses it; I do not build the global exchange-ingestion and edge fan-out system.
- Clearing and settlement mechanics (T+1, DTCC). I model the broker’s internal ledger and treat clearing as an async reconciliation input. Real T+1 settlement, margin lending, and fractional-share inventory are named but not fully built.
- KYC/onboarding, tax-lot reporting, options/crypto. Consumed or deferred; I design cash equities.
The decision that drives everything: the broker is a stateful custodian in front of an irreversible external venue, so buying power and positions live in a strongly-consistent double-entry ledger, orders are a persisted state machine driven by a saga, and the order is sent to the venue exactly once. Correctness comes from reserving funds before routing, ordering the state machine, and idempotency at the venue boundary - not from trusting the exchange to behave.
Non-Functional Requirements (NFR)
- Scale: 25M users, ~5M daily active traders. Average is modest, but the load is bursty: the market open (9:30 ET) and close (16:00 ET) generate order and price-tick spikes 10-20x the daily mean. Peak order throughput is the number that sizes the write path.
- Latency: placing an order (validate + reserve + accept) should return p99 under ~200ms - the user is watching a moving price and wants confirmation. Routing to the venue and the fill are asynchronous; we return “accepted” and stream status updates. Portfolio price updates should feel live, sub-second on-screen.
- Availability: 99.99% during market hours on the order path. Degradation must be safe: if a venue is unreachable we hold or reject clearly, we never silently drop an accepted order or double-send one. Off-hours maintenance is cheap because volume is near zero.
- Consistency: strong for buying power, cash, and positions. A user must never place two orders that together exceed their buying power, never oversell shares they do not hold, and never see cash created or destroyed. The order state machine is strictly ordered and monotonic. Portfolio display value can be eventually consistent (a price tick a few hundred ms stale is fine); the ledger cannot.
- Durability: an accepted order and every ledger entry survive any node loss. A lost “routed” record could mean an order sent to the exchange that we have no memory of - an unaccounted, irreversible trade. Zero loss on the money and order paths.
- Auditability: every order and every cash/position movement is reconstructable for years (FINRA/SEC recordkeeping). Regulators can ask “show me this order’s full path and the buying-power check behind it.”
The tension: the order+ledger core wants strong consistency and perfect durability, which fight throughput; the real-time price+portfolio layer wants massive fan-out and elasticity, which tolerate staleness. So we split the system: a small strongly-consistent core (reserve, state machine, ledger) and a big eventually-consistent read/streaming layer, and we scale them independently.
Back-of-the-Envelope Estimation (BoE)
Real numbers, because they show where the spikes are.
Users and orders:
25M registered users, ~5M daily active traders.
Assume an active trader places ~4 orders/trading day.
=> 5M * 4 = 20M orders/day
Trading day is ~6.5 hours = 23,400 sec
Average order rate = 20M / 23,400 ≈ 855 orders/sec
But volume is NOT uniform. The open (9:30-9:40) can carry ~15% of the
day's orders in ~10 minutes:
0.15 * 20M = 3M orders in 600 sec ≈ 5,000 orders/sec at the open
Peak write path ≈ 5,000-8,000 orders/sec (open + close spikes).
Order status reads and portfolio views:
Users refresh portfolios and check order status far more than they trade.
Say 5M actives * 30 portfolio/status views/day = 150M reads/day
≈ 6,400 reads/sec average, ~40,000/sec at the open.
Plus every price tick reprices open portfolios (streaming, below).
Read:write ≈ 8:1, and reads spike with writes.
Price-tick fan-out (portfolio repricing):
~8,000 US symbols actively traded. At the open, hot symbols tick
many times/sec; assume ~500K ticks/sec aggregate on the consumed stream.
We do NOT push every tick to every user. We fan out per-symbol current
price to subscribers. If 2M users are connected at the open, each
watching ~10 symbols, the interesting set is the union of watched symbols
(≈ all 8,000), conflated to a few updates/sec per symbol per client.
This is a fan-out problem, handled by the streaming layer, not the DB.
Storage per year:
Order row (symbol, side, qty, type, prices, state, refs) ≈ 500 bytes
Execution reports per order (~2-4) ≈ 4 * 200 = 800 bytes
Ledger entries per fill (double-entry, ~4) ≈ 4 * 150 = 600 bytes
Order events (state transitions, ~6) ≈ 6 * 200 = 1,200 bytes
--------------------------------------------------------------------
≈ 3.1 KB per order, round to ~4 KB with indexes
20M orders/day * 4 KB ≈ 80 GB/day
* 252 trading days ≈ 20 TB/year
Retained 6-7 years by regulation, never deleted early: ~130-150 TB over the window. Large but not exotic; the hot transactional set (open orders + current positions) is tiny, and history is archived to cold storage.
Positions / accounts working set:
25M accounts. Cash balance + buying power per account, plus positions
(a user holds maybe ~10 symbols on average):
25M accounts * (~200 bytes cash/account + ~10 * 100 bytes positions)
≈ 25M * 1.2 KB ≈ 30 GB
Fits comfortably in a sharded SQL cluster; hot accounts cache in memory.
The headline: average throughput (~855 orders/sec) is easy; the design is sized by the open/close spike (~5-8K orders/sec of strongly-consistent, must-not-oversell writes) happening at the same time as a ~40K/sec read spike and a 500K-tick/sec repricing firehose. Everything below is about keeping the core correct under that spike and the read layer elastic through it.
High-Level Design (HLD)
The spine: an Order Gateway (stateless, elastic) validates and dedupes on a client key, the Buying Power / Ledger Service reserves funds or shares in one ACID transaction, the Order Management System (OMS) advances a persisted order state machine and drives a Smart Order Router to an external venue over FIX, execution reports flow back through an Execution Handler that settles fills into the ledger, and a separate Portfolio + Market Data layer streams live prices and marks portfolios to market for reads.
┌──────────────────────────────────┐
│ Client (mobile / web app) │
└───────┬───────────────────┬───────┘
place/cancel │ │ portfolio + prices (WS)
(POST order) │ │
┌────────▼─────────┐ ┌─────▼──────────────────────┐
│ Order Gateway │ │ Portfolio / Streaming API │
│ authN/Z, validate │ │ (WebSocket fan-out) │
│ idempotency, rate │ └─────▲──────────────────────┘
└────────┬──────────┘ │ marked-to-market values
│ new order │
┌──────────▼──────────────┐ │
│ Buying Power / Ledger │ │
│ ONE ACID txn: │ │
│ reserve cash (buy) or │ │
│ reserve shares (sell) │ │
│ write order=ACCEPTED │─────┼──► reads: cash, positions
│ + outbox event │ │
└──────────┬───────────────┘ │
│ order.accepted (outbox->bus)
┌──────────▼───────────────┐ │
│ Order Management (OMS) │ │
│ state machine + saga │ │
└───┬───────────────┬───────┘ │
route │ │ record │
┌──────────▼────────┐ ┌───▼──────────┐ │
│ Smart Order Router │ │ Order Event │ │
│ (FIX session mgr) │ │ Log (append) │ │
└──────────┬─────────┘ └──────────────┘ │
│ FIX NewOrderSingle │
┌─────────▼───────────────────────┐ │
│ External Venue │ │
│ (exchange / market maker) │ │
│ IRREVERSIBLE fills │ │
└─────────┬───────────────────────┘ │
│ ExecutionReport (fill/partial/reject/cancel)
┌─────────▼───────────────────────┐ │
│ Execution Handler │ │
│ settle fill -> ledger, advance │ │
│ order state, publish update │────────┘
└─────────┬───────────────────────┘
│
┌─────────▼───────────────────────┐ ┌───────────────────────┐
│ Reconciliation vs clearing / │ │ Market Data Ingest │
│ venue drop-copy (end of day) │ │ (consumed tick stream)│
└───────────────────────────────────┘ └───────────┬───────────┘
per-symbol current price
──► Portfolio/Streaming layer
Synchronous path (what the user waits on, target p99 < 200ms):
- Client calls
POST /orderswith{symbol, side, qty, type, limit_price?, tif}and a client-generatedIdempotency-Key. - Order Gateway authenticates, validates (market open? symbol tradable? qty > 0? limit sane?), rate-limits, and checks the idempotency layer - a known key returns the stored original response and does nothing else.
- The Buying Power / Ledger Service runs one ACID transaction: for a buy, guard
cash_available >= qty * price_estimate + feesand move that amount fromavailabletoreserved; for a sell, guardshares_available >= qtyand reserve the shares. Write theordersrowACCEPTED, insert an outbox event, commit. - Gateway returns 202 Accepted with
order_idand stateACCEPTED. The order is now the OMS’s problem, and the client subscribes to status updates over the WebSocket.
Asynchronous path (the OMS saga drives the rest):
- The OMS consumes
order.accepted, advances the state machine, and asks the Smart Order Router to send the order. The router picks a venue and sends a FIXNewOrderSinglewith a deterministicClOrdIDso the order goes out exactly once. State ->ROUTED/WORKING. - The venue returns ExecutionReports asynchronously: acknowledgment, partial fills, full fill, or reject. The Execution Handler consumes each, updates the ledger in an ACID transaction (convert the reservation into a real cash-for-shares swap, apply fees, handle partials), advances the order state, and publishes an update the client sees.
- Cancels flow the same way: the OMS sends a FIX
OrderCancelRequest; the venue confirms cancel or reports the order already filled (cancel loses the race). On confirmed cancel, the ledger releases the remaining reservation. - End of day, Reconciliation matches our order/ledger records against the venue’s drop-copy and the clearing firm’s report, raising a break for any mismatch.
The structural insight: there is no shared transaction with the exchange, so the order is a durable, monotonic state machine plus a saga, and the only strongly-consistent step is the local reservation and ledger settlement. The portfolio/price layer hangs off the side, reads eventually-consistent projections, and scales on its own axis.
Component Deep Dive
The hard parts, naive-first then evolved: (1) buying-power reservation with no overselling under concurrency, (2) exactly-once order routing over an irreversible venue plus the order state machine, (3) settling fills (including partials) into a double-entry positions ledger, (4) real-time portfolio pricing under the market-open spike.
1. Buying power reservation without overselling
The job: the instant a user places an order, reserve exactly enough cash (buy) or shares (sell) so that concurrent orders can never together exceed what the account has. A user rapidly tapping “buy” on a rising stock must not spend the same dollar twice.
Naive approach: check balance, then place the order
def place_buy(acct, symbol, qty, price):
if acct.cash >= qty * price: # read
route_order(symbol, qty) # send to venue
acct.cash -= qty * price # write, later
Where it breaks:
- Time-of-check to time-of-use race. Two orders read
cash = 1000at the same moment, both pass the>= 800check, both route. The account just bought $1600 of stock with $1000. The overshoot is real shares the broker must pay for and claw back from a customer who does not have it. - Check-then-route-then-debit is three steps with crashes in between. Route succeeds, the process dies before debiting cash: the order is working at the venue but the account still shows full buying power, so it can place another fully-funded order. Double spend by crash.
- Market orders have no fixed price. You do not know the fill price when you place a market buy, so
qty * priceis a guess; a naive exact debit is wrong the moment the price moves.
Read-modify-write with no lock and no reservation prints buying power out of thin air.
Evolved approach: two-bucket balances + a conditional reserve in one ACID transaction, with a price collar for market orders
Split every balance into available and reserved. Placing an order does not debit - it reserves. In one ACID transaction, guarded against races:
BEGIN;
-- BUY: reserve cash. The WHERE clause is the concurrency guard.
UPDATE accounts
SET cash_available = cash_available - :reserve_amount,
cash_reserved = cash_reserved + :reserve_amount,
version = version + 1
WHERE account_id = :acct
AND cash_available >= :reserve_amount; -- 0 rows => insufficient => ROLLBACK
INSERT INTO orders (order_id, account_id, symbol, side, qty, type,
limit_price, reserved_amount, state)
VALUES (:oid, :acct, :sym, 'BUY', :qty, :type, :lim, :reserve_amount, 'ACCEPTED');
INSERT INTO outbox (order_id, topic, payload)
VALUES (:oid, 'order.accepted', ...);
COMMIT;
- The conditional
UPDATE ... WHERE cash_available >= amountis the guard. Row locks serialize concurrent orders on the same account; the second either still fits or fails and rolls back. No lost update, no over-reservation, no external lock service. - Sells reserve shares, not cash. The same pattern against a
positionsrow:UPDATE positions SET shares_available = shares_available - :qty, shares_reserved = shares_reserved + :qty WHERE account_id=:a AND symbol=:s AND shares_available >= :qty. This is what stops a user from selling 100 shares twice or selling shares they do not hold. - Market-order price collar. For a market buy we do not know the fill price, so we reserve a collar:
reserve_amount = qty * (last_price * (1 + buffer)) + fees, withbuffersized to a plausible worst-case move (e.g. 5%). We reserve slightly more than expected; when the fill comes back at the real price, we release the difference back toavailable. Reserving high and refunding is safe; reserving low and overshooting is not. - Reservation is released or converted, never leaked. Every terminal path - fill, cancel, reject, expire - either converts the reservation into a real ledger movement or releases it back to
available. A recovery sweeper reconciles any order stuck with a reservation but no active venue order.
Why this is right: the reservation makes buying power committed the instant the order is accepted, so no concurrent order can double-spend it; the conditional update makes the race impossible at the row level; and the collar handles the unknown fill price of market orders by erring toward over-reserving. The one place the money is still fully in our control - before routing - is exactly where we lock it.
2. Exactly-once order routing over an irreversible venue + the order state machine
The job: the order must reach the exchange exactly once, even though the OMS can crash mid-route, the bus is at-least-once, and the FIX send can fail ambiguously. A duplicate routed order is a second real trade the customer never wanted, and you cannot un-fill it.
Naive approach: consume the event, send FIX, mark it routed
def on_accepted(evt):
fix.send_new_order(evt.symbol, evt.qty, evt.side) # external
db.set_state(evt.order_id, 'ROUTED')
# commit offset
Where it breaks:
- Crash between send and record. FIX message goes out, the OMS dies before writing
ROUTEDor committing the offset. On restart the event redelivers and the order is routed again. Two live orders at the venue, potentially two fills. The customer bought 200 shares when they asked for 100, and it is irreversible. - Ambiguous FIX failure. You send; the TCP session drops before the venue ack. Did the exchange get it? Retrying may double it; not retrying may lose it. The transport error alone cannot say.
- Overwritable status loses ordering and history. A late ack can flip
FILLEDback toWORKING; nothing forbids illegal transitions, and the path is erased on each overwrite - fatal for an auditable record.
At-least-once delivery plus a non-idempotent, irreversible external send is the worst case: duplicated real trades you cannot undo.
Evolved approach: pre-written ROUTING state + deterministic ClOrdID + a monotonic order state machine
1. Order is a persisted, monotonic state machine. Transitions are validated against an allowed-transition table and applied with an optimistic guard.
ACCEPTED -> ROUTING -> ROUTED -> WORKING -> PARTIALLY_FILLED -> FILLED (happy path)
| | | | |
| | | v v
| | | CANCELED FILLED (remaining fills)
| | v
| | REJECTED (venue rejects; release reservation)
| v
| (crash here: recovery re-drives from ROUTING deterministically)
+------> REJECTED (failed local validation; release reservation)
- A transition uses
UPDATE orders SET state=:new, version=version+1 WHERE order_id=:id AND state=:expected AND version=:v. Zero rows changed means someone else moved it; the caller re-reads. This makes retried execution reports and concurrent handlers idempotent - a duplicateFILLEDreport finds the order alreadyFILLEDand is a no-op. - Terminal states are final.
FILLED,CANCELED,REJECTED,EXPIREDhave no outgoing transition. Order state only moves forward.
2. Pre-write the route intent, then send, then confirm. Before sending FIX, the OMS writes ROUTING with the exact ClOrdID it will use, committed. Only then does it send. After any crash, recovery sees ROUTING with a known ClOrdID and knows precisely which order to check on / retry - never “did I send something?”
3. Deterministic ClOrdID toward the venue. FIX orders carry a client order id (ClOrdID) that the venue treats as the order’s identity. We derive it deterministically: ClOrdID = order_id (plus a suffix for replaces), and send the same id on every attempt. A well-behaved venue rejects a duplicate ClOrdID (11= already seen), so a retry after an ambiguous failure cannot create a second live order. Where the venue does not dedupe, the FIX drop-copy feed (a mirror of all our executions) lets recovery check “is there already a live order for this ClOrdID?” before resending.
def route_order(order):
if order.state in ('ROUTED', 'WORKING', 'FILLED', 'PARTIALLY_FILLED'):
return # already out, no-op
if order.state != 'ACCEPTED' and order.state != 'ROUTING':
raise IllegalTransition
clordid = order.id # deterministic
db.set_state(order.id, 'ROUTING', clordid=clordid) # pre-write, commit
resp = fix.send_new_order(order, clordid=clordid) # SAME id on every retry
if resp.acked:
db.set_state(order.id, 'ROUTED', venue_ref=resp.exchange_order_id)
elif resp.permanent_reject:
with db.transaction():
ledger.release_reservation(order) # give buying power back
db.set_state(order.id, 'REJECTED', reason=resp.reason)
else:
raise RetryableError # transient: re-drive with backoff
4. Outbox for the trigger event. order.accepted is written in the same ACID transaction as the reservation (outbox pattern), so reserving buying power and the signal to route commit together. A relay publishes at-least-once; the OMS dedupes on order_id. There is no window where funds are reserved but nothing routes the order, nor one where an order routes without funds reserved.
5. Recovery sweeper. A periodic job scans for orders stuck in ROUTING (crash mid-send) or ROUTED/WORKING past a heartbeat, and reconciles them against the drop-copy - safely, because the deterministic ClOrdID and drop-copy make a resend either a no-op or a positive confirmation of what really happened.
Why this is right: you cannot get exactly-once from an irreversible venue on trust, so you compose it - the state machine makes routing unreachable until funds are reserved, the pre-written ROUTING state makes recovery deterministic, and the deterministic ClOrdID plus drop-copy make the external send happen once regardless of retries. At-least-once plumbing over an irreversible venue becomes exactly-once order routing.
3. Settling fills into a double-entry positions ledger
The job: when execution reports come back - possibly as a series of partial fills at different prices - update cash and positions correctly, keep average cost accurate, apply fees, and never create or destroy money or shares. This must survive duplicate and out-of-order reports.
Naive approach: on fill, subtract cash and add shares
def on_fill(order, fill):
acct.cash -= fill.qty * fill.price
pos.shares += fill.qty # buy
save(acct); save(pos)
Where it breaks:
- Duplicate execution reports double-count. FIX feeds redeliver; the venue resends a fill on reconnect. Applied twice, the account is charged twice and holds twice the shares it bought. Nothing detects the replay.
- Partial fills are not one event. A 1000-share order can fill as 300 @ 150.01, 500 @ 150.02, 200 @ 150.05. Treating “the fill” as one number is wrong for cash, wrong for average cost, and wrong for the reservation release.
- No double-entry means no integrity check. Directly mutating two mutable fields gives you no way to prove cash-out equals shares-in-value; a bug silently unbalances the books and nobody notices until a customer complains.
- The reservation is never reconciled. You reserved a collar; the fill came in cheaper. Without an explicit release step the extra cash is stranded in
reservedforever.
Direct field mutation on at-least-once fills is unauditable and double-counts.
Evolved approach: idempotent fill application + append-only double-entry ledger + reservation drawdown
Each execution report is applied idempotently. Every fill carries a venue exec_id; we record processed exec ids and apply each exactly once.
def apply_fill(order, exec):
with db.transaction():
if db.exists(processed_execs, exec.exec_id): # dedup guard
return
db.insert(processed_execs, exec.exec_id)
gross = exec.qty * exec.price
# BUY: draw from the reservation, credit shares
ledger_post([
entry(order.account, 'CASH', -gross, kind='SETTLE'),
entry(order.account, 'POSITION', +exec.qty, symbol=order.symbol),
entry(order.account, 'CASH', -exec.fee, kind='FEE'),
])
release_from_reserved(order.account, gross + exec.fee) # move out of 'reserved'
pos = get_position(order.account, order.symbol)
pos.avg_cost = (pos.avg_cost*pos.shares + gross) / (pos.shares + exec.qty)
pos.shares += exec.qty
pos.shares_available += exec.qty
order.filled_qty += exec.qty
order.state = 'FILLED' if order.filled_qty == order.qty else 'PARTIALLY_FILLED'
- Double-entry. Every fill posts balanced entries into an append-only
ledger_entriestable: cash decreases, position value increases (for a buy), fees debit a fee account. The invariantSUM of all entries in value terms == 0per order lets an integrity check prove the books balance. Nothing is ever mutated in place; the position/cash balances are projections of the entries. - Partial fills are natural. Each partial is one idempotent
apply_fillthat draws down the reservation and advancesfilled_qty. The order flips toFILLEDonly whenfilled_qty == qty. Average cost is recomputed per partial at the actual fill price. A partial that never completes and then cancels releases the remaining reservation. - Reservation drawdown and refund. Each fill moves cash out of
reserved(notavailable); when the order reaches a terminal state, any leftover reservation (the collar overshoot on a market buy, or the unfilled portion) is released back toavailable. The two-bucket model from part 1 closes the loop here. - Out-of-order safety. Because the ledger is append-only and every application is idempotent and guarded by the order state machine, a report that arrives late or twice is either applied once or ignored - never applied against a terminal order in a way that corrupts state.
Why this is right: idempotent application on exec_id kills duplicate-report double-counting, double-entry makes the ledger provably balanced and auditable, partial fills fall out naturally because each is just another balanced posting, and the reservation drawdown/refund ties settlement back to the buying-power lock so no cash is stranded or conjured. The positions the user sees are a projection of an immutable, balanced ledger.
4. Real-time portfolio pricing under the market-open spike
The job: show a portfolio value that moves with the market, for millions of connected users, when the 9:30 open unleashes a tick firehose and a connection spike at the same time - without hammering the transactional core.
Naive approach: on every tick, recompute every affected portfolio from the DB
def on_tick(symbol, price):
for user in users_holding(symbol): # DB query per tick
portfolio = load_positions(user) # DB read per user
value = sum(p.shares * price_of(p) for p in portfolio)
push(user, value)
Where it breaks:
- A query per tick per holder melts the database. A hot symbol ticking hundreds of times/sec, held by hundreds of thousands of users, is millions of DB reads/sec at the exact moment the transactional core is busiest accepting orders. The order path and the pricing path fight over the same database and both fall over.
- Recomputing the whole portfolio on one symbol’s tick is wasteful. Most positions did not change; you resend an unchanged number.
- Pushing every tick to every client floods the connection and the phone. Nobody needs 200 updates/sec on screen.
Coupling repricing to the transactional store and fanning every tick to every user collapses under the open spike.
Evolved approach: separate read layer, in-memory current-price map, conflation, and delta repricing
Fully separate the pricing/portfolio read path from the order/ledger core. Positions are projected into a read store (and cached in memory per connected user); prices come from the consumed market-data stream, not the DB.
- Current-price map in memory. A price service holds
symbol -> current_price(only ~8,000 symbols, tens of KB) updated by the tick stream. Portfolio math reads this map, never the DB, for the mark. This decouples repricing entirely from the transactional store. - Conflation, not every tick. The streaming layer collapses many ticks per symbol into at most a few updates/sec per symbol (send the latest, drop intermediate). Users see a smoothly moving number without 200 updates/sec. This is the standard tick-conflation trick and it caps both bandwidth and client work.
- Subscription-driven fan-out. Each connected client subscribes to the symbols in its portfolio + watchlist over a WebSocket. The server maintains a
symbol -> set(connections)index; a conflated price update fans out only to subscribers of that symbol. One update per symbol per interval, not per user per tick. - Delta repricing on the client (or edge). The server pushes changed symbol prices; the client holds its share counts (loaded once, updated on fills) and recomputes only the affected line and the portfolio total. A single symbol’s tick updates one line, not a full reload. Share counts change rarely (only on fills), so they are pushed as separate, infrequent events off the ledger’s update stream.
- Elastic, stateless streaming tier. WebSocket servers are stateless behind a connection-aware load balancer and autoscale for the open/close spike, then scale down overnight. Connection state (subscriptions) lives in the streaming node with a shared subscription index in Redis, so a node loss just reconnects clients elsewhere.
Why this is right: repricing never touches the transactional core, so the order path and the pricing path stop fighting; the in-memory price map plus per-symbol conflation turns a 500K-tick/sec firehose into a bounded, few-updates-per-second-per-symbol fan-out; subscription-scoped delivery means each update goes only to users who care; and the whole tier scales independently and elastically to absorb the 9:30 spike. Display value is eventually consistent by design, which is exactly the guarantee a moving price needs - and never the guarantee the ledger uses.
API Design & Data Schema
API
All mutating calls require an Idempotency-Key.
POST /api/v1/orders
Headers: Authorization: Bearer <token>
Idempotency-Key: <client-generated UUID, reused on retries>
Body:
{
"symbol": "AAPL",
"side": "BUY", // BUY | SELL
"quantity": 100,
"type": "LIMIT", // MARKET | LIMIT
"limit_price": 150.00, // required for LIMIT
"tif": "DAY" // DAY | GTC | IOC | FOK
}
Response 202:
{
"order_id": "ord_7f2c...",
"state": "ACCEPTED",
"symbol": "AAPL",
"side": "BUY",
"quantity": 100,
"reserved": 15075.00, // cash reserved (limit*qty + fees), or collar for market
"created_at": "2026-07-27T13:30:01Z"
}
Errors:
400 invalid payload / bad symbol / limit required
402 insufficient buying power (buy) or shares (sell)
409 idempotency key still in progress (retry shortly)
422 market closed / symbol halted / not tradable
429 rate limited
Placing an order returns 202, not 201: we accepted and reserved funds and will route, but the final state (filled, rejected) arrives asynchronously. Reads and lifecycle:
GET /api/v1/orders/{id} -> current state, filled_qty, avg_fill_price, refs
GET /api/v1/orders/{id}/events -> immutable event log (audit view)
GET /api/v1/accounts/{id}/orders?status=&cursor= -> paginated order history (read model)
POST /api/v1/orders/{id}/cancel -> best-effort cancel (may lose the race to a fill)
PUT /api/v1/orders/{id} -> replace (cancel/replace: new ClOrdID)
GET /api/v1/accounts/{id}/portfolio -> cash, buying_power, positions, marked value
GET /api/v1/accounts/{id}/positions -> shares, avg_cost per symbol
WS /api/v1/stream -> subscribe symbols; receive price + order-status pushes
Data stores
Different access patterns, different stores. Be explicit.
1. Accounts, positions, orders, ledger, state machine - SQL, ACID, sharded by account_id. This is where strong consistency, the available >= amount reservation guard, the two-bucket model, and the same-transaction outbox live. A relational database (PostgreSQL, or distributed SQL like CockroachDB/Spanner for HA) is the correct, non-negotiable choice. NoSQL’s eventual consistency would reintroduce the over-reservation and double-spend races we designed out. Shard by account_id so all of a user’s orders, positions, cash, and ledger entries co-locate in one shard - every order-placement transaction is single-shard, no distributed transaction needed.
Table: accounts -- sharded by account_id
account_id BIGINT PRIMARY KEY
cash_available BIGINT -- minor units (cents), spendable
cash_reserved BIGINT -- reserved by working buy orders
version BIGINT -- optimistic concurrency
status STRING -- active | restricted | closed
Invariant: cash_available >= 0 AND cash_reserved >= 0
Table: positions -- (account_id, symbol) composite key
account_id BIGINT
symbol STRING
shares BIGINT -- total held
shares_available BIGINT -- sellable (not reserved by a working sell)
shares_reserved BIGINT
avg_cost BIGINT -- minor units per share
PRIMARY KEY (account_id, symbol)
Table: orders -- state projection, sharded by account_id
order_id UUID PRIMARY KEY
account_id BIGINT INDEX
symbol STRING INDEX (account_id, symbol)
side CHAR -- B | S
qty BIGINT
filled_qty BIGINT
type STRING -- MARKET | LIMIT
limit_price BIGINT NULL
tif STRING
state STRING -- ACCEPTED|ROUTING|ROUTED|WORKING|
-- PARTIALLY_FILLED|FILLED|CANCELED|REJECTED|EXPIRED
version BIGINT -- optimistic guard for transitions
reserved_amount BIGINT
clord_id STRING -- deterministic id sent to venue
venue_ref STRING NULL -- exchange order id
avg_fill_price BIGINT NULL
created_at TIMESTAMP
updated_at TIMESTAMP
Table: ledger_entries -- APPEND ONLY, immutable, double-entry
entry_id UUID PRIMARY KEY
account_id BIGINT INDEX
order_id UUID INDEX
account STRING -- CASH | POSITION:<symbol> | FEE
amount BIGINT -- signed minor units (or shares for POSITION)
kind STRING -- RESERVE | SETTLE | FEE | RELEASE | ADJUSTMENT
exec_id STRING NULL INDEX -- venue exec id, for dedup
created_at TIMESTAMP
Invariant: per order_id, value-sum of entries = 0
Table: order_events -- APPEND ONLY, audit log
event_id UUID PRIMARY KEY
order_id UUID INDEX
seq INT -- per-order monotonic
type STRING -- state change | exec report | cancel | operator action
payload JSON -- old->new state, exec detail, reservation delta
created_at TIMESTAMP
UNIQUE(order_id, seq)
Table: processed_execs -- idempotency for execution reports
exec_id STRING PRIMARY KEY
order_id UUID
created_at TIMESTAMP
Table: idempotency_keys
idem_key STRING PRIMARY KEY -- UNIQUE; the real dedup guarantee
order_id UUID NULL
status STRING -- IN_PROGRESS | DONE
response JSON
created_at TIMESTAMP -- TTL ~24h hot
Table: outbox
outbox_id BIGINT PRIMARY KEY
order_id UUID
topic STRING
payload JSON
sent BOOL INDEX (partial: WHERE sent=false)
2. Order history / portfolio read model - NoSQL wide-column (Cassandra/DynamoDB), by account. Customer-facing order lists are read by account_id in reverse-chronological pages and tolerate eventual consistency. Built by a projector off the event bus so history reads never touch the transactional primary. Partition key account_id, clustering by created_at desc.
3. Current-price map - in-memory (per streaming node), fed by the tick stream. symbol -> price, ~8,000 entries, tens of KB. Never the source of truth for fills (that is the exec report), only for marking portfolios to market.
4. Subscription index - Redis. symbol -> set(connection_id) and connection_id -> node, so conflated price updates fan out to the right connections and reconnects rebind cleanly.
5. Audit / history cold storage - object storage (WORM). order_events and ledger_entries are continuously exported for the 6-7 year retention window, immutable and never purged early.
Why SQL for the core, NoSQL for the projections: the order-and-money path needs multi-row ACID, row-level locking for the reservation guard, monotonic state transitions, and same-transaction idempotency+outbox - exactly what relational databases exist for, and sharding by account_id keeps every order transaction single-shard. Order history and portfolio displays are key-scoped, read-heavy, and eventual-consistency-tolerant, so they go to a wide-column store and an in-memory price map. Match the guarantee to the data: absolute for cash, shares, and orders; relaxed for the read projections and the moving price.
Bottlenecks & Scaling
Where it breaks first, in order, and the fix. Most failures here are correctness/availability under the open spike, not raw average throughput.
1. Over-reservation / double-spend of buying power under concurrency (breaks first, worst impact). Rapid taps or retries let two orders spend the same cash or sell the same shares.
Fix: two-bucket balances (available/reserved) with a conditional UPDATE ... WHERE available >= amount and row locks; a client Idempotency-Key with a DB unique-constraint backstop so a retried place is one order; sharding by account_id so the guard is a fast single-shard row lock.
2. Duplicate irreversible order routing. A retry or crash sends a second FIX order that fills for real.
Fix: pre-written ROUTING state + deterministic ClOrdID on every send so the venue dedupes; the drop-copy feed to confirm what is already live before a resend; the recovery sweeper to re-drive stuck orders safely. This is the single most important mechanism after buying-power correctness.
3. Duplicate / out-of-order execution reports corrupting the ledger. FIX redelivery double-counts fills.
Fix: idempotent application on exec_id (a processed_execs guard), an append-only double-entry ledger with a per-order zero-sum invariant, and the monotonic order state machine so a late report cannot mutate a terminal order.
4. The dual-write problem (DB commit vs bus publish). Funds reserved but the routing event lost leaves an order accepted that never routes.
Fix: the outbox pattern - order.accepted is inserted in the same ACID transaction as the reservation; a relay publishes at-least-once; the sweeper re-drives orders stuck in any non-terminal state.
5. Market-open write spike on the SQL core (~5-8K strongly-consistent orders/sec). The open floods the reservation path.
Fix: shard by account_id so writes spread across shards and each order is single-shard; a stateless, autoscaling Order Gateway in front; admission control / rate limits per account and global; connection pooling and prepared statements to keep the per-order transaction cheap. At ~8K single-shard writes/sec across N shards this is comfortable.
6. Portfolio repricing melting the database at the open. A query per tick per holder is millions of reads/sec on the busy core.
Fix: separate read layer entirely - in-memory symbol->price map, tick conflation to a few updates/sec per symbol, subscription-scoped WebSocket fan-out, and delta repricing on the client. The transactional core never sees a repricing read.
7. Hot symbols / hot connection spikes. A single meme stock’s ticks and a flood of subscribers to it concentrate load. Fix: conflation caps per-symbol update rate regardless of tick volume; the subscription index and stateless streaming tier autoscale; fan-out for a hot symbol is parallelized across streaming nodes, each owning a subset of connections. Hot accounts (a whale placing many orders) still shard to one node, mitigated by per-account rate limits.
8. Venue outage or ambiguous FIX responses. The exchange/market maker is down or answers ambiguously. Fix: the Smart Order Router fails over to a secondary venue where allowed; typed retries with backoff; a circuit breaker; orders queue and hold (state persists) rather than drop; a DLQ for orders that exhaust retries, worked by ops. If no venue is reachable, new orders are rejected clearly (market closed / venue down), never silently dropped or double-sent.
9. Ledger drift from the venue/clearing. Missed reports, fees, or corporate actions diverge our books from reality.
Fix: end-of-day reconciliation against the venue drop-copy and the clearing firm’s report; a suspense account that keeps the ledger balanced while breaks are investigated; internal integrity checks (per-order value-sum == 0, shares = SUM(position entries)); a rising suspense balance pages someone.
10. Single points of failure / durability. Any node loss must lose no accepted order or ledger entry. Fix: SQL with synchronous replication and automatic failover per shard (no committed transaction lost); bus, read model, and Redis replicated; stateless Gateway, OMS, and streaming tiers horizontally scaled; the outbox, DLQ, and sweeper guarantee nothing in flight vanishes silently.
Shard keys, stated plainly: shard accounts, positions, orders, ledger_entries, order_events all by account_id so every order-placement and settlement transaction is single-shard - no distributed transactions on the hot path. The read model partitions by account_id for single-partition reverse-chronological history. The event bus (Kafka) partitions by account_id for per-account ordering, or by order_id where only per-order ordering matters. Never shard by symbol on the core (a hot symbol would hot-shard every trader’s writes) and never by time (concentrates all current writes on the newest partition).
Wrap-Up
The trade-offs that define this design:
- The broker is a custodian in front of an irreversible venue, so reserve locally, route once, settle asynchronously. The only strongly-consistent steps are the local reservation and the ledger settlement. Routing and fills are an ordered sequence of idempotent, recoverable steps, because a trade at the exchange cannot be rolled back.
- Reserve before you route, always. Two-bucket balances plus a conditional update make over-spending buying power and overselling shares impossible under concurrency. We accept a slightly conservative collar on market orders (reserve high, refund the difference) to handle the unknown fill price.
- Exactly-once routing over an irreversible venue. A client idempotency key with a DB unique-constraint backstop, a pre-written
ROUTINGstate, a deterministicClOrdIDon every send, and the drop-copy for confirmation turn at-least-once plumbing into exactly-once order routing. - Split the system by consistency need. A small strongly-consistent,
account_id-sharded SQL core for orders/cash/positions; a big eventually-consistent, elastic read+streaming layer for prices and portfolios. They scale on separate axes so the 9:30 open spike hits each where it is cheap to absorb. - Prove it, do not trust it. An append-only double-entry ledger and immutable order-event log make every order and every cent reconstructable; end-of-day reconciliation against the drop-copy and clearing report proves the books match reality.
One-line summary: a retail brokerage reserves buying power in a strongly-consistent, account_id-sharded double-entry ledger the instant an order is accepted, drives a monotonic order state machine that routes to an external venue exactly once with a deterministic ClOrdID, settles fills (including partials) idempotently back into the ledger, and prices millions of portfolios through a separate, conflated, autoscaling streaming layer that absorbs the market-open spike without ever touching the transactional core.
Comments