“Show live prices for every stock on every exchange to millions of people at once” reads like a websocket that pushes a number. Then you look at what is actually flowing. Global exchanges emit millions of price ticks per second across hundreds of thousands of symbols. You have millions of users connected, each watching a different handful of symbols, each expecting the number to move within a blink of it moving on the exchange floor. The problem is not “get a price,” it is fan-out: one tick for a hot symbol must reach potentially a million users’ screens in under 100ms, and you cannot afford to send every tick to every user or run a query per user per tick.
The core insight is that this is a write-once, read-by-millions streaming problem where the write firehose (ticks) and the read fan-out (subscribers) are both enormous but connected by a narrow waist: the set of distinct current prices. There are only ~300K symbols, each with one current price, but millions of users subscribing to overlapping subsets. So the job is to collapse the firehose to a per-symbol current-price stream once, then fan that out along a subscription tree to millions of connections, at the edge, close to the user, within the latency budget. Let me build it properly.
Functional Requirements (FR)
In scope:
- Live price stream. A user subscribes to a set of symbols (a watchlist, a portfolio, a single quote page) and receives price updates pushed to them in real time as the market moves. A price update is
{symbol, last, bid, ask, volume, ts}. - All symbols, all exchanges. Equities, ETFs, indices, FX, futures across global exchanges (NYSE, NASDAQ, LSE, NSE, TSE, and so on), normalized into one symbol namespace with currency and exchange metadata.
- Charts. Historical price series at multiple resolutions (1m, 5m, 1h, 1d candles) for a symbol, plus the live tail so a chart updates as ticks arrive.
- Alerts. “Notify me when AAPL crosses $250.” A threshold-crossing notification hooked off the same price stream.
- Subscribe / unsubscribe dynamically. A user scrolls a table, switches tabs, opens a symbol - the set of symbols they care about changes constantly and cheaply.
Explicitly out of scope (own the scope):
- Order placement and matching. This is a market-data read plane, not a trading engine. Placing orders, the order book, and matching are a separate system (I designed that separately). Here we consume the resulting trades and quotes.
- The exchange feed licensing and physical connectivity. We consume normalized feeds (FIX/FAST, ITCH, exchange multicast); we do not build the co-located feed handler’s licensing or the cross-connect. We do build the normalization stage.
- The notification delivery pipeline for alerts (push/email/SMS provider fan-out, retries). We produce a “fire this alert” event and hand it to an existing notification service, and I detail the crossing match since it rides the same stream.
- Deep analytics (technical indicators computed server-side, backtesting, options greeks). Charts serve raw OHLCV; indicators are a client or separate-service concern that reads the same candle store.
The decision that drives everything: the read fan-out dwarfs everything else. Ingesting millions of ticks/sec is hard but bounded by the number of symbols. Delivering those ticks to millions of users, each with a different overlapping subscription, within 100ms, worldwide, is the real scale problem. So the whole design is organized around a subscription fan-out tree that collapses the firehose once and multiplies it out to connections at the edge, never per-user-per-tick from the core.
Non-Functional Requirements (NFR)
- Scale: ~300K tradable symbols worldwide. Peak ingest ~5M ticks/sec across all exchanges during overlapping market hours, heavily concentrated in a few thousand liquid symbols. ~10M concurrent connected users at peak (streaming apps and web), each subscribed to on average ~30 symbols (a watchlist plus a visible table), so ~300M active subscriptions. Fan-out amplification: a single hot-symbol tick can require delivery to ~1M subscribers.
- Latency: exchange tick to user screen p99 under 100ms end to end. This is the headline number - a quote that is 500ms stale is visibly wrong on a fast-moving stock. Subscribe/unsubscribe acknowledged in under 50ms. Chart history load p99 under 200ms.
- Availability: 99.99% on the streaming path during market hours. A dropped feed or a dead edge means blank or frozen prices, which is an immediately visible, trust-destroying failure.
- Consistency: prices are eventually consistent but monotone per symbol - a client must never see prices go backward in time (an old tick after a newer one). We accept that two users may be a few ms apart; we do not accept a client seeing a stale price after a fresh one. Charts are read-consistent snapshots.
- Durability: the live stream itself is ephemeral (a missed tick is superseded by the next one within ms), but the candle/history store is durable - it is the record of what the market did and feeds charts and compliance. Ticks are also archived for the tape.
- Freshness over completeness on the live path: if a client falls behind, we send it the latest price for its symbols, not every intermediate tick. Conflation is a feature, not a loss.
Back-of-the-Envelope Estimation (BoE)
Real numbers, with arithmetic, because they pick the architecture.
Ingest (the firehose):
Peak ticks: ~5,000,000 ticks/sec (all exchanges, overlapping hours)
Per tick on wire: symbol_id(4) + last(8) + bid(8) + ask(8)
+ vol(4) + ts(8) + seq(4) = ~44 bytes
5M * 44 bytes = ~220 MB/sec = ~1.8 Gbps of raw normalized ingest
5M/sec is bounded by symbols: there are only ~300K symbols, so at peak an average symbol is ~16 ticks/sec, but the distribution is brutally skewed - the top ~2K symbols carry the majority of ticks (SPY, AAPL, TSLA, index futures can each do 50K-100K+ ticks/sec).
The narrow waist (current-price state):
300K symbols * (last+bid+ask+vol+ts ~ 40 bytes) = 12 MB
That is the whole point. The state of the entire world’s prices is ~12 MB. Everything downstream is fan-out of updates to that tiny state, not storage of it.
Fan-out (the read amplification, where the scale is):
10M concurrent users * 30 symbols avg = 300M active subscriptions
Amplification: 5M ticks/sec is the input, but each tick for a
subscribed-to symbol multiplies by its subscriber count.
Worst case naive (send every tick to every subscriber):
A hot symbol at 50K ticks/sec with 1M subscribers
= 50,000 * 1,000,000 = 5 * 10^10 messages/sec for ONE symbol.
That is absurd. This is why conflation is mandatory, not optional.
With conflation to 10 updates/sec per symbol on the live path:
hot symbol: 10 * 1,000,000 = 10M messages/sec for that symbol.
Aggregate delivered messages across all symbols/users:
~10M users * 30 symbols * (avg ~2 visible-symbol updates/sec)
~= a few hundred million delivered messages/sec, spread across
thousands of edge servers.
Conflation is the difference between 5 * 10^10 and 10^7 for one hot symbol. It is the single most important lever on the read side.
Per-edge-server capacity:
Target ~100K concurrent WebSocket connections per edge server
(well within reach for an epoll/async server with small per-conn state).
10M users / 100K per server = ~100 edge servers minimum for connections,
call it a few hundred across regions for headroom and locality.
Each edge holds a subscription table: conn -> symbols, symbol -> conns.
Chart / history storage:
1m candles: 300K symbols * ~400 trading-minutes/day = 120M candles/day
Per candle: OHLCV + ts ~= 48 bytes -> 120M * 48 = ~5.8 GB/day of 1m candles
* ~250 trading days = ~1.5 TB/year of 1m candles (before rollups/compression)
Coarser resolutions (5m/1h/1d) are rollups, far smaller.
Raw tick tape (for replay/compliance): 5M/sec * 44B * 23400s (6.5h)
~= 5 TB/day compressed a lot -> archived to cold object storage.
The headline: the live state is tiny (12 MB), the ingest is large but symbol-bounded (5M/sec), and the fan-out is the monster (hundreds of millions of delivered msgs/sec). So we spend engineering on collapsing the firehose once and fanning it out at the edge with conflation, and on a separate durable candle pipeline for charts.
High-Level Design (HLD)
Three planes. The ingest plane collapses global exchange feeds into one normalized, sequenced, per-symbol current-price stream. The fan-out plane distributes per-symbol updates from the core out to regional edge servers, which hold the millions of client connections and their subscriptions. The history plane consumes the same tick stream to build durable candles for charts. Alerts ride the tick stream through a matcher. The client always talks to the nearest edge PoP over a persistent WebSocket.
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ NYSE / NASDAQ│ │ LSE / NSE / │ │ FX / Futures│ exchange feeds
│ feeds │ │ TSE feeds │ │ feeds │ (FIX/ITCH/multicast)
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
└──────────────────┼──────────────────┘
┌─────────▼──────────┐
│ Feed Handlers / │ parse, map symbol->symbol_id,
│ Normalizer │ normalize price/currency,
│ (per exchange) │ stamp per-symbol seq, drop dups
└─────────┬──────────┘
│ normalized ticks
┌─────────▼──────────┐
│ Tick Bus (Kafka) │ partitioned by symbol_id
│ partitioned by │
│ symbol_id │
└──┬───────┬───────┬──┘
┌─────────────┘ │ └─────────────┐
│ │ │
┌───────▼────────┐ ┌────────▼────────┐ ┌────────▼─────────┐
│ Price State / │ │ Candle Builder │ │ Alert Matcher │
│ Conflator │ │ (history plane)│ │ (per-symbol │
│ (core, per │ │ 1m OHLCV -> │ │ threshold sets) │
│ symbol shard) │ │ time-series DB │ └────────┬─────────┘
│ latest + 10/s │ └────────┬────────┘ │ fire events
│ conflated feed │ │ ▼
└───────┬─────────┘ ┌────────▼────────┐ notification system
│ per-symbol │ Chart API │
│ conflated │ (reads candles)│
│ updates └─────────────────┘
│
┌─────▼───────────────────────────────────────────┐
│ Fan-out Backbone (region -> PoP) │
│ core publishes per-symbol updates; regional │
│ relays subscribe only to symbols their PoPs need │
└─────┬──────────────────┬───────────────────┬─────┘
│ │ │
┌───────▼──────┐ ┌───────▼──────┐ ┌───────▼──────┐
│ Edge PoP (US)│ │ Edge PoP (EU)│ │ Edge PoP (AS)│ each: ~100K WS conns
│ sub tables: │ │ │ │ │ symbol->conns index
│ conn<->symbol│ │ │ │ │ local conflation
└───────┬──────┘ └───────┬──────┘ └───────┬──────┘
│ WebSocket │ │
┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
│ millions │ │ millions │ │ millions │
│ of clients│ │ of clients│ │ of clients│
└───────────┘ └───────────┘ └───────────┘
Live price flow (the fast path, must be under 100ms):
- A feed handler in an exchange’s region parses the raw feed, maps the exchange symbol to an internal
symbol_id, normalizes the price and currency, stamps a monotonic per-symbol sequence number, drops duplicates from redundant A/B feeds, and publishes onto the Tick Bus partitioned bysymbol_id. - The Price State / Conflator consumes each partition and, per symbol, keeps the latest quote and emits a conflated update at a bounded rate (e.g. up to 10/sec per symbol on the live path, or immediately if the symbol is quiet). This is the narrow waist: raw firehose in, per-symbol current-price stream out.
- The core publishes per-symbol conflated updates onto the fan-out backbone. Regional relays subscribe only to the symbols their PoPs currently have subscribers for, so a symbol nobody in Asia watches is never shipped to Asia.
- Each Edge PoP receives updates for its subscribed symbols, looks up its local
symbol -> [connections]index, and pushes the update over each subscriber’s WebSocket. A final per-connection conflation guards slow clients. - The client renders the new number. Round trip from exchange to screen stays inside the 100ms budget because the only per-user work is a hash-set lookup and a socket write at the edge, near the user.
Subscribe flow (dynamic, cheap):
- Client sends
{"action":"subscribe","symbols":["AAPL","TSLA"]}over its existing WebSocket to its edge PoP. - The edge adds the connection to
symbol -> connsfor each symbol and the symbols toconn -> symbols. If the PoP was not already subscribed to a symbol upstream, it sends a subscribe request up the fan-out backbone to the regional relay (which subscribes to the core if it was not already). Reference-counted per symbol. - The edge immediately pushes the current snapshot for those symbols (from a local last-value cache) so the client shows a price instantly, then streams deltas.
- Unsubscribe reverses this and decrements the reference count; the upstream subscription is torn down only when the last subscriber for a symbol in that PoP leaves.
The structural key: the firehose is collapsed to a per-symbol current-price stream exactly once at the core, and fan-out is a subscription tree (core -> region -> PoP -> connection) where each level only carries the symbols anyone below it wants. Delivery cost is proportional to (subscribers * conflated-update-rate), not to (ticks * users), and the expensive part happens at the edge, near the user, in parallel across hundreds of servers.
Component Deep Dive
The hard parts, naive-first then evolved: (1) fan-out to millions of connections, (2) conflation and the latency/freshness trade-off, (3) global distribution and PoP subscription routing, (4) charts and the alert match off the same stream.
1. Fan-out to millions of connections
The job: one tick for a hot symbol must reach up to a million subscribers within the 100ms budget, and the set of subscribers changes constantly.
Naive approach: query subscribers per tick and push from the core
The obvious first cut: the core, on each tick, looks up who is subscribed and pushes to each of them.
on tick(symbol, price):
subs = db.query("SELECT conn_id, edge FROM subscriptions WHERE symbol=?", symbol)
for s in subs:
send(s.edge, s.conn_id, price)
Where it breaks:
- The fan-out multiplies at the source. A hot symbol at 50K ticks/sec with 1M subscribers is 5 * 10^10 sends/sec for one symbol. The core cannot originate that many messages, and it duplicates identical prices a billion times.
- A subscription query per tick against 300M subscriptions at 5M ticks/sec is 5M lookups/sec into a huge table. No store serves that; and the subscription set is churning every second as users scroll.
- The core holds no client connections - the actual WebSockets live somewhere, and routing every message individually from core to the right edge to the right socket centralizes all the work in the one place that cannot scale horizontally per-connection.
Pushing per-tick, per-subscriber from the core conflates the firehose and the fan-out into one impossible number.
Evolved approach: subscription tree with edge-held connections and per-symbol reference counting
Two moves: hold connections at the edge and make fan-out a reference-counted tree.
Connections live on edge PoPs, indexed both ways. Each edge server owns ~100K WebSocket connections and keeps two in-memory maps:
conn_subs: conn_id -> set<symbol_id> (what this client wants)
symbol_subs: symbol_id -> set<conn_id> (who wants this symbol)
snapshot: symbol_id -> latest quote (for instant subscribe)
When a conflated update for symbol_id arrives at the edge, it does one hash-set lookup (symbol_subs[symbol_id]) and writes to each connection. That is O(subscribers-on-this-edge) local work, no database, no cross-network call. A million subscribers to AAPL are spread across hundreds of edges, so each edge only pushes to its slice (a few thousand), in parallel.
Fan-out is a subscription tree, deduplicated by reference count. The core does not know about individual connections. It publishes each symbol’s conflated update once per downstream relay that wants it. The tree is core -> regional relay -> edge PoP -> connection. At each edge, a symbol is subscribed upstream once regardless of how many local connections want it:
edge subscribe(conn, symbol):
symbol_subs[symbol].add(conn)
if len(symbol_subs[symbol]) == 1: # first local subscriber
upstream.subscribe(symbol) # tell regional relay once
conn_subs[conn].add(symbol)
send(conn, snapshot[symbol]) # instant current price
So the AAPL update travels: core sends it once to each region that has any subscriber, each region sends it once to each PoP that has any subscriber, each PoP fans it to its local subscribers. The multiplication happens only at the leaf, spread across hundreds of edge servers, each doing a bounded amount. The core emits AAPL’s conflated update at ~10/sec to a handful of regional relays, not 10^10 times.
core AAPL update (10/sec)
├── region US ──┬── PoP us-east ── 4,000 conns
│ └── PoP us-west ── 3,500 conns
├── region EU ──── PoP eu-1 ── 2,000 conns
└── region AS ──── PoP as-1 ── 1,200 conns
each arrow carries the update ONCE; leaves fan to connections
Why this is right: it separates the two scaling axes. Ticks-to-current-price is collapsed once at the core (bounded by symbols). Current-price-to-millions is a tree whose internal edges are deduplicated by reference count and whose leaves parallelize across hundreds of edge servers near the users. No single node does more than O(local subscribers) per update.
2. Conflation and the latency/freshness trade-off
The job: a hot symbol emits far more ticks than any human can perceive or any socket should carry; send the freshest price, not every intermediate one, without adding meaningful latency.
Naive approach: forward every tick to every subscriber
Stream each tick through unchanged; the client gets the true tape.
Where it breaks:
- The socket and the client drown. 50K ticks/sec on one symbol to a phone renders as an unreadable blur and saturates the connection; the browser cannot repaint that fast and the bytes are wasted.
- Slow clients build unbounded queues. A client on a weak network cannot drain 50K/sec; the server-side send buffer grows without bound, memory blows up, and the client falls further and further behind - eventually showing prices that are seconds stale, which is worse than a slightly conflated but fresh price.
- Head-of-line staleness. If we queue every tick in order, a backed-up client is served old ticks first; it should instead jump to the newest price and skip the stale ones.
Forwarding every tick optimizes for a completeness nobody needs and destroys freshness for anyone not on a perfect network.
Evolved approach: latest-value conflation at every hop, bounded rate, coalesce not queue
Conflation = keep only the latest per symbol per client. At each hop (core conflator, and again per-connection at the edge) maintain a latest-value cache per symbol and a dirty set of symbols that changed since the last flush. On a fixed cadence (e.g. every 100ms, or immediately if idle), flush the current value of each dirty symbol and clear the set. Intermediate ticks are overwritten in place, never queued.
per symbol (or per conn+symbol):
on tick: latest[symbol] = tick; dirty.add(symbol) # O(1), overwrite
every flush_interval (e.g. 100ms):
for s in dirty: send(latest[s])
dirty.clear()
This bounds the outbound rate to 1 / flush_interval per symbol per client (10/sec at 100ms) regardless of tick rate, and every value sent is the freshest at flush time. A spike-and-retrace within a window shows the client the settled price - which is what it should see. Memory is bounded: one slot per (client, subscribed symbol), never a growing queue.
Per-connection back-pressure. The edge tracks each connection’s send-buffer high-water mark. If a client cannot keep up, its conflation naturally coalesces more (it flushes less often relative to tick rate, so more intermediate ticks are dropped) and it always gets the latest, never a backlog of stale ticks. A truly stuck connection is dropped and told to reconnect and re-snapshot.
Adaptive cadence by symbol activity. A quiet symbol (a few ticks/min) is sent immediately - no point waiting 100ms and adding latency for nothing. A blazing symbol is conflated to the cap. So the flush is “immediate if idle, capped if hot,” which keeps p99 latency low for the common case and only conflates where conflation is needed.
Why (min,max) matters for the chart tail, not the quote. For the live quote the client wants the latest price, so latest-value conflation is exactly right. For the live chart tail the client wants the candle to be correct, so the conflator also tracks the running high/low/volume for the current in-progress candle and ships those alongside, so a conflated quote update never makes the chart lose a wick. The quote and the candle are conflated with different rules on the same stream.
The trade-off stated plainly: we give up delivering every tick (which no client wants or can use) to guarantee bounded memory, bounded bandwidth, and freshest-price delivery even to slow clients. Conflation is the mechanism that makes millions of subscribers to a hot symbol survivable.
3. Global distribution and PoP subscription routing
The job: users are on every continent; the exchange is in one place. Deliver sub-100ms worldwide without shipping every symbol everywhere.
Naive approach: one central cluster, clients connect over the internet
All the streaming servers sit in the exchange’s region; every client worldwide opens a WebSocket to them.
Where it breaks:
- Physics. A user in Singapore talking to a US-east cluster eats ~180ms+ round trip on the network alone; the 100ms budget is blown before any processing. You cannot beat the speed of light with software.
- A single region is a blast radius. One regional outage darkens every user’s prices worldwide. 99.99% is impossible from one location.
- Connection concentration. 10M WebSockets terminating in one region is a colossal single point of load and a fat target for congestion.
A central cluster cannot meet a global sub-100ms budget - the wire time alone exceeds it for distant users.
Evolved approach: edge PoPs near users, per-symbol subscription routing up the tree
Terminate connections at edge PoPs near the user. Clients connect to the nearest PoP (geo-DNS / anycast), so the last mile is short and the WebSocket is regional. The exchange-to-user path becomes: exchange region -> core -> regional relay (one long-haul hop over a dedicated backbone) -> nearby PoP -> user (short). The one unavoidable long-haul hop is on an optimized private backbone, not the public internet, and everything after it is local.
Ship a symbol to a region only if someone there wants it. The subscription tree (deep dive 1) means a symbol’s conflated stream flows to a region only when that region has a subscriber, reference-counted. Tokyo Electron trades in Tokyo; if only Asian users watch it, its stream never crosses to the US or EU backbone. This keeps the backbone carrying the union of subscribed symbols per region (a few tens of thousands typically, not all 300K), which is a small, bounded flow.
Sub routing (reference-counted at each level):
client subscribes NVDA at PoP as-1
-> as-1 has 0 NVDA subs -> as-1 subscribes NVDA to region AS relay
-> region AS has 0 NVDA subs -> subscribes NVDA to core
later client at as-1 also subscribes NVDA -> just refcount++, no upstream msg
all as-1 NVDA subs leave -> as-1 unsubscribes NVDA upstream (refcount 0)
Snapshot-on-connect from a regional last-value cache. Each PoP (or its regional relay) keeps the last value for every symbol it is subscribed to, so a new subscriber gets an instant price without a round trip to the core. When a PoP first subscribes to a symbol upstream, the core replies with the current snapshot before the delta stream, so there is no gap.
Failure handling. PoPs are stateless over rebuildable subscription tables; a client that loses its PoP reconnects (to the same or a failover PoP), re-sends its subscription list, and re-snapshots - a sub-second, self-healing recovery. Regional relays are redundant; the core is replicated per symbol-shard. Because the client re-declares its subscriptions on reconnect, no subscription state needs to survive a PoP crash.
Sequence-gap detection for correctness. Every update carries the per-symbol sequence number. A client (or a PoP) that sees a gap after a reconnect requests a fresh snapshot rather than trusting a possibly-missed delta, preserving the “never show a stale price after a fresh one” guarantee.
Why this is right: latency is won by locality (short last mile, one optimized long hop), the backbone is kept small by per-symbol subscription routing (only carry what a region wants), and availability comes from many independent PoPs with cheap, self-healing reconnect. The 100ms budget is met because the only distant hop is a fast private backbone and all per-user work is local to a nearby edge.
4. Charts and the alert match off the same stream
The job: charts need durable historical candles plus a live tail; alerts need threshold-crossing detection. Both ride the same tick stream but have different durability and access patterns from the live quote path.
Naive approach: build candles by querying the tick archive on demand, scan alerts per tick
For a chart, query the raw tick tape and aggregate on the fly. For alerts, on each tick scan the user’s alerts for this symbol and fire any that match.
Where it breaks:
- On-demand aggregation is too slow. Building a 1-year daily chart by scanning a year of raw ticks for a symbol at chart-load time blows the 200ms budget by orders of magnitude; ticks are billions of rows.
- Re-firing alerts.
price >= targetmatches on every tick once the price is past the target, so a one-shot alert fires forever. It tests “is past,” not “just crossed.” - Scan per tick. Matching by scanning alerts on every one of 5M ticks/sec is the same firehose-vs-scan trap as the quote path.
Aggregating charts on read and scanning alerts on every tick both do work proportional to the firehose when it should be proportional to results.
Evolved approach: pre-rolled candle pipeline + per-symbol crossing index
Charts: a streaming candle builder writes durable OHLCV; rollups for coarser resolutions. A dedicated consumer of the Tick Bus maintains the in-progress 1m candle per symbol (open, high, low, close, volume), and at minute close writes it to a time-series store keyed by (symbol_id, resolution, bucket_ts). Coarser resolutions (5m, 1h, 1d) are cheap rollups of the 1m series computed incrementally. The Chart API then serves a range read of pre-built candles - a single indexed scan, fast - and stitches the live in-progress candle from the conflator’s current high/low/volume so the chart updates as ticks arrive.
Tick Bus --> Candle Builder --> time-series DB
per symbol: update in-progress 1m (o,h,l,c,v)
at minute boundary: flush 1m candle; roll into 5m/1h/1d aggregates
Chart read: GET candles(symbol, 1d, from, to)
-> range scan pre-built 1d candles (fast, indexed)
-> + live tail (current in-progress candle from conflator)
Alerts: per-symbol sorted threshold sets, compare against last price, fire on crossing. The Alert Matcher consumes the same conflated per-symbol stream. For each symbol it keeps two sorted structures keyed by target price: above (waiting to rise to target) and below (waiting to fall to target). It also keeps last_price. On a price move from p_old to p_new it does a range query for the crossed band, not a scan:
on conflated update, symbol S, p_old -> p_new:
if p_new > p_old:
crossed = above[S].range((p_old, p_new]) # only alerts just crossed
else:
crossed = below[S].range([p_new, p_old))
for a in crossed: fire(a) # O(log n + k), k = fires
last_price[S] = p_new
This fires exactly once per crossing (it compares against last price, so it does not re-fire while the price sits past the target) and touches only the alerts in the crossed band, O(fires) not O(alerts). Using the conflation window’s (min, max) instead of just last guarantees a spike-and-retrace inside a window still fires. Fire events go to a dedup/trigger stage (idempotency key per crossing) and then the notification system. This is the same crossing-index technique as a dedicated price-alert system; here it is a consumer bolted onto the shared stream.
Why this is right: charts do their heavy aggregation once at write time into a durable, indexed candle store so reads are cheap range scans plus a live tail; alerts index thresholds so a price move is a range query, not a scan. Both reuse the single normalized tick stream rather than re-deriving prices, and each has the durability model it needs (candles durable, quotes ephemeral).
API Design & Data Schema
API
The live stream is a persistent WebSocket; control is JSON messages, data is compact binary or JSON deltas.
WS wss://stream.example.com/v1/prices (connect to nearest PoP)
--> subscribe:
{ "action":"subscribe", "symbols":["AAPL","TSLA","NVDA"] }
<-- snapshot then deltas:
{ "type":"snapshot", "symbol":"AAPL", "last":249.80, "bid":249.79,
"ask":249.81, "vol":40233110, "ts":1721539200140, "seq":880123 }
{ "type":"delta", "symbol":"AAPL", "last":250.02, "ts":1721539200240, "seq":880451 }
--> unsubscribe:
{ "action":"unsubscribe", "symbols":["TSLA"] }
--> heartbeat / resync (client saw a seq gap):
{ "action":"resync", "symbols":["AAPL"] } -> server resends snapshot
Charts and metadata are normal request/response (served from the edge/CDN with the live tail merged client-side):
GET /v1/candles?symbol=AAPL&resolution=1d&from=2026-01-01&to=2026-07-21
200: { "symbol":"AAPL", "resolution":"1d",
"candles":[ {"t":..,"o":..,"h":..,"l":..,"c":..,"v":..}, ... ],
"live": {"t":..,"o":..,"h":..,"l":..,"c":..,"v":..} // in-progress
}
GET /v1/symbols/search?q=app -> symbol lookup / autocomplete
GET /v1/quote?symbols=AAPL,TSLA -> one-shot REST snapshot (no stream)
POST /v1/alerts { "symbol":"AAPL","condition":"above","target":250.00 }
201: { "alert_id":"a_1f...", "status":"armed" }
GET/DELETE /v1/alerts[/{id}] -> manage alerts
Errors:
400 unknown symbol / bad resolution
401 unauthenticated
429 too many subscriptions for this connection (per-conn cap, e.g. 500)
Data stores
1. Price State - in-memory, per symbol-shard (the narrow waist). Not a database. The entire world’s current prices are ~12 MB; this is an in-memory map symbol_id -> latest quote in the core conflator and mirrored as a last-value cache at each PoP. Rebuildable instantly from the tick stream (the next tick refreshes any symbol). No durability needed - a missed value is superseded within ms.
PriceState (in-memory):
symbol_id -> { last, bid, ask, vol, ts, seq } // ~40 bytes each
2. Subscription tables - in-memory, per edge PoP. Not a database. The hot fan-out index. Ephemeral - rebuilt by clients re-subscribing on reconnect.
per PoP (in-memory):
conn_subs: conn_id -> set<symbol_id>
symbol_subs: symbol_id -> set<conn_id>
upstream_ref: symbol_id -> refcount // when to sub/unsub upstream
3. Candle store - time-series DB (durable). Wide-column / TSDB, not relational. Chart history. Append-heavy writes (one 1m candle per symbol per minute), range-scan reads by (symbol, resolution, time). A time-series or wide-column store (Cassandra/Bigtable/ClickHouse-style) fits the access pattern far better than a relational row store, and there are no cross-row transactions to want. Partitioned by symbol_id, clustered by (resolution, bucket_ts) so a chart read is one contiguous scan.
Table: candles (wide-column / TSDB)
symbol_id INT -- partition key
resolution SMALLINT -- 1m | 5m | 1h | 1d } clustering
bucket_ts BIGINT -- start of the bucket } key
open high low close DECIMAL(18,4)
volume BIGINT
PARTITION BY symbol_id, CLUSTER BY (resolution, bucket_ts)
TTL: 1m kept ~2y, coarser kept longer; raw ticks archived to object storage
4. Symbol metadata - relational / KV (small, read-heavy, cached everywhere). The symbol_id mapping, exchange, currency, tick size, trading hours, name for search. Tiny (~300K rows), strongly consistent, cached in every node. Relational for the rich search/filter queries; replicated read-only to the edge.
Table: symbols
symbol_id INT PRIMARY KEY
ticker VARCHAR -- "AAPL"
exchange VARCHAR -- "NASDAQ"
currency CHAR(3)
tick_size DECIMAL
name VARCHAR
trading_hours ...
INDEX (ticker, exchange), FULLTEXT (name) -- search/autocomplete
5. Alert store - relational (sharded by user_id). Source of truth for alerts. Structured, user-edited, needs read-your-writes consistency; write volume is modest. The matcher’s crossing index is an in-memory materialized view of alerts WHERE status=armed, organized by symbol_id, fed by CDC. (Full treatment of the crossing index and exactly-once firing is its own design.)
6. Tick tape / archive - object storage (durable, cold). Every normalized tick appended for replay, compliance, and rebuilding candles if the pipeline changes. Partitioned by date/exchange/symbol, compressed, never on the read-latency path.
Why this split: the live path deliberately uses no durable store - price state and subscription tables are in-memory, rebuildable views, because touching a DB per tick or per subscriber would blow the 100ms budget. Durability lives only where it is needed and off the hot path: candles in a TSDB tuned for range scans, alerts in a consistent relational store, the raw tape in cold object storage. Symbol metadata is small and cached everywhere. Each store matches its access pattern instead of forcing one engine to do all of it.
Bottlenecks & Scaling
Where it breaks first, in order, and the fix for each.
1. Read fan-out amplification (breaks first). One hot-symbol tick to 1M subscribers at 50K ticks/sec is 5 * 10^10 messages/sec if done naively.
Fix: conflate to a bounded per-symbol rate (latest-value, ~10/sec) so a hot symbol is 10 * subscribers, and fan out through a reference-counted tree (core -> region -> PoP -> connection) so the multiplication happens only at the leaves, spread across hundreds of edge servers each doing O(local subscribers). Delivery cost becomes proportional to (subscribers * conflated-rate), not (ticks * users).
2. Connection concentration and per-server limits. 10M WebSockets cannot terminate in one place. Fix: edge PoPs near users, ~100K connections each, hundreds of them across regions. Connections and their subscription tables are in-memory and rebuilt on reconnect, so PoPs scale horizontally and fail independently.
3. Backbone bandwidth (shipping every symbol everywhere). Sending all 300K symbols to every region wastes long-haul capacity. Fix: per-symbol subscription routing - a region receives a symbol’s stream only when it has a subscriber, reference-counted at each tree level. The backbone carries the union of subscribed symbols per region (tens of thousands), not all of them.
4. Hot symbols / hot partitions on ingest and core. A few thousand symbols carry most of the 5M ticks/sec; one symbol’s partition can be a hot CPU.
Fix: partition the Tick Bus by symbol_id so ticks spread across consumers and preserve per-symbol order; shard the core conflator by symbol so hot symbols land on different conflator nodes. Because conflation caps each symbol’s output rate, a hot symbol’s downstream cost is bounded regardless of its input tick rate - the hotness is absorbed at the conflator.
5. Slow clients and unbounded buffers. A client on a weak network backs up and either OOMs the server or shows seconds-stale prices. Fix: per-connection latest-value conflation with back-pressure - coalesce, never queue; always send the freshest value; drop and force-reconnect a truly stuck connection. Memory per connection is bounded to one slot per subscribed symbol.
6. Chart reads vs raw aggregation. Building charts from raw ticks at read time is far too slow.
Fix: pre-roll candles at write time into a TSDB partitioned by symbol_id and clustered by (resolution, bucket_ts); reads are contiguous range scans plus a live in-progress candle. Coarser resolutions are incremental rollups.
7. Alert matching vs scan. Scanning alerts per tick is the firehose-vs-scan trap again.
Fix: per-symbol sorted threshold sets, range-query the crossed band, compare against last price so it fires once per crossing - O(fires) not O(alerts). Fires go to a durable, deduplicated trigger stage.
8. Correctness: stale-after-fresh and sequence gaps. Redundant feeds and reconnects can deliver an old price after a new one. Fix: monotonic per-symbol sequence numbers stamped at the Normalizer, de-dupe of A/B feeds at ingest, and client-side gap detection that triggers a resnapshot rather than trusting a possibly-missed delta. The client never renders a lower seq after a higher one.
9. Single points of failure. Every stateful piece is replicated: Tick Bus (Kafka replication), core conflator (per-symbol-shard with standby that rebuilds from the stream), TSDB and alert store (replicated), object archive (durable by construction). PoPs and relays are stateless-over-rebuildable-tables with self-healing client reconnect, so no single node’s loss darkens prices for long.
Shard keys, stated plainly: Tick Bus -> partition by symbol_id (spreads ticks, preserves per-symbol order, co-locates a symbol’s conflation). Core conflator -> shard by symbol_id. Candle store -> partition by symbol_id, cluster by (resolution, bucket_ts). Alert store -> user_id for CRUD, symbol_id-organized view for the matcher. Fan-out routing -> by symbol_id subscription reference counts at each tree level. Never key the live path by user or by time - you would scatter a symbol’s price across shards and break the co-location and per-symbol ordering that the whole design relies on.
Wrap-Up
The trade-offs that define this design:
- Collapse the firehose once, fan out through a subscription tree. The single most important move: the core turns 5M ticks/sec into a per-symbol current-price stream (bounded by ~300K symbols), and delivery is a reference-counted tree (core -> region -> PoP -> connection) so the fan-out multiplication happens only at edge leaves, near users, in parallel. Cost is proportional to subscribers, not to ticks * users.
- Conflate for freshness, not completeness. We deliberately drop intermediate ticks and send the latest value per symbol at a bounded rate. We trade the full tape (which no client wants or can render) for bounded memory, bounded bandwidth, and freshest-price delivery even to slow clients.
- Win latency with locality. Edge PoPs terminate connections near users; the only long hop is one optimized private-backbone leg. We trade the simplicity of one central cluster for the sub-100ms-worldwide budget and independent-region availability.
- Route symbols by demand. A region gets a symbol’s stream only if someone there watches it, reference-counted, so the backbone carries the union of wanted symbols, not all 300K.
- Different durability where it belongs. The live path uses no durable store (in-memory, rebuildable state and subscription tables); durability lives off the hot path in a candle TSDB for charts, a consistent store for alerts, and cold object storage for the tape. Charts pre-roll at write time; alerts index thresholds and range-query the crossed band.
One-line summary: normalized exchange feeds are sequenced and partitioned by symbol, collapsed once into a ~12 MB per-symbol current-price state by a conflator, and pushed out along a reference-counted subscription tree to edge PoPs that hold millions of WebSockets near their users - so a single tick reaches a million screens in under 100ms via latest-value conflation and local edge fan-out, while a separate candle pipeline and a per-symbol crossing index serve charts and alerts off the same stream.
Comments