Everyone starts this problem by drawing a box labeled “score database” and an arrow to a phone. The interviewer nods and then tells you it is the second innings of an IPL final, the match is going to the last over, and 100 million people have the app open at the same moment refreshing to see whether the next ball is a six or a wicket. Now the box-and-arrow picture is useless. A single ball event has to reach 100 million screens in under a second, and it has to reach them in the right order, and if the third umpire reverses a wicket thirty seconds later every one of those 100 million screens has to correct itself, and when a fan opens the app in the 15th over they need the full scorecard of everything that already happened. This is not a database read problem. It is a real-time fan-out problem with a durable, correctable event log underneath it - and the “live score” a fan sees is just the latest projection of that log pushed to their screen.
The whole design comes down to four decisions: how you ingest ball events from the scorer as an ordered, correctable, durable sequence (the write path and source of truth); how you fan one ball event out to 100 million concurrent connections in under a second without melting (the fan-out plane, which is the hard part); how you let a fan who just opened the app catch up to “now” and then ride the live stream without a gap (snapshot plus stream); and how you handle corrections and reconnections so no one is ever stuck on a wrong or stale score. Let me build it properly.
Functional Requirements (FR)
In scope:
- Live score delivery. A fan watching a match sees the current score (runs, wickets, overs, this-ball commentary, run rate, required rate) update within a second of each ball being scored, without manually refreshing.
- Ball-by-ball ingestion. A trusted scorer (or an automated feed from the official data provider) submits each ball as a discrete event: runs, extras, wicket, bowler, batsman, over/ball number. The system accepts these in order and makes them the source of truth.
- Corrections and reversals. The scorer can revise a ball (a wide re-marked as a legal delivery, a wicket overturned on review). The correction must propagate to every connected fan and the stored history must reflect the corrected version.
- Catch-up / scorecard on open. A fan opening the app mid-match gets the full current state - full scorecard, fall of wickets, both innings so far, this over - immediately, then transitions seamlessly to the live stream.
- Multiple matches concurrently. Several matches can be live at once; a fan subscribes to the match(es) they care about.
- Durable, replayable history. After the match, the complete ball-by-ball record is permanently queryable - scorecards, commentary, every delivery - and can be replayed.
- Secondary update types. Not just score: commentary text, player-of-the-match, toss, line-ups, match status (innings break, rain delay, result).
Explicitly out of scope (say this so you own the scope):
- Video / live streaming. Delivering the actual video of the match is a completely separate CDN/encoding system with a different cost structure. I deliver the data - scores, text, events - not frames.
- The scoring UI and the human scorer. How the scorer’s tablet app works, and the officiating/umpiring, are upstream of me. I define the event contract and trust the authenticated feed; I do not adjudicate cricket.
- Fantasy, betting odds, and predictions. Downstream consumers of my event stream. I publish a clean event log they can subscribe to; I do not compute odds.
- Login/accounts and personalization. Assume an auth and profile system exists. Subscriptions (“I follow India”) are in scope only as routing keys, not as a full social graph.
The one idea that drives everything: the score is a derived view, not a stored number. You never “UPDATE score SET runs = runs + 6”. You append an immutable ball event to an ordered log, and the current score is a fold over that log. Corrections are new events that supersede old ones. That single decision is what makes fan-out, catch-up, replay, and correction all tractable at once.
Non-Functional Requirements (NFR)
- Scale: up to 100M concurrent connections on a marquee match, and a long tail of smaller matches. Total registered users in the hundreds of millions. A big match sees a sharp spike at the start and brutal micro-spikes at the end of a close game.
- Latency / freshness: a ball event must reach fans within ~1 second p99 end to end (scorer submit -> 100M screens). Sub-second is the product promise and the thing that is hard.
- Ingest rate: tiny. A cricket match is ~1 ball every 30-40 seconds. Even with corrections, commentary, and several concurrent matches, writes are tens to low hundreds of events/sec globally. The write path is trivially small; the fan-out is enormous. This asymmetry is the whole shape of the system.
- Availability: 99.99%+ on the read/fan-out path during a live match. A fan losing the stream for a few seconds and auto-reconnecting is acceptable; the app showing a visibly wrong score is not.
- Consistency: eventual, but ordered and convergent. Fans can be a few hundred ms apart from each other, but every fan must converge to the same correct score, and no fan may see ball N+1 before ball N. After a correction, everyone converges to the corrected value within a second.
- Durability: the ball-by-ball log is the official record and must be durable and exactly-once - never lose a ball, never double-count one. The ephemeral fan-out layer can drop and rebuild; the log cannot.
- Correctness under reconnection: a fan whose connection drops (subway, wifi flake) and reconnects must resync to the exact current state with no missed and no duplicated balls.
Back-of-the-Envelope Estimation (BoE)
Real numbers, because the asymmetry here is the entire point.
Write path (ingest):
1 ball every ~35s per match. A live slate might be ~10 matches.
Balls/sec globally = 10 / 35 ≈ 0.3 balls/sec.
Add commentary, corrections, stats updates: call it ~50 events/sec peak, globally.
This is nothing. A single modest node ingests the entire planet's live cricket.
Read / fan-out path (the real problem):
Peak concurrency: 100M connections on one match.
Each ball must be delivered to all 100M. Ball rate ~0.3/sec, but the delivery
is the cost: 100M messages per ball.
At ~1 ball/30s that is 100M / 30 ≈ 3.3M message-sends/sec sustained for ONE match,
and in the last over, balls (and corrections, and commentary) come faster and the
app is being opened/closed constantly -> spikes well above that.
Connection footprint:
A single server with tuned epoll + lightweight connections holds ~100k-500k
idle WebSocket/SSE connections (memory-bound: ~10-50 KB per idle conn).
Take 200k per edge node.
100M / 200k = 500 edge nodes just to HOLD the connections for one match.
In practice we push far more fan-out to the CDN edge (see deep dive) so origin
holds far fewer; this 500 is the "if we held them all ourselves" ceiling.
Message / bandwidth:
A score-update message is small: score delta + this-ball info ≈ 300-500 bytes.
100M * 400 bytes = 40 GB pushed per ball event, per match.
At one ball / 30s: ~1.3 GB/sec egress sustained for a single match,
spiking to tens of GB/sec in a burst (correction + commentary + new ball together).
This MUST be served from edge/CDN, not from origin links.
Catch-up / snapshot:
Match open surge: at the start of the final, say 100M fans open within ~10 min.
100M / 600s ≈ 170k snapshot fetches/sec, each returning the full scorecard
(a few KB). ~170k * 5 KB ≈ 0.85 GB/sec of snapshot reads, all cacheable.
This is a cache/CDN problem, not a DB problem - the scorecard is identical for
every fan and changes only once per ball.
Storage (history):
One match ≈ 2 innings * 120 balls = 240 balls, * (say) 20 events each counting
commentary/stats ≈ ~5k events, each ~500 bytes ≈ 2.5 MB of raw events per match.
Even 100k matches/year * 2.5 MB = 250 GB/year. Trivial. History is cheap;
keep it all hot-ish, archive cold after a season.
The takeaways, and they are lopsided on purpose: the write side is a rounding error (tens of events/sec, ~250 GB/year of history), and the entire engineering problem is fan-out - delivering a 400-byte message to 100M connections within a second, repeatedly, through micro-spikes, while letting newcomers catch up and reconnectors resync. Everything below is built around that asymmetry: a tiny, bulletproof source-of-truth log, and a massive, layered, mostly-stateless fan-out pyramid.
High-Level Design (HLD)
Three planes. A tiny ingest plane that accepts ball events, orders them, and writes them durably (the source of truth). A fan-out plane - the pyramid - that takes each committed event and pushes it down through regional relays and the CDN edge to 100M connections. A snapshot plane that serves the current full state to anyone opening or reconnecting, so they can join the stream at the right point.
┌──────────────────────────────────────────┐
Scorer / official feed ─▶│ Ingest Service (authenticated) │
(ball events, ~50/s) │ - validates, assigns per-match seq no │
│ - idempotency key dedup │
│ - appends to Event Log (source truth) │
└───────────────┬──────────────────────────┘
│ append (seq N)
┌─────────────▼──────────────┐
│ Event Log (Kafka, per- │ ── consumed by
│ match partition, ordered, │ replay / fantasy /
│ durable, replicated) │ analytics (downstream)
└─────────────┬──────────────┘
│ committed event
┌──────────────────────▼──────────────────────┐
│ Projection / State Builder │
│ - folds events -> current match state │
│ - writes Snapshot (score + scorecard) │
│ - publishes delta to Fan-out │
└──────┬───────────────────────────┬──────────┘
│ snapshot write │ delta publish
┌─────────────▼────────────┐ ┌─────────▼──────────────────┐
│ Snapshot Store │ │ Fan-out Dispatcher │
│ (Redis + CDN-cached KV) │ │ (per-match pub/sub topic) │
│ full state per match, │ └─────────┬──────────────────┘
│ versioned by seq no │ │ one message per ball
└─────────────▲────────────┘ ▼ (to regions)
│ GET snapshot ┌───────────────────────────────┐
│ (on open / │ Regional Relay Fleet │
│ reconnect) │ (fan-out multiplier, 1->many) │
│ └───────────────┬───────────────┘
│ │ push
│ ┌───────────────▼───────────────┐
│ │ Edge Push Nodes + CDN │
│ │ (hold WS/SSE conns OR serve │
│ │ CDN "live object" at edge) │
│ └───────────────┬───────────────┘
│ │ sub-second push
└──────────────── Fans (100M) ──┘
open app -> GET snapshot(seq N)
then subscribe -> receive deltas N+1, N+2...
Request flow, step by step:
- Ingest. The scorer submits “over 19, ball 4: 6 runs, bowler X, batsman Y” with an idempotency key. The Ingest Service authenticates it, validates it against match state (is this the expected next ball?), assigns a monotonically increasing per-match sequence number
seq, and appends it to the Event Log. Only after the log durably commits is the ball “real.” - Project. The Projection service consumes the committed event in order, folds it into the current match state (new score, updated scorecard, run rate), and produces two outputs: it writes the new snapshot (versioned at
seq N) to the Snapshot Store, and it publishes a compact delta message (“at seq N: +6 runs, this-ball info”) to the match’s fan-out topic. - Fan-out. The Dispatcher publishes the delta once; the Regional Relay fleet multiplies it - one message in, thousands out per relay - and hands it to the Edge layer, which either pushes it over the fans’ held WebSocket/SSE connections or updates a tiny CDN-cached “live object” that edge-connected clients are long-polling/streaming. 100M screens update.
- Open / catch up. A fan opening the app first does
GET snapshot(served from CDN/Redis, the same bytes for everyone), rendering the full current scorecard atseq N. The snapshot response includesN. The client then subscribes to the live stream and asks for deltas afterseq N, so there is no gap and no duplicate. - Reconnect. A dropped client reconnects, says “I last saw
seq M,” gets any missed deltasM+1..N(or a fresh snapshot if it fell too far behind), and resumes.
The genius of this shape is that the expensive part (fan-out, snapshot serving) is stateless and cacheable and can be scaled horizontally and pushed to the CDN edge, while the part that must be strongly correct (the ordered, durable event log) is tiny and lives in one well-protected place.
Component Deep Dive
1. Fan-out to 100M connections (the hard part)
Naive approach. Every fan opens a WebSocket to our servers. When a ball is scored, the app server loops over all connected clients for that match and writes the message to each socket.
Where it breaks. Two places, immediately. First, connection holding: 100M concurrent sockets at ~20 KB each is ~2 TB of RAM just for connection state, needing ~500 edge nodes at 200k conns each - manageable but only if those nodes do nothing else. Second, and fatal: the fan-out multiplication happens in one place. If a single process (or even a single fleet tier) must itself write 100M messages per ball, that is 3M+ socket-writes/sec from one logical publisher. The publisher’s NIC, the loop, and the tail latency all collapse. The last fan on the list gets the ball many seconds after the first. You cannot do 1-to-100M from one tier.
The fix: a fan-out pyramid (tree), plus push the leaves to the CDN. You never fan out 1-to-many in one hop. You fan out in layers, each layer multiplying by a bounded factor:
Dispatcher (1 publish per ball)
│ 1 -> ~30
┌────────┴─────────┐
Regional Relay Regional Relay ... (~30 relays, 1 per region/cell)
│ 1 -> ~500 │
┌─────┴─────┐
Edge node Edge node ... (~500 edge nodes per region)
│ 1 -> ~200k
held conns / CDN edge objects -> fans
- Dispatcher -> Regional Relays. The Dispatcher publishes each delta once to a per-match pub/sub topic (Kafka or a Redis/NATS fan-out bus). ~30 regional relays subscribe. That is a 1-to-30 fan-out - trivial.
- Regional Relay -> Edge nodes. Each relay forwards to the ~hundreds of edge nodes in its region holding connections for this match. 1-to-500. Also cheap.
- Edge node -> fans. Each edge node writes to the ~200k sockets it actually holds. 1-to-200k, done in parallel by many cores, and the fan-out factor at each tier is bounded so no single machine does more than a few hundred thousand writes.
At every hop the multiplication factor is bounded (tens to a few hundred thousand), so no single node is the bottleneck and tail latency stays sub-second. This is the same tree-fan-out idea used by large chat/presence systems.
But even 500 edge nodes per match is a lot, and the cheapest message is one you do not send from origin. For a marquee match, layer the CDN in front of the push tier. The current match state is identical for every fan and changes only once per ball. So instead of pushing to 100M held sockets from our own fleet, publish each new state as a tiny immutable, versioned object (match/{id}/seq/{N}) and let the CDN edge serve it:
- Clients do a long-poll or HTTP streaming request to the CDN for “the state after
seq N.” The CDN serves it from edge cache. One origin fetch per POP per ball; the POP serves it to millions. This turns 100M origin pushes into ~(number of POPs) origin fetches per ball - a 1000x+ reduction. - WebSocket is reserved for fans who need the lowest latency (or when the product wants true server-push); the long-tail majority ride CDN-fronted streaming. You pick per-client based on load.
The rule: minimize how many messages originate from your own machines by inserting a bounded fan-out tree and a caching CDN layer; the only thing that must scale linearly with 100M is the leaf (edge/CDN POP) layer, which is exactly what CDNs are built for.
2. Ordering, correctness, and the catch-up gap
Naive approach. Push each ball to connected clients as it happens. New clients, on open, just fetch the latest score number and start listening.
Where it breaks. Races. A client opens and reads “score = 150 at over 18.2,” then subscribes to live. But between its read and its subscribe, ball 18.3 was pushed - and the client missed it, so it is now permanently one ball behind and will stay behind forever (every future delta applies to a wrong base). Or the reverse: it subscribes first, receives 18.3, then its snapshot read returns the older 150 and overwrites the newer state - it goes backwards. And network reordering can deliver ball N+1 before ball N, showing a six before the run that preceded it.
The fix: a global per-match sequence number and snapshot-then-stream with an overlap. Every event the Projection emits carries a monotonic seq. The snapshot is versioned: “this scorecard is current as of seq N.” The client protocol is:
GET snapshot-> returns full state and itsseq = N.- Subscribe to the live stream requesting deltas strictly greater than N.
- The edge/relay replays any buffered deltas in
(N, current]it is holding, then streams new ones live. - Client applies deltas strictly in
seqorder; it buffers an out-of-order delta until the gap fills, and if a delta forseq <= lastAppliedarrives it is discarded (idempotent).
Because the snapshot names its own seq and the stream is requested relative to that seq, there is no gap and no double-apply, regardless of interleaving. The edge nodes keep a small ring buffer of recent deltas (last ~N balls) precisely so a just-arrived or just-reconnected client can be handed (lastSeen, now] without hitting origin. If a reconnecting client’s lastSeen is older than the buffer window, the edge tells it to re-snapshot instead - cheap, bounded, and self-healing.
3. Corrections and reversals
Naive approach. When the third umpire overturns a wicket, mutate the stored score and push a new “score = X” message.
Where it breaks. You have mutated the source of truth in place, so you have lost the audit trail and cannot explain how the score got there, and replay is now impossible. Worse, clients that applied the original wicket as a delta ("-1 wicket, new batsman") have no clean way to undo it from a bare “score = X” push - their derived scorecard (fall of wickets, batsman stats) is now wrong even if the top-line number matches.
The fix: corrections are new, immutable events that supersede, plus a resync signal. Because the score is a fold over an append-only log, a correction is just another appended event: a revise event that references the seq it corrects and carries the corrected ball. The Projection re-folds from the corrected point and emits a new snapshot at a new seq, marked as a correction. For delivery:
- For the top-line and simple deltas, the correction delta carries the net adjustment plus a flag.
- For anything structural (a wicket reversal changes batsman stats, partnerships, fall-of-wickets), the correction delta sets a
resyncRequiredflag or aresyncFromSeq. Clients that cannot safely apply the correction as a delta simply re-fetch the (freshly updated, CDN-cached) snapshot at the newseq. Since the snapshot is already produced and cached per ball, this is nearly free.
The event log keeps both the original and the revise event forever, so the history is honest and replay reproduces exactly what happened, including the reversal. This is why source-of-truth-as-an-append-log is not optional here: it is what makes corrections both correct and auditable.
4. The ingest path and exactly-once truth
Naive approach. Scorer POSTs a ball, the API increments the score in a database row.
Where it breaks. The scorer’s tablet is on a flaky stadium network. It POSTs “ball 18.3: 6 runs,” times out, retries - and now the ball is counted twice. Or two ingest instances both accept a submission. Concurrency and retries corrupt the one thing that must be perfect: the official record.
The fix: idempotent, sequenced, single-writer-per-match append. Each ball submission carries a client-generated idempotency key (e.g. matchId:inning:over:ball:attempt) and the expected prior seq. Ingest:
- Routes all events for a given match to a single logical writer (partition the Event Log by
matchId, so one Kafka partition = one match = a total order). This gives per-match serialization for free - no distributed lock. - Dedups on the idempotency key: if this key was already appended, return the existing
seq(the retry is a no-op). This makes submission exactly-once from the client’s perspective. - Optionally uses an expected-seq / optimistic check: the submission says “I believe the last ball was
seq N”; if the log head is already past that, reject or reconcile, catching the scorer’s double-submits and ordering mistakes.
The Event Log is replicated (e.g. Kafka with acks=all, replication factor 3) so a committed ball survives node loss. Downstream (projection, fan-out, replay, fantasy, analytics) are all just consumers of this one ordered log - which is why the design stays clean: there is exactly one source of truth, and everything else is a projection or a subscriber.
API Design & Data Schema
Ingest API (scorer / official feed, authenticated, private):
POST /v1/matches/{matchId}/events
Auth: scorer service token (mTLS + signed)
Body: {
"idempotencyKey": "m123:1:18:3:a1",
"expectedPrevSeq": 441,
"type": "ball", // ball | revise | commentary | status | lineup
"ball": {
"inning": 1, "over": 18, "ballInOver": 3,
"runs": 6, "extras": 0, "extraType": null,
"wicket": null, // or { "kind": "caught", "batsman": "...", ... }
"bowler": "p789", "striker": "p456", "nonStriker": "p457"
}
}
200 -> { "seq": 442, "accepted": true }
409 -> { "error": "seq_conflict", "currentSeq": 443 } // retry/reconcile
POST /v1/matches/{matchId}/events type:"revise"
Body: { "idempotencyKey": "...", "revisesSeq": 440, "ball": { ...corrected... } }
200 -> { "seq": 443, "accepted": true, "resyncRequired": true }
Read / snapshot API (fan-facing, public, CDN-cached):
GET /v1/matches/{matchId}/snapshot
-> 200 (Cache-Control: public, max-age=1, stale-while-revalidate)
{
"seq": 442,
"status": "live",
"score": { "inning": 1, "runs": 151, "wickets": 4, "overs": "18.3",
"runRate": 8.15, "requiredRate": 9.2 },
"scorecard": { ...batsmen, bowlers, fall-of-wickets... },
"lastBall": { "over": 18, "ballInOver": 3, "runs": 6, "text": "SIX!" }
}
Live stream API (fan-facing):
GET /v1/matches/{matchId}/stream?afterSeq=442
WebSocket OR SSE (server picks based on load/capability).
Client sends: { "subscribe": "m123", "afterSeq": 442 }
Server streams deltas:
{ "seq": 443, "type": "ball", "delta": { "runs": +1, ... }, "lastBall": {...} }
{ "seq": 444, "type": "revise", "resyncFromSeq": 440 } // -> client re-snapshots
{ "seq": 445, "type": "commentary", "text": "..." }
Heartbeat ping every ~15s; client reconnects with its lastSeenSeq.
History API (post-match, public, cacheable forever):
GET /v1/matches/{matchId}/history?fromSeq=0 -> full ordered event list (replay)
GET /v1/matches/{matchId}/scorecard -> final immutable scorecard
Data stores and schema.
The system is event-sourced, so the primary store is an append-only log, not a mutable table. Choice: log (Kafka) for the source of truth + a key-value/document store for materialized state + object storage for cold history. SQL is used only for the small relational metadata (matches, teams, players), where joins and integrity matter and the volume is tiny.
Event Log (Kafka topic match-events, partitioned by matchId so each match is a totally ordered partition):
key: matchId
value: {
seq int64 // per-match monotonic, assigned at append
idempotencyKey string // dedup
type enum // ball | revise | commentary | status | lineup
payload json/avro
scorerId string
ts int64
}
Retention: compacted/retained through match + copied to object store (cold archive).
match_state (Redis + document store, materialized view, key = matchId):
{ matchId, seq, status, inning, score{...}, scorecard{...}, lastBall{...}, updatedTs }
Index: primary key matchId. Versioned by seq. CDN-cacheable snapshot derived from this.
matches (Postgres, relational metadata - SQL because it is small, relational, and read-often):
match_id UUID PK
series_id UUID FK (indexed)
team_a, team_b UUID FK -> teams
venue, start_ts, status, format // T20 | ODI | Test
INDEX (status, start_ts) -- "what is live / upcoming"
players, teams (Postgres): standard reference tables, FK’d from events by id.
match_history (object storage, e.g. S3, one compressed object per match): the full ordered event list, written once the match ends, for replay and analytics. Cheap, immutable, infinitely retained; hot matches also stay in the document store for fast scorecard reads.
Why this split: the log gives ordering, durability, exactly-once, and replay; the KV/document materialized view gives O(1) snapshot reads that the CDN can cache; SQL handles the small relational metadata where integrity and joins earn their keep; object storage makes keeping every ball of every match forever essentially free.
Bottlenecks & Scaling
Where it breaks first: the fan-out leaf, not the database. This system has an almost nonexistent write load and a monstrous read/push load, so classic DB sharding is irrelevant and the pressure is entirely on getting one message to 100M screens. The failure modes and fixes:
| Pressure point | Why it breaks | Fix |
|---|---|---|
| 1-to-100M fan-out | One publisher cannot write 100M sockets/sec; tail latency explodes | Bounded fan-out tree (Dispatcher -> regional relays -> edge nodes), each hop multiplies by a small factor |
| Origin egress (40 GB/ball) | Links saturate; cost explodes | Push the leaf to the CDN: serve versioned “live state” objects from edge POPs, one origin fetch per POP per ball |
| Snapshot-on-open surge (170k/sec) | DB cannot serve 170k scorecard reads/sec | Snapshot is identical for all fans -> CDN + Redis cache, max-age=1s; origin recomputes once per ball |
| Connection memory (2 TB RAM) | Holding 100M sockets is expensive | Stateless edge nodes, horizontal scale; shed most clients to CDN streaming/long-poll, reserve WS for low-latency tier |
| Hot key: the one marquee match | All load on a single match topic/partition | The event log partition is fine (writes are tiny); for reads, replicate the match’s materialized state to every region and every edge, and fan out through the tree - reads never touch the single partition |
| Thundering herd on reconnect | A regional network blip drops millions at once; all reconnect together | Edge delta ring buffers serve (lastSeen, now] without origin; jittered backoff on reconnect; fall back to re-snapshot (cached) if too far behind |
| Correction storms | A reversal forces millions to resync at once | Resync target is the already-cached snapshot at the new seq - the herd hits the CDN, not origin |
| Single-writer per match | If the ingest partition leader dies mid-over | Kafka partition replication + fast leader election; idempotency keys make the scorer’s retry safe across the failover |
Sharding / partitioning. By matchId everywhere that is stateful: the event log partitions by match (natural total order per match, and matches are independent), the materialized state keys by match, the fan-out topics are per match. There is no cross-match transaction, so matches scale out linearly - a 200-match day is just more independent partitions. The only “hot shard” is a single huge match, and we deliberately do not solve that by splitting the match (it must stay totally ordered); we solve it by making reads replicated and tree-fanned-out and CDN-served, which scales reads without touching the single ordered write partition.
Replication. Event log RF=3, acks=all. Materialized match state replicated to every region (it is tiny - a few KB per match) so edge nodes read locally and no fan ever makes a cross-region snapshot call. Postgres metadata uses a primary with read replicas; it is barely loaded.
Async processing. Everything after the committed log is asynchronous and decoupled: projection, fan-out, fantasy, analytics, and archival are all independent consumers. If the analytics consumer lags, fans are unaffected. If a fan-out relay dies, its clients reconnect to a sibling and resync from lastSeen - no data is lost because the log, not the relay, is the truth.
Single points of failure. The ingest/log is the one component that must never lose data, so it is replicated and the writer fails over with idempotency protecting the scorer’s retries. Every read-path component (edge nodes, relays, CDN, snapshot cache) is stateless and replaceable; losing one costs some fans a few seconds of reconnect, never correctness. Graceful degradation: if the live push tier is overwhelmed, clients fall back to polling the CDN snapshot every 1-2s - slightly less “live,” but the score is never wrong and the system never hard-fails during the last over.
Wrap-Up
The trade-offs that define this design:
- Event sourcing over mutable state. We pay with more moving parts (a log, a projection, materialized views) and buy ordering, exactly-once truth, clean corrections, and perfect replay. For an official score record that must handle reversals and reconnects, this is non-negotiable.
- Eventual (but ordered and convergent) consistency over strict global consistency. Fans can be a few hundred ms apart; they must never see balls out of order and must always converge to the same correct value. We enforce this with a per-match sequence number and snapshot-then-stream, not with a global lock.
- CDN and a fan-out tree over brute-force server push. We refuse to originate 100M messages per ball. A bounded fan-out pyramid plus CDN-served versioned state objects turns an impossible 1-to-100M problem into a tree of cheap, bounded hops and edge cache hits.
- Tiny write path, massive read path, and we optimized the right one. The whole architecture leans into the asymmetry: a single well-protected ordered log on the write side, and a stateless, replicated, CDN-fronted pyramid on the read side.
One-line summary: an append-only, per-match-ordered event log is the single source of truth, a projection turns it into a CDN-cacheable snapshot plus a sub-second delta stream, and a bounded fan-out tree backed by the CDN delivers each ball to 100M fans in order, with sequence numbers and cached snapshots making catch-up, reconnection, and score corrections all converge correctly.
Comments