A video platform looks trivial for about five seconds: POST /video to upload, GET /video to play. Then you remember that one uploaded file is a 4GB 4K master that has to become a dozen different renditions, that a viewer on a train switching between 5G and a tunnel needs the stream to drop from 1080p to 240p without stalling, that the actual bytes are served to billions of people from machines physically close to them and never from your origin, and that the view counter under the video is being incremented tens of thousands of times per second on the popular ones. The upload button is the easy 1%. The real system is this: turn one giant immutable master file into many small streamable segments, store those segments once and serve them from the edge, let the player pick the right quality moment to moment, and count views and watch time at a scale where a naive UPDATE ... SET views = views + 1 would fall over on day one.
Everything hard here - the transcoding pipeline, adaptive bitrate, the CDN tiering, view-count aggregation - is machinery in service of two numbers: time-to-first-frame (how fast playback starts) and rebuffer ratio (how often it stalls). You do not “serve a video”; you serve a manifest that points at thousands of tiny segment files, and the player stitches them into a continuous stream while constantly re-deciding which quality to fetch next. Let me build it properly.
Functional Requirements (FR)
In scope:
- Upload video. A creator uploads a source file (any common container/codec, up to several GB). It is durably stored and accepted even if playable renditions are not ready yet.
- Transcode into multiple renditions. Each upload is converted into a ladder of resolutions and bitrates (240p through 4K) and packaged into segmented streaming formats (HLS/DASH).
- Adaptive bitrate playback. A viewer streams the video; the player automatically switches quality based on available bandwidth and buffer health, and the user can also pin a quality manually.
- Global low-latency delivery. Playback starts fast and rarely stalls anywhere in the world, served from edge locations close to the viewer.
- View count and watch metrics. Each video shows a view count; the system tracks views and watch time for ranking, recommendations, and creator analytics.
- Basic metadata and search-by-id. Title, description, thumbnails, duration, uploader, visibility (public/unlisted/private). Fetch a video’s metadata and playback manifest by id.
- Resumable, fault-tolerant upload. A dropped connection mid-upload resumes rather than restarting the whole multi-GB transfer.
Explicitly out of scope (say it out loud so you own the scope):
- Recommendations / the home feed ranking model. A huge system of its own; I consume its signals (views, watch time) and note where they hook in, but I am not building the ranker.
- Full-text search and discovery. Assume a separate search service indexes metadata; I expose the metadata it needs.
- Comments, likes, subscriptions, monetization/ads. Social and money layers on top of the core video plane; mentioned only where they touch it.
- Live streaming. Live has a different ingest and latency model (sub-second, no full pre-transcode). I design video-on-demand (VOD); I will note in one line where live diverges.
- DRM / content-ID copyright matching. Real and important, but orthogonal to the streaming architecture; I flag the hook point and move on.
The one decision that drives everything: this is a write-once, read-astronomically-many, latency-and-durability-critical system where the expensive one-time work (transcoding) is done offline and asynchronously, and the hot path (playback) is a pure cacheable read of immutable segment files. Unlike a chat system (route each message once) the same bytes are served billions of times; unlike a sync system (bytes-moved is the budget) here the budget is edge cache hit rate and time-to-first-frame. That asymmetry - do the heavy lifting once at ingest, serve immutable chunks from the edge forever - is the whole problem.
Non-Functional Requirements (NFR)
- Scale: ~2B monthly users, assume ~500M daily active viewers. ~500 hours of video uploaded per minute (real YouTube number). Hundreds of billions of watch events per day. Petabytes of new source video per day; exabytes stored.
- Latency: time-to-first-frame under ~1s at p95 from a warm edge; rebuffer ratio under ~0.5% of playback time. Manifest fetch under ~100ms. View count can be seconds-to-minutes stale - nobody cares if it says 1,000,240 vs 1,000,251.
- Availability: 99.99% on the playback (read) path - if people cannot watch, the product is down. Upload and transcoding can tolerate brief degradation (a video being “still processing” for a few extra minutes is acceptable); playback of already-published videos must not.
- Consistency: eventual consistency everywhere on the read side. A newly uploaded video appearing a minute late, a view count lagging, a rendition not ready yet - all fine. The only place I want stronger guarantees is the upload commit (a video record and its stored master must not disagree) and not losing an uploaded master.
- Durability: 11 nines on the uploaded source master and the published renditions - an accepted upload is never lost. Source masters are the crown jewels; if a transcode is buggy you must be able to re-run it from the original.
- The real budget is edge hit rate and TTFF. It is acceptable for a fresh upload to take minutes to fully process; it is never acceptable for playback of a popular video to miss the edge cache and stall. The whole design pushes immutable segments as close to viewers as possible and does the expensive transcoding once, offline.
Back-of-the-Envelope Estimation (BoE)
Real numbers with the arithmetic, because they dictate the architecture.
Upload / ingest volume:
500 hours of video uploaded per minute (YouTube's stated figure).
500 hrs/min / 60 = ~8.3 hours of video uploaded per second.
Say the average source is ~5 GB per hour of footage (mixed quality).
500 hrs/min * 5 GB = 2,500 GB/min of source = ~42 GB/sec of raw ingest.
Per day: 500 * 60 * 24 = 720,000 hours/day * 5 GB = 3.6 PB/day of SOURCE.
Uploads/sec as discrete files: say avg clip ~10 min.
500 hrs/min = 30,000 min of video/min => ~3,000 uploads/min => ~50 uploads/sec.
Peak (bursty ~5x) ~250 uploads/sec.
Transcoding compute (the offline heavy lifting):
Each source becomes a ladder of ~6-8 renditions (240p,360p,480p,720p,1080p,1440p,4K)
plus multiple codecs (H.264 for compatibility, VP9/AV1 for efficiency).
Transcoding is roughly ~1-5x realtime PER rendition on a CPU core (AV1 much worse),
GPU/ASIC encoders bring it far below realtime.
8.3 hrs of video/sec * ~7 renditions = ~58 rendition-hours to encode per second.
At ~0.5x realtime on hardware encoders => ~29 machine-hours of encode per wall second
=> a fleet of tens of thousands of encoding cores/ASICs, always busy.
This is why transcoding is an async, autoscaled, batch fleet - never on the request path.
Storage:
Source: 3.6 PB/day. Renditions typically total ~1.5-2x the source across the ladder
(many small low-res + a few large high-res). Say ~2x => ~7 PB/day of renditions.
Total new bytes ~ 3.6 + 7 = ~10 PB/day landing in object storage.
Per year: ~10 PB * 365 = ~3.6 EB/year of new stored video. Exabyte scale.
Source masters are kept (needed for re-encode) but on COLD storage (cheap, rarely read).
Hot renditions of recent/popular videos live on fast storage + CDN; the long tail
demotes to cheaper tiers over time.
Playback / read side (where the QPS lives):
500M DAU * say 20 videos/day watched = 10B video starts/day.
10B / 86,400 ≈ ~115,000 video starts/sec (steady). Peak ~5x ≈ ~575,000/sec.
But each "start" is NOT one request - a video is thousands of ~4-6s SEGMENTS,
each a separate GET. A 10-min video at 6s segments = 100 segment GETs.
115,000 starts/sec * ~tens of segment fetches in the first minutes
=> MILLIONS of segment GETs/sec globally. ~99%+ of these must hit the CDN edge.
Read:write is absurdly lopsided: ~50 uploads/sec vs millions of segment reads/sec.
=> ~100,000:1. The origin must almost never be touched by playback.
Bandwidth (the actual product cost):
Concurrent viewers: say 50M watching at once, avg ~3 Mbps (mixed quality ladder).
50M * 3 Mbps = 150 Tbps of egress at peak. Served almost entirely from CDN edges.
Origin egress must be a tiny fraction (cache-fill only), or the bandwidth bill
and origin NICs both explode. Edge hit rate is a first-order cost lever, not a nicety.
View counting:
10B video starts/day + heartbeats every ~10-30s during playback for watch-time.
A 10-min view => ~20-60 heartbeat events. 10B starts * ~30 events
= ~300B watch events/day => ~3.5M events/sec steady, ~15M/sec peak.
No transactional database updates a row 15M times/sec. This MUST be an
aggregation pipeline (stream + rollups), never a synchronous counter write.
The takeaways: ingest is a fat but modest-QPS write path feeding a massive offline transcode fleet; storage is exabyte-scale with a hot/cold split; playback is a millions-of-QPS, ~100,000:1 read-heavy firehose that lives or dies on CDN edge hit rate; and view/watch counting is a 15M-events/sec stream that must be aggregated, never synchronously counted. Those four - transcode pipeline, ABR, CDN, and counting - are the interview.
High-Level Design (HLD)
The architecture splits along the fundamental seam of a video platform: the write/ingest plane (upload, transcode, package - expensive, async, done once) is completely separate from the read/playback plane (manifest + segment reads - cheap, cacheable, served billions of times from the edge). A creator uploads a master to blob storage; an async pipeline transcodes it into a rendition ladder, packages each into segmented HLS/DASH with a manifest, and publishes the segments to storage fronted by a CDN. A viewer fetches the manifest, then the player pulls segments adaptively from the nearest edge. View events flow into a separate streaming aggregation pipeline.
UPLOAD / WRITE PLANE (async, expensive, done once)
┌──────────┐ resumable ┌──────────────┐ raw master ┌─────────────────┐
│ Creator │─────────────>│ Upload Svc │──────────────>│ Source Blob │
│ (browser │ chunked PUT │ (validate, │ │ Store (cold, │
│ / app) │ signed URLs │ dedup, emit)│ │ durable master) │
└──────────┘ └──────┬───────┘ └────────┬────────┘
│ "new upload" event │ read master
▼ ▼
┌───────────────┐ split into ┌──────────────────┐
│ Job Queue │──────────────>│ Transcoding │
│ (Kafka/SQS) │ GOP segments │ Fleet (workers, │
└───────────────┘ │ GPU/ASIC encode) │
│ ladder + codecs │
└────────┬─────────┘
│ renditions
┌───────────────────────────▼─────────┐
│ Packaging (HLS/DASH: segments + │
│ manifests) -> publish; write │
│ Metadata DB "ready" + manifest ptr │
└───────────────┬───────────────────────┘
│ segments (immutable)
─────────────────────────────────────────────────────┼───────────────────────
READ / PLAYBACK PLANE (cheap, cacheable, billions/day) ▼
┌──────────┐ 1. GET meta+manifest ┌──────────────┐ ┌──────────────────────┐
│ Viewer │────────────────────────>│ API / Meta │ │ Rendition Blob Store │
│ (ABR │ <- manifest (segment │ Service │ │ (hot; segments + │
│ player) │ URLs + bitrates) │ (read-heavy,│ │ manifests) │
│ │ │ cached) │ └──────────┬───────────┘
│ │ 2. GET segments ........│..............│..............│ cache-fill
│ │<────── from EDGE ────────┤ │ ┌─────▼──────┐
└────┬─────┘ (99%+ hit at edge) └──────────────┘ │ ORIGIN │
│ 3. view/heartbeat events │ shield │
▼ └─────▲──────┘
┌──────────────┐ ┌──────────────────┐ ┌────────────────┐ │
│ Ingest gate │──>│ Event Stream │──>│ Aggregation │ ┌────┴────────┐
│ (dedup, auth)│ │ (Kafka) │ │ (Flink/Spark: │ │ CDN EDGE │
└──────────────┘ └──────────────────┘ │ rollups) │ │ (PoPs near │
└───────┬────────┘ │ the viewer) │
│ counts └──────────────┘
┌───────▼────────┐
│ View Count / │
│ Analytics Store │
└────────────────┘
Upload flow (creator uploads a master):
- The client asks the Upload Service to start a resumable upload; it gets back a set of signed URLs (or a resumable session id) and uploads the master in chunks directly to blob storage, bypassing the app tier entirely. Chunked + resumable means a dropped connection resumes from the last acked chunk, not byte zero.
- Once all chunks land, the Upload Service validates the file (container/codec probe, size, virus/content checks), writes a video record in the Metadata DB with status
uploaded, and emits a “new upload” event onto a durable job queue (Kafka/SQS). - The creator immediately gets a video id and can set title/thumbnail/visibility; the video is “processing” and not yet publicly playable.
Transcode flow (async, offline, the heavy lifting):
- The Transcoding Fleet consumes the job. A master is split into independent GOP-aligned segments (chunks that start at a keyframe), and each segment is transcoded in parallel across many workers into every rung of the rendition ladder (240p…4K) and each codec (H.264, VP9/AV1). Parallelizing by segment turns a “2-hour movie takes 2 hours to encode serially” into “encode 1000 segments across 1000 workers in near-constant time.”
- Each transcoded rendition is packaged into a streaming format: cut into ~4-6s media segments plus a manifest (HLS
.m3u8/ DASH.mpd) listing the ladder and pointing at every segment URL. - Segments and manifests are written to the rendition blob store (hot tier, CDN-fronted). When all target renditions are published, the pipeline flips the video’s Metadata DB status to
readyand stores the manifest pointer. Only now is it publicly playable.
Playback flow (viewer watches, online):
- The client calls the API/Metadata Service for the video: it returns metadata and the manifest URL. The player fetches the manifest (small, cacheable), which lists the available quality levels and the segment URLs.
- The ABR player picks a starting rendition (conservative, so playback starts fast), then fetches segments one after another from the nearest CDN edge. As it measures throughput and watches its buffer, it steps up or down the ladder segment by segment.
- Segments are immutable and content-addressed, so ~99%+ of segment GETs are served from the edge cache; a miss pulls from the origin shield (a mid-tier cache) and only then from the blob store, so the origin is touched rarely.
- In parallel the player fires view and heartbeat events to an ingest gate, which lands them on the event stream; a stream-processing job aggregates them into view counts and watch-time rollups asynchronously.
The key insight: all the cost is paid once, offline, at ingest (transcode + package), producing immutable segments that the read path serves from the edge with almost no compute. The write plane and read plane share nothing but the blob store and a status flag. That separation is what lets playback hit millions of QPS on cheap cached reads while transcoding churns away on its own autoscaled fleet.
Component Deep Dive
The hard parts, naive-first then evolved: (1) the upload + transcoding pipeline - how one giant master becomes a segmented ladder without a single machine choking; (2) adaptive bitrate streaming - how the player picks quality without stalling; (3) CDN strategy - how immutable segments reach viewers from the edge; (4) view-count at scale - how to count billions of events without a synchronous counter.
1. Upload and transcoding pipeline: why naive fails, then segment-parallel
Naive approach: the upload hits the application server, which holds the whole file in memory / on local disk, then calls ffmpeg inline to produce each rendition one after another, and returns when done.
Where it breaks:
- The app tier cannot absorb multi-GB uploads. Buffering a 4GB file through the request tier blows memory, ties up a worker for the entire (slow, residential-uplink) upload, and one dropped connection restarts the whole transfer. The bytes should never transit the app tier at all.
- Synchronous transcoding on the request path is a non-starter. Encoding a 2-hour 4K video into 7 renditions is tens of minutes to hours of CPU. You cannot hold an HTTP request open that long, and you cannot let one upload monopolize a server. Transcoding must be async and off the request path.
- Serial per-file encoding does not scale. Encoding rendition after rendition, video after video, on one box means a long movie takes as long to encode as to watch - times the number of renditions. Throughput collapses under load.
- One failure kills the whole job. If the encoder crashes 90% through a 2-hour file, you restart from zero.
Evolution A: direct-to-blob resumable upload + async job queue. The client uploads directly to blob storage via signed URLs in chunks (e.g. 8MB), tracked by a resumable session so a failure resumes from the last acked chunk. The app tier only issues URLs and, on completion, validates and enqueues a job. Now the bytes never touch the app tier, uploads survive flaky networks, and transcoding is decoupled onto a fleet that consumes from a durable queue - upload success no longer waits on encode.
Where it still breaks: a single worker encoding a whole 2-hour master is still slow (hours of wall-clock per video) and still fragile (a crash near the end wastes it all). Throughput per video is bounded by one machine.
Evolution B: segment-parallel transcoding (map-reduce for video). Split the master into independent chunks at GOP boundaries - each chunk begins with a keyframe (IDR frame) so it can be decoded and encoded with no dependency on any other chunk. Fan those chunks out across the whole worker fleet; each worker encodes its chunk into every rung of the ladder; then a reduce step stitches the encoded segments back in order and packages them.
Master (2 hours) ──split at GOP/keyframe boundaries──> [seg0][seg1][seg2]...[segN]
│ │ │ │
each segment encoded INDEPENDENTLY ▼ ▼ ▼ ▼
into the full ladder, in parallel: W3 W17 W42 ... W991
│ │ │ │
reduce: concatenate per-rendition, package HLS/DASH <───┴─────┴─────┴───────┘
=> a 2-hour video encodes in ~the time of ONE segment * (segments/workers)
Why GOP-aligned splitting is mandatory: video frames are interdependent (P and B frames reference other frames). If you cut mid-GOP, a chunk lacks the keyframe its frames depend on and cannot be decoded independently. Cutting only at IDR keyframes makes each chunk self-contained. The same alignment later defines the streaming segment boundaries, so the split does double duty.
Benefits that fall out:
- Near-constant encode latency regardless of video length - a 2-hour film and a 2-minute clip finish in similar wall-clock if the fleet is big enough, because both are just “N independent segments.”
- Cheap fault tolerance. A worker crash re-queues just that one segment, not the whole video. Idempotent, retryable, at segment granularity.
- The rendition ladder and codecs are just more parallel jobs. 7 resolutions x 3 codecs x N segments is one big embarrassingly-parallel batch.
Pipeline orchestration. A DAG orchestrator drives: validate -> split -> [fan-out: transcode each segment x each ladder rung x each codec] -> package (per rendition) -> publish -> flip status to ready. It tracks completion, retries failed segments, and only marks the video ready when a viable set of renditions is published (you can publish 360p/720p first so the video goes live fast, then backfill 4K/AV1). Source masters go to cold storage afterward - kept for re-encode (new codec, bug fix) but rarely read.
The result: uploads are fast, resumable, and never touch the app tier; transcoding is an autoscaled offline fleet that turns any-length master into a segmented ladder in near-constant time with per-segment fault tolerance; and the output is exactly the immutable segments the read path will serve.
2. Adaptive bitrate streaming (ABR): why one file fails, then the segment ladder
Naive approach: store one MP4 per video at a fixed quality (say 720p) and serve it via a single GET / HTTP progressive download.
Where it breaks:
- One bitrate fits nobody. 720p stalls constantly on a weak mobile connection and looks soft on a fiber-connected TV. You cannot pick a single quality that serves a phone on 3G and a 4K living-room screen.
- No mid-stream adaptation. Network throughput varies wildly during a single view (walking into an elevator, wifi congestion). A single fixed file cannot switch down when bandwidth drops, so it buffers; it cannot switch up when bandwidth recovers, so it looks bad forever.
- Progressive download wastes bytes and seeks badly. You download bytes far ahead of the playhead (wasted if the user abandons) and seeking means byte-range gymnastics on a monolithic file.
Evolution: the rendition ladder + segmented streaming + client-side ABR (HLS/DASH). Encode a ladder of renditions and cut each into short (~4-6s) segments. A manifest describes the ladder and lists every segment. The player fetches segments one at a time and, before each fetch, chooses which rung to pull based on measured throughput and current buffer depth.
Manifest (HLS master .m3u8 / DASH .mpd):
240p @ 0.3 Mbps -> seg0_240.ts, seg1_240.ts, ...
360p @ 0.7 Mbps -> seg0_360.ts, seg1_360.ts, ...
720p @ 2.5 Mbps -> seg0_720.ts, seg1_720.ts, ...
1080p @ 4.5 Mbps -> seg0_1080.ts, seg1_1080.ts, ...
4K @ 16 Mbps -> seg0_4k.ts, seg1_4k.ts, ...
Player loop (per ~4-6s segment):
estimate bandwidth (recent segment download speeds)
check buffer occupancy (seconds of video buffered ahead)
pick rung: high enough to use bandwidth, low enough to keep buffer safe
fetch next segment at that rung -> decode -> play -> repeat
The ABR decision is a control loop balancing two signals:
- Throughput-based: “my last few segments downloaded at 3 Mbps, so I can afford the 2.5 Mbps rung.” Reactive but can oscillate.
- Buffer-based: “I have 20s buffered, safe to try a higher rung; I have 3s buffered, drop down NOW to avoid a stall.” Buffer occupancy is the anti-rebuffer safety signal. Production algorithms (e.g. BOLA-style, or hybrids) blend both, with hysteresis so quality does not flicker every segment.
Why short segments (~4-6s): shorter segments mean the player can react faster to bandwidth changes (it re-decides every segment) and start playback sooner (fetch one small segment, not a huge file). Too short and per-segment HTTP overhead and manifest bloat grow; ~4-6s is the sweet spot. The first segment is often fetched at a low rung to minimize time-to-first-frame, then the player ramps up.
Crucially, ABR is a client-side decision. The server just hosts immutable segment files and a manifest; the player is the brain. That is what makes the read path stateless and infinitely cacheable - every viewer hits the same segment files, and no server holds per-viewer streaming state. The segments produced here are exactly what the transcode pipeline emitted, and exactly what the CDN caches.
Note where live diverges: same segmented ABR model, but segments are produced continuously in near-realtime (low-latency HLS/LL-DASH with ~1-2s segments and chunked transfer), and there is no full pre-transcode - encoding happens on the fly. Same read-path shape, different ingest timing.
3. CDN strategy: why origin-serving fails, then multi-tier edge caching
Naive approach: serve segments straight from the blob store / origin data center to every viewer worldwide.
Where it breaks:
- Latency from distance. A viewer in Mumbai hitting an origin in Virginia eats ~200ms+ round-trips per segment fetch - time-to-first-frame balloons and every segment fetch is slow, so the buffer starves and it rebuffers.
- Origin bandwidth and NICs melt. Recall the BoE: ~150 Tbps of peak egress. No origin fleet serves that; the network cards, the cross-region backbone, and the bill all explode. Serving a viral video’s segments to 10M concurrent viewers from origin is physically impossible.
- Redundant transfer of identical bytes. The same popular segment is shipped from origin millions of times over the same long-haul links - pure waste, since the bytes are immutable and identical for everyone.
Evolution: a multi-tier CDN with immutable, cacheable segments pushed to the edge.
Viewer ──(few ms)──> CDN EDGE PoP (thousands, close to users)
│ miss (~1% of requests)
▼
REGIONAL / ORIGIN-SHIELD cache (fewer, larger)
│ miss (rare)
▼
ORIGIN (rendition blob store) <- touched almost never
The design levers:
- Immutability makes caching trivially safe and permanent. A segment file never changes (a re-encode is a new file/URL), so segments carry long/immutable
Cache-Controland can live at the edge indefinitely with zero invalidation problem. This is the single biggest reason the video read path is so cacheable - unlike a mutable API response, a segment is write-once. - Tiered caching to protect the origin (shielding). Edge misses do not all stampede the origin; they collapse onto a regional/origin-shield tier. Many edges in a region share one shield, so a cold-but-popular video fills the shield once and every edge in that region pulls from it, not from origin. This tames the thundering herd on a newly-viral video.
- Push vs pull placement. Most segments are pull-through (cached lazily on first request). For predictably popular content - a big creator’s premiere, a video the recommender is about to blast to millions - you can pre-warm/push segments to edges ahead of demand so even the first viewers hit a warm cache.
- Hot/cold tiering at the origin. Recent and popular videos’ renditions sit on fast storage; the long tail demotes to cheaper cold storage. Source masters are always cold. This keeps the expensive fast tier small.
- Fill efficiency. Because ABR means most viewers converge on a few mid-ladder rungs (say 480p/720p on mobile), those specific rungs’ segments are the hot set and cache extremely well; the rarely-watched 4K rung of an obscure video may simply never be cached (served from origin on the rare hit - fine).
The payoff ties straight back to the NFR budget: ~99%+ edge hit rate keeps TTFF near the edge RTT (tens of ms), keeps rebuffering rare, and keeps origin egress to a cache-fill trickle instead of the full 150 Tbps. The CDN is not an optimization here; it is the delivery system, and immutable segmentation is what makes it work.
4. View-count at scale: why a counter column fails, then stream aggregation
Naive approach: UPDATE videos SET view_count = view_count + 1 WHERE id = ? on every play.
Where it breaks:
- Write throughput. From the BoE, ~15M watch events/sec at peak. No single row, and no single database, absorbs 15M writes/sec - and a viral video concentrates a huge share of those on one hot row, serializing every increment behind a row lock. The hot row is the killer even before total throughput.
- It couples the hot read path to a write. Playback should be a pure cached read; making every view do a synchronous DB write drags database contention onto the latency-critical path.
- Fraud and correctness. A raw increment counts bots, refreshes, and double-fires. Real view counting needs dedup and rules (a “view” is watch-past-N-seconds, one per user per window), which you cannot enforce in a blind increment.
Evolution: treat views as an event stream and aggregate asynchronously; never count synchronously.
Player heartbeat/view events
-> Ingest gate (auth, sample, dedup by (user,video,session) window, drop bots)
-> Event Stream (Kafka), partitioned BY video_id
-> Stream processor (Flink/Spark Streaming):
* windowed aggregation: per-video counts per 1-min window
* de-dup + view-rules (watched > N sec = 1 view)
* watch-time sums, unique-viewer approximation
-> write ROLLUPS (not per-event) to the count store:
incremental counters updated once per window per video, not per event.
The moves that make it work:
- Aggregate before you write. The stream processor collapses millions of raw events into one rollup write per video per time window. A viral video getting 1M views/min becomes one counter update per minute, not 1M row updates. Throughput to the datastore drops by orders of magnitude and the hot-row problem evaporates.
- Partition the stream by
video_id. All events for a given video land on the same partition, so a single processor task owns its running count - no cross-node coordination to aggregate one video. A pathologically hot video can be sub-partitioned (split its events across shards, sum the shard counts) to avoid one overloaded partition - the classic hot-key fix. - Approximate where exactness is pointless. Unique-viewer counts use HyperLogLog (kilobytes of state for billions of items, ~1-2% error) instead of storing every viewer id. Nobody needs the exact unique count; they need a good estimate cheaply.
- Two-speed counting. A fast, approximate path (in-memory increment in a cache like Redis, or a short-window stream) gives a near-real-time number for the UI within seconds; a slow, exact batch path (reprocess the event log) produces the authoritative, fraud-filtered number for creator analytics and monetization. The UI count being slightly ahead of or behind the audited number is fine.
- Idempotency and replay. Events carry ids; the pipeline is idempotent so a Kafka replay after a processor failure does not double-count. The raw event log is the source of truth and can be reprocessed if the counting rules change.
The same pipeline produces watch-time, retention curves, and engagement signals that feed the recommender and creator analytics - all as rollups off the one event stream, never as synchronous writes on the playback path. Counting billions of views becomes “aggregate a partitioned stream into per-window rollups,” which is boringly scalable, instead of “hammer a row 15M times a second,” which is impossible.
API Design & Data Schema
Most traffic is manifest fetches + segment GETs (served by the CDN, not these APIs) plus view events; the APIs below handle upload orchestration, metadata, and the playback handshake.
REST / RPC endpoints
# --- Upload (orchestration; bytes go direct-to-blob) ---
POST /api/v1/videos/upload/init
Body: { "title":"...", "size_bytes": 4200000000, "content_type":"video/mp4" }
-> creates a video record (status=uploading), returns a resumable session.
Response 201: { "video_id":"v_9f3", "upload_id":"u_77",
"chunk_urls":[ "https://blob/...part=0&sig=...", ... ],
"chunk_size": 8388608 }
PUT <signed chunk_url> # client uploads each chunk DIRECT to blob store
-> 200 per chunk; resumable (re-PUT a failed chunk only).
POST /api/v1/videos/upload/complete
Body: { "video_id":"v_9f3", "upload_id":"u_77", "parts":[{"n":0,"etag":"..."}] }
-> validates, marks status=uploaded, enqueues transcode job.
Response 200: { "video_id":"v_9f3", "status":"processing" }
# --- Metadata + playback handshake ---
GET /api/v1/videos/{id}
-> metadata + status; when ready, the manifest URL for the player.
Response 200: { "video_id":"v_9f3", "title":"...", "duration_s":612,
"status":"ready", "uploader_id":"c_12",
"manifest_url":"https://cdn/.../v_9f3/master.m3u8",
"thumbnails":[ "...jpg" ], "view_count": 1043201 }
# status may be "processing" (renditions not ready) -> player shows "processing".
GET https://cdn/.../v_9f3/master.m3u8 # the MANIFEST (served by CDN)
-> lists ladder + variant playlists (per-rendition segment lists).
GET https://cdn/.../v_9f3/720p/seg_042.ts # a SEGMENT (served by CDN edge)
-> immutable, long cache TTL. 99%+ edge hit. This is the hot path.
# --- View / watch telemetry (fire-and-forget to the ingest gate) ---
POST /api/v1/events/watch
Body: { "video_id":"v_9f3", "session_id":"s_88", "user_id":"u_5",
"type":"heartbeat", "position_s": 45, "played_s": 10 }
-> 202 Accepted; lands on the event stream. NO synchronous count write.
PATCH /api/v1/videos/{id} { "visibility":"public", "title":"..." }
-> update mutable metadata (owner only).
Data store schemas
1. Video Metadata DB - SQL / NewSQL, sharded by video_id. Small structured records, relational (videos -> renditions, videos -> uploader), needs modest consistency on the upload commit. Read-heavy on the playback handshake, so front it with a cache + read replicas.
Table: videos
video_id BIGINT / UUID PRIMARY KEY / SHARD KEY
uploader_id BIGINT INDEX
title TEXT
description TEXT
duration_s INT
status ENUM(uploading, uploaded, processing, ready, failed)
visibility ENUM(public, unlisted, private)
manifest_key TEXT -- object-store key of the master manifest
source_key TEXT -- object-store key of the cold master
created_at TIMESTAMP
INDEX (uploader_id, created_at DESC) -- a creator's videos, newest first
Table: renditions -- which ladder rungs/codecs are published for a video
rendition_id BIGINT PRIMARY KEY
video_id BIGINT SHARD KEY, INDEX
resolution TEXT -- 240p..4K
codec TEXT -- h264 / vp9 / av1
bitrate_kbps INT
manifest_key TEXT -- variant playlist key
status ENUM(pending, ready, failed)
INDEX (video_id)
2. Segment / Rendition Blob Store - object storage (S3-like), immutable, CDN-fronted. Keyed by object path ({video_id}/{rendition}/seg_{n}.ts). No transactions; durability from erasure coding across zones; hot/cold tiering by access. Source masters live here too, in a cold class.
Key: v_9f3/720p/seg_042.ts -> bytes (immutable segment)
Key: v_9f3/master.m3u8 -> bytes (manifest)
Key: v_9f3/source/master.mp4 -> bytes (cold, kept for re-encode)
Cache-Control: public, max-age=31536000, immutable (segments never change)
3. View / Analytics Store - wide-column / time-series (Cassandra / Bigtable / a columnar OLAP store). Holds per-video, per-window rollups and running totals written by the stream processor. High write of rollups (not raw events), point + range reads for the UI and analytics.
Table: view_rollups -- one row per (video, time bucket)
video_id STRING PARTITION KEY
bucket STRING CLUSTERING KEY -- e.g. 2026-07-09T14:30 (1-min)
views COUNTER
watch_time_s COUNTER
unique_hll BLOB -- HyperLogLog sketch for uniques
Table: video_totals -- denormalized running total for the UI
video_id STRING PARTITION KEY
total_views COUNTER
total_watch_s COUNTER
4. Raw Event Log - the durable stream (Kafka), partitioned by video_id. The source of truth for counts; retained long enough to reprocess if counting rules change. Not a queryable DB - it feeds the aggregation jobs.
Why these choices: video metadata is relational and small with a clean upload commit - SQL/NewSQL sharded by video_id (a video and its renditions co-locate). Segments are immutable, enormous, streamed - an object store, where SQL would be catastrophic, and where content addressing plus long TTLs make the CDN trivially safe. View data is append-heavy, aggregate-read, tolerant of approximation - a wide-column/OLAP store fed by rollups, never a transactional counter. Each layer gets exactly the consistency and cost model its access pattern needs; forcing one store to do all four is the naive mistake.
Bottlenecks & Scaling
Where it breaks first, in order, and the fix for each:
1. Transcoding fleet saturation (breaks first under upload spikes). A surge of uploads (or a batch re-encode to a new codec) can back up the job queue for hours.
Fix: transcoding is a queue-fed, autoscaled batch fleet - scale workers on queue depth; it never blocks upload acceptance (upload success only needs the master stored + a job enqueued). Prioritize: encode the common rungs (360p/720p) first so a video goes ready fast, then backfill 4K/AV1 lazily. Segment-parallel encoding keeps per-video latency near-constant, and per-segment retries mean a crash re-does one segment, not a whole film. Use GPU/ASIC encoders for throughput; reserve expensive AV1 for popular videos where the bandwidth savings pay for the CPU.
2. CDN edge hit rate / origin protection (the delivery bottleneck). Fix: this is what immutable segmentation + tiered caching exist for. ~99%+ edge hits keep origin egress a trickle; a regional origin-shield tier collapses edge misses so a newly-viral video fills the shield once, not the origin N times (thundering-herd defense). Pre-warm predictably-hot content (premieres, recommender pushes). Long/immutable cache TTLs because segments never mutate.
3. Hot video / hot key (a video goes viral).
Fix on the read side: the CDN absorbs it - immutable segments cache perfectly, and the shield tier prevents an origin stampede; replicate hot segments to more edges/zones. Fix on the count side: a single hot video_id partition in the event stream is sub-partitioned (spread its events across shards, sum the shard counts), the classic hot-key split, so one viral video does not overload one aggregation task.
4. View-count write amplification. Fix: never write per event. The stream processor aggregates to one rollup write per video per window; a fast approximate path (Redis/short-window) feeds the UI in near-real-time while a slow exact batch path produces audited numbers. HyperLogLog for uniques keeps state tiny. The raw event log is replayable and idempotent so failures do not double-count.
5. Metadata DB read pressure on the playback handshake. Every playback start reads the video record + manifest pointer.
Fix: this is read-heavy and highly cacheable (metadata rarely changes) - front the Metadata DB with a cache (video record + manifest URL) and read replicas; the manifest itself is served by the CDN, not the DB. Shard by video_id so reads spread uniformly.
6. Upload of huge files over flaky networks. Fix: chunked resumable upload direct to blob storage via signed URLs - bytes never transit the app tier, a dropped connection resumes from the last acked chunk, and parallel chunk uploads use the creator’s uplink fully. The app tier only orchestrates.
7. Storage cost at exabyte scale. Fix: hot/cold tiering - recent/popular renditions on fast storage, the long tail demoted to cheap cold classes; source masters always cold (kept only for re-encode). Erasure coding (~1.4x) instead of 3x replication at equal durability. Drop or lazily-generate rarely-watched rungs (do not pre-encode 4K AV1 for a video nobody watches in 4K).
8. Shard keys, stated plainly. Metadata DB -> video_id (a video and its renditions are one co-located unit; playback reads and the upload commit both key on it). Event stream + view store -> video_id (all of a video’s events aggregate on one partition; sub-partition the hot ones). Segment blob store -> content path {video_id}/{rendition}/seg_n (path is the placement, and immutability makes it CDN-cacheable). Sharding metadata by uploader_id would hot-spot on mega-creators; sharding view events by user_id would scatter one video’s count across every partition and make aggregation a cross-shard join. The video is the unit of metadata and counting; the segment path is the unit of storage identity.
9. Single points of failure. API/upload/metadata services are stateless and horizontally scaled behind load balancers. The Metadata DB is quorum-replicated with failover. Blob storage is erasure-coded across zones for 11-nines durability. The CDN is inherently redundant (thousands of PoPs; a dead edge just routes to the next). The event stream (Kafka) is replicated; the aggregation jobs checkpoint and resume. No single box whose loss loses an accepted upload or blocks playback of a published video.
10. Multi-region. Blob storage and CDN are global by construction - segments replicate cross-region and serve from the nearest edge. Metadata has a home region per video with async cross-region read replicas (playback needs only a read; a few seconds of replication lag on a fresh upload is fine per the NFR). The transcode fleet runs region-local to the source to avoid shipping masters across the backbone. Playback correctness never depends on any one region being up; only a brand-new upload’s global visibility depends on replication, and that is allowed to lag.
Wrap-Up
The trade-offs that define this design:
- Do the expensive work once, offline. Transcoding is heavy and slow, so it is pushed entirely off the request path onto an autoscaled, queue-fed, segment-parallel fleet - trading eventual readiness (a fresh video is “processing” for minutes) for a playback path that is pure cheap cached reads. Synchronous/inline transcoding was rejected because it cannot scale and cannot hold an HTTP request open for an hour.
- Immutable segments over monolithic files. Cutting each rendition into short, keyframe-aligned segments enables client-side ABR (smooth quality adaptation), trivially-safe permanent CDN caching (no invalidation), and per-segment fault-tolerant encoding - at the cost of a manifest indirection and per-segment overhead. A single fixed-bitrate MP4 was rejected because it fits no one and cannot adapt mid-stream.
- The CDN is the delivery system, not an add-on. ~100,000:1 read:write and 150 Tbps peak egress mean the origin must almost never be touched; immutable segments + tiered edge/shield caching keep hit rate at ~99%+, which is simultaneously the latency lever (TTFF near edge RTT) and the cost lever (origin egress as a trickle). Serving from origin was rejected as physically impossible at scale.
- Count with a stream, never a counter. Views and watch-time are aggregated from a
video_id-partitioned event stream into per-window rollups - one write per video per window instead of 15M row updates per second - with HyperLogLog for uniques, sub-partitioning for hot keys, and a two-speed (fast-approximate + slow-exact) split. A synchronousviews + 1was rejected because the hot row and the write volume both make it impossible. - Consistency spent where it matters. Strong-ish only at the upload commit and on not losing the master; everything on the read side (new video visibility, view counts, rendition readiness) is eventually consistent, which is exactly what buys the cacheability and scale.
One-line summary: a video platform that accepts resumable direct-to-blob uploads, transcodes each master offline into a keyframe-aligned rendition ladder via a segment-parallel fleet, packages it into immutable HLS/DASH segments served from a tiered CDN so ~99%+ of playback bytes come from the edge, lets a client-side ABR player pick quality segment-by-segment to keep time-to-first-frame low and rebuffering rare, and counts billions of views by aggregating a video_id-partitioned event stream into per-window rollups instead of ever touching a synchronous counter - so the expensive work happens once at ingest and the read path stays a cheap, cacheable, globally-distributed firehose.
Comments