Someone uploads a 4GB 4K master. Before anyone can watch it, that one file has to become roughly a dozen different versions - 360p, 480p, 720p, 1080p, 1440p, 4K, each in two or three codecs - and every version has to be cut into short segments and wrapped in a manifest so an adaptive player can stream it. That is tens of minutes to several hours of CPU work per upload. The naive mental model is “call ffmpeg on the file and wait.” That model dies the instant you have more than one user, because encoding a 2-hour movie serially takes as long as the movie, times the number of renditions, on one machine that might crash 90% of the way through and lose everything.
This post is not about delivery. It is not about CDNs, adaptive bitrate decisions in the player, or counting views. Those are a separate problem (I wrote that one up for a video streaming platform). This is specifically about the transcoding pipeline: how you turn one immutable master into a published rendition ladder across a fleet of worker machines, how you model the work as a DAG of encoding jobs so it parallelizes, how you survive a worker dying mid-job without redoing hours of work, and how you get time-to-first-viewable down from “the whole thing is encoded” to “a watchable rendition is live in seconds.” The hard part is the orchestration, the fault tolerance, and the latency to first playable rung. Let me build it properly.
Functional Requirements (FR)
In scope:
- Ingest a master and produce a rendition ladder. Given a durably-stored source file (any common container and codec, up to several GB), produce every rung of a configured ladder (360p through 4K) in every target codec (H.264 for compatibility, VP9/AV1 for efficiency).
- Package into streaming output. Each rendition is cut into short keyframe-aligned segments plus a manifest (HLS
.m3u8/ DASH.mpd) so a player can stream it adaptively. - Orchestrate the work as a DAG. The pipeline models probe, split, per-segment encode, per-rendition stitch/package, QC, and publish as a dependency graph, runs independent nodes in parallel across a worker fleet, and tracks each node’s state.
- Survive worker failures mid-job. A worker crashing, getting preempted (spot/VM reclaim), or hanging in the middle of encoding a segment must not fail the whole video or lose completed work. The orchestrator retries at the smallest sensible unit.
- Minimize time-to-first-viewable (TTFV). Publish a watchable low rung as fast as possible (seconds to low minutes), then backfill the heavy rungs, instead of waiting for the entire ladder.
- Report progress and status. The uploader sees
processingwith progress, thenready; a failed transcode surfaces asfailedwith a reason, and the job is replayable from the master. - Be idempotent and re-runnable. Re-encoding the same master (new codec added, encoder bug fixed) is a first-class operation that reuses the stored source and does not double-charge already-done work.
Explicitly out of scope (say it so you own the boundary):
- Delivery / CDN / edge caching. The pipeline’s output is immutable segments in object storage. How those reach viewers is a separate system; I stop at “published.”
- The ABR player and client-side quality switching. I produce the ladder and manifest; the player’s bitrate decisions are not my problem here.
- View counting, watch-time, recommendations. Downstream analytics off a separate event stream. Not touched.
- Upload mechanics. I assume the master is already durably in blob storage via some resumable direct-to-blob upload; I start from the “new master available” event.
- DRM / content-ID / thumbnails and sprite sheets. Real work that hangs off the same DAG as extra leaf nodes; I note the hook point and move on.
- Live / low-latency streaming. Live has no full pre-transcode - it encodes on the fly with a different latency model. I design video-on-demand (VOD) transcoding and note in one line where live diverges.
The one decision that drives everything: transcoding is expensive, offline, embarrassingly parallel per-segment, and must be fault-tolerant at a fine grain, and the single most user-visible metric is time-to-first-viewable, not time-to-fully-encoded. Everything below falls out of modeling the job as a DAG of small idempotent segment-level tasks, scheduling them across a fleet that treats worker death as normal, and prioritizing the graph so a watchable rung publishes first.
Non-Functional Requirements (NFR)
- Scale: on the order of a large UGC platform - assume ~500 hours of video uploaded per minute (a real YouTube-class figure), petabytes of new source per day, and a fleet of tens of thousands of encoder cores / hardware encoders kept busy. Bursty: upload spikes and bulk re-encode campaigns can multiply steady load several times over.
- Latency targets: this pipeline is throughput-bound, not request-latency-bound, with one exception that dominates design - time-to-first-viewable (TTFV) should be seconds to low single-digit minutes for a short clip and a few minutes even for long content, by publishing a low rung first. Full-ladder completion (4K, AV1) is allowed to take many minutes to hours and run lazily.
- Availability: the pipeline can tolerate brief degradation - a video stuck at
processinga few extra minutes is acceptable and never user-facing as an outage. What is not acceptable is losing an accepted master or silently dropping a job so a video never becomes playable. - Consistency: per-job state (which DAG nodes are done) needs to be correct and durable so a crashed orchestrator resumes exactly where it left off and never double-publishes or declares
readyprematurely. The published output is write-once immutable. Eventual consistency is fine for progress reporting. - Durability: 11 nines on the source master (the crown jewel - every re-encode depends on it) and on published renditions. Intermediate per-segment encoded files can be lower durability since they are cheap to recompute from the master.
- Efficiency is a first-class NFR. Encode compute is the dominant cost. The design must avoid wasted work: fine-grained retries (redo one segment, not one video), reuse across re-encodes, and not pre-encoding rungs nobody will watch. A pipeline that is correct but burns 2x compute is a failed design at this scale.
Back-of-the-Envelope Estimation (BoE)
Real numbers, with the arithmetic, because they set the fleet size and the queue design.
Ingest volume:
500 hours of video uploaded per minute (YouTube's stated figure).
500 hrs/min / 60 = ~8.3 hours of video uploaded per SECOND.
Discrete files: say avg clip ~10 min of footage.
500 hrs/min = 30,000 min of video/min / 10 = ~3,000 uploads/min = ~50 uploads/sec.
Peak (bursty ~5x) ~250 uploads/sec. These become transcode JOBS.
Encode compute (the thing that sizes the fleet):
Each master -> a ladder of ~7 rungs (360,480,720,1080,1440,4K + one extra)
x ~2-3 codecs (H.264 always; VP9/AV1 for popular/high rungs).
Per-rendition encode speed on a CPU core: ~0.5x to a few x realtime (AV1 far worse,
hardware ASIC encoders far faster than realtime).
8.3 hrs of video/sec * ~7 rungs = ~58 rendition-hours of video to encode each wall second.
At ~0.5x realtime on hardware encoders => ~29 machine-hours of encode per wall second
=> a steady fleet of ~100,000 encode-seconds of capacity per wall second,
i.e. tens of thousands of busy encoder cores/ASICs at all times.
AV1 at software speeds can be 10-50x slower than H.264 - which is exactly why AV1 is
reserved for popular videos and run lazily, never on the TTFV path.
Segment fan-out (the DAG width):
Split a master into GOP-aligned chunks of ~4-6s each.
A 10-min video = 600s / 5s = ~120 segments.
A 2-hour film = 7200s / 5s = ~1,440 segments.
Each segment encodes into every (rung x codec) = ~7 x ~2 = ~14 variants.
10-min video => 120 * 14 = ~1,680 independent encode TASKS.
2-hour film => 1,440 * 14 = ~20,000 independent encode tasks.
At ~50 uploads/sec, steady task creation is on the order of
50 * ~1,680 (short-clip avg) = ~84,000 encode tasks/sec entering the queue.
This is the number that matters: the orchestrator and queue see ~10^5 tasks/sec,
each small, idempotent, independently retryable.
Intermediate and output storage:
Source: petabytes/day (kept, cold, for re-encode).
Per-segment encoded outputs (intermediate) total ~1.5-2x source bytes across the ladder;
they are transient - deletable once the final per-rendition file is stitched + published.
Published renditions: ~1.5-2x source, immutable, hot tier then demoted.
Per-job scratch (demuxed segments, logs) is small and TTL'd aggressively.
Job state / control plane:
Concurrent in-flight jobs: ~50 new/sec * say ~5 min avg in pipeline = ~15,000 jobs in flight,
peak ~75,000. Each job's DAG state (nodes, statuses, attempt counts) is a few KB.
=> low-GB of hot control-plane state. Trivial for a datastore; the challenge is not
size but CORRECTNESS of state transitions under worker churn.
The takeaways that dictate architecture: encode compute is the dominant cost and needs a tens-of-thousands-core autoscaled fleet; the orchestrator deals with ~10^5 small idempotent tasks/sec, so tasks must be cheap to create, schedule, retry, and dedup; intermediate storage is large but transient; and the control plane is small but must be transactionally correct because it is what lets a crashed worker or orchestrator resume without losing or double-doing work. TTFV is won by prioritizing the graph, not by making encode faster.
High-Level Design (HLD)
The pipeline is a control plane (orchestrator plus durable job state plus a task queue) driving a data plane (a stateless, autoscaled worker fleet that reads the master and intermediate chunks from blob storage and writes encoded chunks back). A “new master” event creates a job, the orchestrator expands that job into a DAG of tasks, pushes ready tasks onto a queue, workers pull and execute them against blob storage, report results back, and the orchestrator advances the DAG until a publishable set of renditions exists - at which point it publishes and flips status. Crucially the orchestrator owns the truth of DAG state; workers are dumb, stateless, and interchangeable.
CONTROL PLANE (owns job/DAG state, schedules, retries)
┌───────────┐ "new master" ┌──────────────────┐ persist ┌────────────────┐
│ Ingest / │───────────────>│ Orchestrator / │<──────────>│ Job State DB │
│ upload svc│ event (ref │ DAG Scheduler │ DAG nodes │ (jobs, nodes, │
│ │ to master) │ (expand->schedule│ + statuses │ attempts) SQL │
└───────────┘ │ ->advance) │ └────────────────┘
└───────┬───────────┘
ready │ ^ results (done/fail/ ┌────────────────┐
tasks │ │ heartbeat) │ Blob Storage │
▼ │ │ - source master│
┌────────────────────┐ │ - intermediate │
│ Task Queue │ │ seg encodes │
│ (priority, visible-│ │ - published │
│ ity timeout, DLQ) │ │ renditions │
└─────────┬──────────┘ └───────▲────────┘
DATA PLANE (stateless, autoscaled) │ pull task │ read master
┌───────────────────────────────────▼──────────────────────────┐ │ chunk; write
│ WORKER FLEET (probe / split / encode-segment / stitch / │──────┘ encoded chunk
│ package / QC) - each task is ONE DAG node, idempotent, │
│ key-addressed output, lease-based, heartbeating. │
│ Mix of GPU/ASIC (fast rungs) and CPU (AV1) workers. │
└───────────────────────────────────────────────────────────────┘
│ on publishable set ready
▼
┌────────────────┐ flip status=ready, store manifest ptr ┌────────────────┐
│ Publisher │──────────────────────────────────────────>│ Metadata DB │
│ (atomic flip) │ (low rung first => TTFV) │ (video status) │
└────────────────┘ └────────────────┘
End-to-end request flow:
- The ingest side stores the master durably and emits a “new master” event carrying the object key (never the bytes). The orchestrator creates a job row with status
probing. - The orchestrator schedules a probe task. A worker runs
ffprobeon the master: duration, resolution, codec, keyframe (GOP) structure, audio tracks, HDR, corruption. This decides the ladder (do not make a 360p source into a 4K rung) and the split points. - The orchestrator expands the DAG: a
splitnode, then oneencodenode per(segment x rung x codec), then per-renditionstitch/packagenodes, thenQCandpublishnodes. It persists all nodes aspendingin the Job State DB. - It pushes ready nodes (those whose dependencies are satisfied) onto the task queue, with priority - the lowest viewable rung’s path first (for TTFV).
- Workers pull tasks, lease them, read the needed input chunk from blob storage, run
ffmpeg, write the output chunk to a deterministic key, heartbeat while working, and reportdone(orfailed) to the orchestrator. - The orchestrator marks the node
done, checks whether that unlocks downstream nodes (e.g. all segments of the 360p H.264 rung done -> itsstitchbecomes ready), and schedules them. - As soon as one full low rung is stitched, packaged, and QC’d, the Publisher atomically flips the video to
readywith a manifest that contains that rung. The video is now watchable - TTFV achieved - while the orchestrator keeps grinding the heavy rungs and adds them to the manifest as they finish. - When the configured publishable set is complete, the job is
done; intermediate scratch is garbage-collected; the master is demoted to cold storage, retained for future re-encode.
The key insight: the orchestrator is a durable state machine over a DAG, workers are stateless and disposable, and every unit of work is a small idempotent task whose output lands at a deterministic key. That combination is what makes the whole thing both massively parallel and tolerant of constant worker death, and what lets the scheduler prioritize the path to first-viewable.
Component Deep Dive
The hard parts, each naive-first then evolved: (1) modeling the work as a DAG of encoding jobs and scheduling it; (2) surviving worker failure mid-job without losing or duplicating work; (3) minimizing time-to-first-viewable; (4) the split and stitch mechanics that make segment-parallel encoding correct.
1. The encoding DAG: why a linear script fails, then a scheduled dependency graph
Naive approach: a single script per upload - probe; for rung in ladder: for codec in codecs: ffmpeg master -> rendition; package; publish - run on one worker that holds the job start to finish.
Where it breaks:
- No parallelism across the thing that is embarrassingly parallel. A 2-hour master encoded rung-by-rung, codec-by-codec, serially on one box takes many hours of wall-clock. Meanwhile ten thousand idle cores sit next to it. The work is independent per segment and per rung, and the linear script throws all of that away.
- The whole job is one failure domain. The box dies at 90% and the entire multi-hour effort is gone; you restart from zero. There is no notion of “redo just the broken piece.”
- No prioritization. The script cannot publish a watchable 360p first, because it has no model of the work as separable units with dependencies - it just runs top to bottom. TTFV is “whenever the whole script finishes,” which is terrible.
- One worker’s resources bound one video. A long video cannot borrow the fleet; throughput per video is capped at one machine’s encoder.
Evolution: model the job as an explicit DAG and let a scheduler run it. Represent every unit of work as a node with typed dependencies:
┌──> encode(seg0, 360p, h264) ─┐
├──> encode(seg0, 720p, h264) ─┤
probe ──> split ────┼──> encode(seg1, 360p, h264) ─┼──> stitch(360p,h264) ─┐
│ ├──> encode(seg1, 720p, h264) ─┤ ├──> package ──┐
│ └──> ... (seg x rung x codec) ─┘──> stitch(720p,h264) ─┘ │
│ (per rendition) ▼
└─> (ladder decision feeds which encode nodes exist) QC ──> publish
The DAG has clear layers:
probe(1 node): inspect the master, decide the ladder and GOP-aligned split points. Everything depends on it.split(1 node, or itself fanned out): cut the master into independent keyframe-aligned chunks (details in part 4).encode(segment, rung, codec)(the fan-out, thousands of nodes): the bulk of the compute. Each is independent - it reads one input chunk, writes one output chunk. This is where the ~10^5 tasks/sec lives.stitch(rung, codec)(one per rendition): depends on all that rendition’s segment-encodes; concatenates them in order into the final rendition file.package: produce HLS/DASH segments and manifests from the stitched renditions.QC: validate output (duration matches, no black frames, VMAF/PSNR quality gate, manifest well-formed).publish: atomically make renditions live.
Scheduling the DAG. The orchestrator keeps each node’s state (pending, ready, running, done, failed) in the durable Job State DB. A node becomes ready when all its dependencies are done; ready nodes are pushed to the task queue. This is a standard topological execution, but at scale three things matter:
- The DAG is wide, not deep. The critical path is short (
probe -> split -> encode -> stitch -> package -> QC -> publish, ~7 levels) but each encode level is thousands of nodes wide. Wall-clock is therefore bounded by(encode time of one segment) + (fleet-is-big-enough)rather than by video length. A 2-hour film and a 2-minute clip finish the encode layer in similar wall time if the fleet can absorb the width. - Node granularity is a deliberate trade-off. Too coarse (encode a whole rung in one node) and you lose fault-tolerance and parallelism; too fine (one node per 1s) and orchestration overhead and queue pressure dominate. Segment-sized (~4-6s) encode nodes are the sweet spot - small enough to retry cheaply, large enough that per-task overhead is a rounding error against encode cost.
- The scheduler is sharded. At 10^5 tasks/sec a single orchestrator process is a bottleneck. Shard orchestration by
job_id(one job’s DAG is owned by one orchestrator shard, so no cross-shard coordination to advance a graph), with the job-to-shard mapping in a consistent-hash ring. Each shard drives its jobs’ state machines against its slice of the Job State DB.
The DAG model is what unlocks everything else in this post: parallelism (independent nodes), fault tolerance (retry a node), and TTFV (prioritize a subgraph). A linear script has none of those because it never names the pieces.
2. Surviving worker failure mid-job: why “retry the job” fails, then leases plus idempotent tasks
At this fleet size, workers dying mid-job is not an exception, it is the steady state. Spot/preemptible VMs get reclaimed with seconds of notice, machines OOM on a pathological 8K HDR segment, encoders hang on corrupt input, a rack loses power. If the design treats worker death as rare, it is already broken.
Naive approach: if a worker does not finish, fail the whole job and re-run it from the start. Detect failure by “worker did not report done within T.”
Where it breaks:
- Catastrophic wasted work. A single preemption in segment 1,400 of 1,440 throws away ~20,000 completed encode tasks. At spot-reclaim rates this means almost no long video ever finishes - you are perpetually restarting.
- Timeout is the wrong signal, and it is ambiguous. Did the worker die, or is it a legitimately slow AV1 segment, or did it finish and the ACK got lost? A blind job-level timeout cannot tell, and either kills live work or waits far too long.
- Retrying non-idempotently corrupts output. If a “retry” re-runs a task whose previous attempt actually half-wrote its output, you can get duplicated or interleaved bytes. Without idempotency, retry is unsafe.
- A single “poison” segment kills the job forever. If one segment always crashes the encoder (genuinely corrupt input), a job-level retry loops endlessly, re-doing 1,439 good segments every time to die on the same bad one.
Evolution: fine-grained leased tasks, idempotent key-addressed outputs, heartbeats, bounded retries, and a dead-letter path.
Leases, not fire-and-forget. When a worker pulls an encode task it acquires a lease (a visibility timeout on the queue plus a running row with worker_id, attempt, lease_expiry in the Job State DB). The worker must heartbeat to extend the lease while encoding. If heartbeats stop (crash, preemption, hang) the lease expires and the task becomes ready again - re-queued automatically, at segment granularity. Only that one ~5s segment is redone, not the video.
worker pulls encode(seg1400, 1080p, av1)
-> lease(attempt=1, expiry=now+90s), status=running, worker=W42
-> heartbeat every ~20s extends expiry (progress is visible, lease stays alive)
-> W42 is PREEMPTED at 70% ... heartbeats stop
-> lease expires -> orchestrator marks task ready again (attempt bumped on next pull)
-> worker W87 pulls it, re-encodes the SAME segment from the SAME input chunk
-> writes to the SAME deterministic key -> overwrites/wins cleanly. No corruption.
Net loss: one partially-encoded ~5s segment. Not the 2-hour job.
Idempotency via deterministic keys. Each encode task writes its output to a content/coordinate-deterministic key: jobs/{job_id}/enc/{segment_idx}/{rung}/{codec}.ts (optionally suffixed with a hash of the encode parameters). Re-running the task overwrites the same key with identical bytes (encoding is deterministic given the same input, codec, and settings). So a duplicate execution from a lost ACK or a slow-but-not-dead worker is harmless - last writer wins and the bytes are the same. The orchestrator treats “output key exists and is valid” as the real completion signal, not just the worker’s word. This also makes the “finished but ACK lost” ambiguity safe: on re-pull the worker sees the output already exists (or just recomputes it) - either way the DAG node ends done with correct bytes.
Detecting failure precisely. Three independent signals, not one timeout:
- Heartbeat/lease expiry: the fast path for crash and preemption (lease TTL on the order of a couple of minutes, heartbeats far more often). Preemptible VMs also send a termination notice (seconds of warning); a well-behaved worker catches it and voluntarily NACKs its lease so the task re-queues immediately instead of waiting for TTL.
- Explicit failure report: the worker’s
ffmpegexited non-zero -> it reportsfailed(reason)right away, no waiting on a timeout. - QC catch-all: even if a task reports
done, the downstreamQCnode revalidates the stitched rendition, so a subtly-bad encode (wrong duration, corruption) is caught before publish.
Bounded retries with backoff, then dead-letter. A task that fails is retried with exponential backoff up to a cap (say 3-5 attempts), ideally rescheduled onto a different worker (and for OOM, onto a higher-memory worker class). If it still fails, it goes to a dead-letter queue and the node is marked failed with its logs. Now the orchestrator makes a policy decision rather than looping forever:
- If the failed node is on a non-critical rung (the 4K AV1 variant), the job can still publish the rest and mark that single rendition
failed- a video missing its 4K rung is fine; it is not missing entirely. - If the failed node is on the minimum viable rung (can produce no watchable output at all), the job itself is marked
failed, surfaced to the uploader with a reason, and is replayable from the master after a fix. The poison segment never takes the good segments down with it - they are already done and durable.
Orchestrator failure too. The orchestrator is not special - it can die. That is why DAG state lives in the durable Job State DB, not in orchestrator memory. State transitions are written transactionally (claim a node, advance a node) so a restarted or failed-over orchestrator shard reloads its jobs and resumes exactly where it was - no node is lost, none is double-scheduled (the claim is a compare-and-set). The task queue’s visibility timeout covers tasks that were in flight when the orchestrator blipped. There is no single point whose death loses an accepted job.
The result: worker death costs one segment of redo, preemption is handled gracefully, duplicate execution is harmless because outputs are idempotent and key-addressed, a poison input fails one rendition instead of looping forever, and even the orchestrator is disposable because the truth is in the database.
3. Minimizing time-to-first-viewable: why “encode everything then publish” fails, then prioritized partial publish
Naive approach: encode the entire ladder, package all of it, then publish the video as ready.
Where it breaks:
- TTFV equals full-encode time, which can be hours. The uploader (and, for a premiere, waiting viewers) stare at “processing” until the slowest, heaviest rung - 4K AV1 at 10-50x slower than realtime - is done. The most expensive rung gates the first viewable frame of the cheapest rung. That is backwards.
- It wastes the asymmetry of the ladder. A 360p H.264 segment encodes in a fraction of the time a 4K AV1 segment does, yet the naive publish makes the fast output wait on the slow output for no reason.
- The critical path is mis-identified. The user-visible latency is “when can someone press play,” which only needs one complete, watchable rung. The naive design optimizes “when is the whole catalog of renditions done,” a completely different and much later moment.
Evolution: prioritize the subgraph that produces the minimum viable rung, publish partially, then backfill. TTFV is a scheduling problem, not an encode-speed problem. Three moves:
Move 1: priority scheduling of the first-viewable subgraph. The DAG scheduler pushes tasks with a priority field. The path probe -> split -> encode(all segments, 360p or 480p, H.264) -> stitch -> package -> QC -> publish for the lowest, cheapest, most-compatible rung gets top priority and jumps the queue ahead of every 4K/AV1 task. The fleet drains the first-viewable subgraph first. Because H.264 360p is cheap and the segments run in parallel across the fleet, this subgraph completes in seconds to low minutes even for long video.
Priority 0 (TTFV path): 360p/480p H.264 -> stitch -> package -> QC -> PUBLISH (seconds-min)
Priority 1 (common): 720p/1080p H.264 (minutes)
Priority 2 (heavy): 1440p/4K H.264, VP9 (minutes-tens)
Priority 3 (lazy/rare): 4K AV1, exotic rungs - only if the video warrants it (hours, lazy)
Move 2: partial, additive publish. The manifest is not all-or-nothing. As soon as the first-viewable rung is QC’d, the Publisher writes a manifest containing just that rung and flips the video to ready - playable now. Each subsequently finished rung is added to the manifest (an immutable new manifest version; the player picks up more quality levels as they appear). Publishing is an append to the ladder, so TTFV is decoupled from full completion.
Move 3: pipeline the first segments (fast-start within the rung). Even within the first-viewable rung you do not need every segment before playback can start - a player only needs the opening segments to begin. So the scheduler further prioritizes early segments of the first rung and the packager can publish a growing manifest (like a live window) so playback can begin while the tail of the video is still encoding. For a long upload this drops TTFV from “the whole 360p rung is done” to “the first few 360p segments are done,” i.e. seconds.
What “minimum viable rung” is depends on the probe. For an SD source, 360p may be the target and there is no 4K rung at all. For a vertical phone upload the ladder differs. The probe node picks the ladder so you never waste compute upscaling (encoding a 4K rung from a 480p master is pure waste and is skipped). The TTFV rung is always “the cheapest rung that is genuinely watchable on the dominant device for this content.”
Trade-off made explicit: partial publish means a viewer in the first minute might only have 360p/480p available and steps up as higher rungs land. That is a deliberate and correct trade - watchable now at modest quality beats unwatchable for an hour at perfect quality. The only thing that must be strongly correct is that the manifest never points at a rung before that rung is fully QC’d and durable, which the DAG dependency (publish depends on QC depends on package depends on stitch) guarantees.
Note where live diverges: live streaming is this idea taken to the limit - there is no pre-transcode and no “full ladder done” moment at all; segments are encoded on the fly into a sliding window and published continuously with ~1-2s segments. Same prioritized-partial-publish shape, no batch tail.
4. Split and stitch: why cutting anywhere fails, then GOP-aligned segmentation
The whole parallel-encode scheme depends on being able to cut the master into independently encodable chunks. Get this wrong and the parallelism is either incorrect (artifacts at boundaries) or impossible.
Naive approach: split the master into N equal time slices (every 5 seconds by wall-clock) and encode each slice independently.
Where it breaks:
- Video frames are interdependent. Modern codecs use I-frames (keyframes, self-contained), P-frames (reference earlier frames), and B-frames (reference earlier and later frames). If you cut at an arbitrary timestamp, a chunk can begin on a P or B frame whose reference frame is in the previous chunk. That chunk cannot be decoded in isolation - the encoder has nothing to predict from, producing corruption or a forced bad keyframe at the seam.
- Boundary artifacts and drift. Independent encodes that do not share GOP boundaries produce visible glitches and timestamp drift where segments join, and the stitched file can have gaps or overlaps in presentation timestamps.
- Segments do not line up across rungs. For ABR to switch rungs cleanly, the 360p and 1080p renditions must have segment boundaries at the same timestamps so the player can swap rung at a segment edge. Arbitrary per-rung cuts destroy that alignment.
Evolution: split only at GOP boundaries (IDR keyframes), aligned across the whole ladder.
- Cut at IDR keyframes. An IDR (Instantaneous Decoder Refresh) keyframe is a hard boundary: nothing after it references anything before it. Cutting only at IDR frames makes each chunk self-contained and independently decodable/encodable. The probe step reads the master’s keyframe positions; if the source keyframes are too sparse or irregular, the pipeline does a fast pre-pass that forces a consistent GOP structure (a keyframe every ~N seconds) so the split points are clean and uniform.
- Fixed GOP cadence shared by all rungs. Every rung is encoded with the same keyframe interval and the same segment boundaries (e.g. a keyframe and a segment edge every 4-6s at identical timestamps). This guarantees cross-rung alignment so ABR can switch at any segment boundary, and it means the
splitpoints defined once drive everyencodenode. - Stitch is an ordered concat, not a re-encode. Because each encoded segment is already a valid independently-decodable unit with aligned boundaries, the
stitch(rung, codec)node just concatenates the segment outputs in index order (a container-level concat / remux, no re-encoding) and thepackagenode emits the HLS/DASH segment list and manifest. Cheap, lossless, and parallelizable per rung. - Deterministic ordering. Each segment carries its
segment_idx, so stitch is a sort-by-index-then-concat. A re-run segment (from a retry) drops into the same index slot - order is never ambiguous.
master.mp4 ──probe (keyframe map)──> split points at IDR frames (uniform ~5s)
[seg0][seg1][seg2]...[segN] each starts on an IDR keyframe
│ │ │ │
encode each INDEPENDENTLY into (rung x codec), boundaries identical across rungs
▼
stitch per (rung,codec): concat seg0..segN in index order (remux, no re-encode)
▼
package: emit HLS/DASH segments + manifest; boundaries aligned so ABR switches cleanly
This is the mechanical foundation under parts 1-3: GOP-aligned splitting is why segments are independent (so the DAG fans out), why a retry can redo one segment cleanly (it is self-contained), and why cross-rung alignment lets the first-viewable rung and later rungs share a coherent manifest. Cut anywhere else and none of it holds.
API Design & Data Schema
The pipeline is mostly internal (event-driven, worker-to-orchestrator RPC). The external surface is small: submit/track a job and (for ops) inspect the DAG.
Endpoints (control plane)
# --- Job submission (usually fired as an EVENT from ingest, exposed as RPC too) ---
POST /api/v1/transcode/jobs
Body: { "video_id":"v_9f3", "source_key":"blob://masters/v_9f3.mp4",
"ladder_profile":"ugc_default", "priority_ttfv":true }
-> creates a job, status=probing, begins DAG expansion after probe.
Response 202: { "job_id":"j_5521", "status":"probing" }
# --- Track a job ---
GET /api/v1/transcode/jobs/{job_id}
-> status + progress + which renditions are published so far.
Response 200: {
"job_id":"j_5521", "video_id":"v_9f3",
"status":"processing", # probing|processing|ready|partially_failed|failed
"ttfv_published": true, # is a watchable rung live yet?
"progress": { "encode_tasks_done": 1420, "encode_tasks_total": 1680 },
"renditions": [
{"rung":"360p","codec":"h264","status":"ready"},
{"rung":"720p","codec":"h264","status":"ready"},
{"rung":"4k","codec":"av1","status":"encoding"}
],
"manifest_url":"blob://pub/v_9f3/master.m3u8"
}
# --- Re-encode (add a codec / fix a bug), reuses the stored master ---
POST /api/v1/transcode/jobs/{job_id}/reencode
Body: { "add_codecs":["av1"], "rungs":["1080p","4k"] }
-> new job over the SAME source_key; only the new (segment x rung x codec) nodes run.
# --- Ops: inspect the DAG ---
GET /api/v1/transcode/jobs/{job_id}/dag
-> nodes with {type, deps, status, attempts, worker_id, lease_expiry}.
# --- Worker <-> orchestrator (internal RPC) ---
POST /internal/tasks/lease { "worker_id":"W42", "capabilities":["gpu","av1"] }
-> leases the highest-priority ready task the worker can run.
Response: { "task_id":"t_88", "node":"encode", "segment_idx":1400,
"rung":"1080p","codec":"av1","input_key":"...","output_key":"...",
"lease_expiry": "....Z" }
POST /internal/tasks/{task_id}/heartbeat { "progress_pct": 70 } # extends lease
POST /internal/tasks/{task_id}/complete { "output_key":"...","vmaf":96.2 }
POST /internal/tasks/{task_id}/fail { "reason":"ffmpeg exit 1: ...", "oom":false }
Data stores
1. Job State DB - SQL / NewSQL, sharded by job_id. This is the control-plane source of truth. It needs transactional state transitions (claim a node, advance a node, bump an attempt) so a crashed orchestrator or duplicate worker cannot corrupt the DAG. Relational: a job has many nodes; nodes have dependency edges. Modest size (low GB hot), but correctness-critical, so a transactional store, not an eventually-consistent one.
Table: jobs
job_id BIGINT / UUID PRIMARY KEY / SHARD KEY
video_id BIGINT INDEX
source_key TEXT -- cold master object key (for re-encode)
ladder JSON -- rungs x codecs decided by probe
status ENUM(probing, processing, ready, partially_failed, failed)
ttfv_at TIMESTAMP NULL -- when the first watchable rung published
created_at TIMESTAMP
INDEX (status, created_at) -- scheduler scans for work
Table: dag_nodes
node_id BIGINT PRIMARY KEY
job_id BIGINT SHARD KEY, INDEX
type ENUM(probe, split, encode, stitch, package, qc, publish)
segment_idx INT NULL -- for encode nodes
rung TEXT NULL
codec TEXT NULL
priority SMALLINT -- 0 = TTFV path .. 3 = lazy
status ENUM(pending, ready, running, done, failed)
attempt INT DEFAULT 0
worker_id TEXT NULL
lease_expiry TIMESTAMP NULL
output_key TEXT NULL -- deterministic key of this node's output
INDEX (job_id, status, priority) -- find ready work, highest priority first
INDEX (status, lease_expiry) -- find expired leases to re-queue
Table: dag_edges -- dependency graph
job_id BIGINT SHARD KEY
node_id BIGINT -- depends on ->
depends_on BIGINT
PRIMARY KEY (job_id, node_id, depends_on)
2. Task Queue - a priority queue with visibility timeouts and a DLQ (Kafka with priority tiers, or SQS-like, or a purpose-built scheduler queue). Holds ready tasks; workers pull the highest-priority one they are capable of. Visibility timeout backs the lease so a vanished worker’s task reappears. A dead-letter queue catches tasks that exhaust retries. This is a derived view of dag_nodes (status ready); the DB remains the truth.
3. Blob Storage - object storage (S3-like), the data plane. Three key classes:
Key: masters/{video_id}.mp4 -> source master (COLD, 11 nines, kept)
Key: jobs/{job_id}/split/{segment_idx}.ts -> demuxed input chunk (transient, TTL)
Key: jobs/{job_id}/enc/{segment_idx}/{rung}/{codec}.ts -> encoded chunk (transient, TTL)
Key: pub/{video_id}/{rung}/{codec}/seg_{n}.ts -> PUBLISHED segment (immutable, hot->cold)
Key: pub/{video_id}/master.m3u8 -> manifest (immutable per version)
Cache-Control on published: public, max-age=31536000, immutable
Intermediate split/enc objects are lower durability and TTL’d - cheap to recompute from the master, so there is no reason to pay 11-nines for scratch. Published output and the master are the durable ones.
4. Metadata DB - SQL, the handoff to the rest of the product. The pipeline writes only the video’s transcode status and manifest pointer here; delivery and discovery read it. Kept separate so the transcode control plane and the product metadata plane evolve independently.
Table: videos (subset the pipeline writes)
video_id BIGINT PRIMARY KEY
status ENUM(uploaded, processing, ready, failed)
manifest_key TEXT -- set/updated as rungs publish (additive)
ttfv_ms INT NULL -- observed time-to-first-viewable, for SLO tracking
Why these choices: the Job State DB is SQL/NewSQL because DAG scheduling is a correctness problem - claim-and-advance must be transactional and resumable, and the data is relational (nodes, edges) and small; a NoSQL store with weak consistency would let two workers both “own” a node or an orchestrator resume into a corrupt graph. The queue is separate from the DB because pull/lease/retry at 10^5/sec is a messaging workload, not a query workload, and it can be a cheap derived view. Blob storage holds the enormous media because it is immutable, streamed, and erasure-coded for durability, where SQL would be absurd - and the deterministic keys are what make retries idempotent. Intermediate media is deliberately cheap and transient because the master makes it recomputable. Each layer gets exactly the consistency and cost model its job needs; forcing the DAG state into the blob store or the media into SQL is the naive mistake.
Bottlenecks & Scaling
Where it breaks first, in order, and the fix for each.
1. Encode fleet saturation (breaks first). An upload spike or a bulk re-encode campaign (new codec across the back catalog) floods the queue; jobs back up for hours.
Fix: the fleet is queue-depth autoscaled - add workers as ready task count climbs. It never blocks ingest (a job needs only the master stored and the job row created). Priority tiers protect TTFV: a backlog delays 4K AV1, not the first-viewable 360p path, because priority-0 tasks always drain first. Use hardware encoders (GPU/ASIC) for the common rungs to get far-above-realtime throughput, and reserve software AV1 for genuinely popular videos where bandwidth savings repay the CPU - never spend AV1 compute on the long tail. Bulk re-encodes run as low-priority background jobs so they soak spare capacity without starving live uploads.
2. Orchestrator / scheduler throughput (the control-plane bottleneck). A single orchestrator advancing DAGs and dispatching 10^5 tasks/sec is a wall.
Fix: shard the orchestrator by job_id over a consistent-hash ring; one shard owns a job’s whole DAG (no cross-shard coordination to advance a graph). Each shard scales independently; a dead shard’s jobs fail over to another and resume from the Job State DB. The queue absorbs dispatch rate; the DB indexes (job_id, status, priority) make “find ready work” cheap.
3. The hot job (one pathological upload). A single 10-hour 8K upload explodes into ~100k encode nodes and could monopolize the fleet or overwhelm one orchestrator shard. Fix: per-job concurrency caps / fair scheduling so one giant job cannot starve thousands of small ones - cap how many of a job’s tasks run at once and interleave across jobs. The job still finishes (its TTFV rung publishes early); its heavy tail just shares the fleet politely. If one shard runs hot from a huge DAG, the node set can be sub-sharded across orchestrator workers (encode nodes are independent, so splitting ownership of a job’s encode layer is safe).
4. The poison segment / repeated worker kills. One corrupt or pathological segment repeatedly crashes or OOMs workers. Fix: bounded retries with backoff, reschedule onto a different (and for OOM, bigger) worker class, then dead-letter. The poison node fails in isolation - the good segments are already durable, and the job either publishes without that one rendition (non-critical rung) or fails cleanly with logs (if it is the only viable rung). It never loops redoing good work.
5. Mass preemption (spot-fleet reclaim event). The cloud reclaims a big slice of spot workers at once; thousands of leases expire together. Fix: leases + idempotent key-addressed outputs mean every reclaimed task simply re-queues and costs one segment of redo each. Workers honor the preemption termination notice to NACK voluntarily (re-queue instantly, no TTL wait). Keep a baseline of on-demand workers for the priority-0 TTFV path so first-viewable latency is not hostage to spot availability, and run the heavy lazy rungs on cheap spot where redo is tolerable.
6. Blob storage / intermediate I/O amplification. Thousands of workers reading master chunks and writing encoded chunks hammer object storage; a single huge master read by many split tasks is a hot object.
Fix: split writes per-segment input chunks once (the split node) so each encode task reads a small chunk, not the whole master - the master is read once, not thousands of times. Intermediate encoded chunks are TTL’d and low-durability to cap storage. Co-locate workers in the same region as the master so chunk I/O stays on the local backbone, not cross-region.
7. State DB write pressure. Every lease, heartbeat, and node transition is a DB write; at 10^5 tasks/sec with frequent heartbeats this is heavy.
Fix: heartbeats extend the lease in the queue layer (cheap) and only periodically checkpoint to the DB, not every beat. Batch node-status writes per orchestrator tick. Shard the DB by job_id so write load spreads with the orchestrator shards. The DB sees node-level transitions (thousands/sec per shard), not raw heartbeats (which stay in the queue/cache).
8. Shard keys, stated plainly. Job State DB and orchestrator -> job_id (one DAG is one co-located, independently-owned unit; scheduling never crosses jobs). Encode task placement -> any capable worker (tasks are location-independent; the scheduler matches capabilities like gpu/av1). Blob keys -> deterministic coordinate path (job_id/segment/rung/codec), which is simultaneously the idempotency key and the placement. Sharding the control plane by video_id instead of job_id would split a re-encode job (a second job over the same video) awkwardly; job_id keeps each run self-contained. The job is the unit of orchestration; the segment coordinate is the unit of work identity.
9. Single points of failure. Orchestrator shards are stateless over the DB and fail over. The Job State DB is quorum-replicated with failover; its transactional claims prevent split-brain node ownership. The queue is replicated with a DLQ. Blob storage is erasure-coded across zones (11 nines) for the master and published output. Workers are disposable by design. No single component’s death loses an accepted job or blocks a video from eventually becoming playable - the master plus the durable DAG state make every job replayable.
10. Multi-region. Run the transcode fleet region-local to the master to avoid shipping multi-GB sources across the backbone; the control plane can be a globally-replicated NewSQL or a per-region deployment keyed by the master’s home region. Published output replicates out to wherever delivery needs it (a separate system). A regional outage pauses in-flight jobs in that region; they resume on recovery (state is durable) or fail over to another region that can read the master. Correctness never depends on one region staying up - only in-flight latency does, and that is allowed to degrade.
Wrap-Up
The trade-offs that define this design:
- Model the work as a DAG of small idempotent tasks, not a script. Naming every unit (probe, split, per-segment encode, stitch, package, QC, publish) is what buys parallelism (independent nodes fan out across the fleet), fault tolerance (retry one node), and TTFV prioritization (drain one subgraph first). A linear per-video script has none of these and was rejected because it caps a video at one machine and one failure domain.
- Treat worker death as the normal case. Leases plus heartbeats plus deterministic key-addressed outputs make a mid-job crash or a spot preemption cost exactly one ~5s segment of redo, make duplicate execution harmless (idempotent overwrite), and make a poison segment fail in isolation instead of looping. The orchestrator itself is disposable because DAG state is transactional in a durable DB, not in memory. “Retry the whole job on timeout” was rejected as catastrophic at spot-reclaim rates.
- Time-to-first-viewable is a scheduling problem, not an encode-speed problem. Prioritize the cheapest watchable rung’s subgraph, publish it partially and additively (append higher rungs to the manifest as they finish), and even pipeline the opening segments - so a video is watchable in seconds while 4K AV1 grinds for an hour. “Encode everything, then publish” was rejected because it gates the first viewable frame of the cheapest rung on the slowest, most expensive rung.
- Split only at GOP boundaries, aligned across the ladder. Keyframe-aligned segments are what make chunks independently encodable (so the DAG fans out), cleanly retryable (self-contained), and cross-rung aligned (so the manifest is coherent and ABR switches cleanly). Stitch is then a cheap ordered remux, not a re-encode. Cutting at arbitrary timestamps was rejected because interdependent frames make isolated chunks undecodable.
- Spend consistency on the control plane, not the media. The DAG state is transactionally correct and resumable (no lost or double-scheduled work); the media is immutable, cheap intermediate scratch is transient and recomputable from the master. Each layer gets the store its access pattern demands.
One-line summary: a video transcoding pipeline that probes each master, expands it into a GOP-aligned DAG of thousands of small idempotent encode tasks, schedules them across an autoscaled worker fleet with leases and heartbeats so worker death costs one segment of redo, prioritizes the cheapest watchable rung so time-to-first-viewable is seconds rather than hours, and publishes the ladder additively into immutable segments - keeping the expensive, failure-prone compute behind a transactional orchestrator that treats every worker, and itself, as disposable.
Comments