The recommendation engine is the product. Netflix does not have a search box you visit with intent; it has a home screen that must guess, before you type anything, which of ~100,000 titles you want to watch tonight - and it must guess differently for you than for the 200M other people staring at their own home screens. Get it wrong and the user scrolls for four minutes, gives up, and cancels next month. The naive framing (“show popular stuff”) dies immediately: popularity is the same for everyone, and the whole point is that it must not be. The real problem is this: for every one of 200M users, score a catalog of ~100,000 titles against everything we know about that user, return the top few dozen ranked for their taste in under ~100ms, and shift those recommendations within seconds of them finishing an episode - without re-scoring 100,000 titles per user per request, and without waiting hours for a nightly batch job to notice they just binged a new genre.

Everything hard here - candidate generation, ranking models, embedding stores, real-time feature updates - is machinery in service of two numbers: relevance (did they click and keep watching) and time-to-recommendation (how fast the home screen paints, and how fast it reacts to what you just watched). You do not “run a model”; you split the problem into a cheap step that narrows 100,000 titles to a few hundred and an expensive step that ranks those few hundred precisely, you precompute almost everything offline, and you keep a thin real-time layer that nudges the result as behavior streams in. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Personalized home screen. For a given user, produce a ranked set of titles (and the ranked rows/genres they sit in) tailored to that user’s taste. This is the core read.
  • Learn from behavior. Plays, completions, thumbs up/down, pauses, abandons, search, add-to-list, time-of-day and device - all feed the models that drive recommendations.
  • Update as the user watches. When a user finishes an episode or abandons a title mid-stream, their recommendations should reflect it within seconds-to-minutes, not the next day.
  • Candidate generation + ranking. Narrow the ~100k catalog to a few hundred plausible candidates per user, then rank those precisely into a final ordered list.
  • Cold start. A brand-new user (no watch history) and a brand-new title (no viewers yet) both get reasonable recommendations.
  • Contextual recs. Because you watch differently on a phone at lunch than on a TV at 9pm, context (device, time, current session) shifts the output.

Explicitly out of scope (say it out loud so you own the scope):

  • Video encoding, storage, and streaming delivery. That is the CDN/transcoding system; the recommender consumes its metadata and play events but does not serve bytes.
  • The training of the underlying ML models themselves as a research problem. I design the systems that train, store, serve, and update models; I treat “which loss function” as a black box and focus on the pipelines around it.
  • Search / explicit query. A separate retrieval system; recommendations are the no-query case. I note where search signals feed back in.
  • Billing, auth, profiles-vs-account plumbing. Assume a profile_id is the unit of personalization (one account, many profiles) and move on.
  • A/B testing infra and the full experimentation platform. Real and essential, but orthogonal; I flag the hook where models are swapped behind an experiment.

The one decision that drives everything: this is a read-astronomically-heavy, latency-critical personalization system where scoring the full catalog per user per request is impossible, so the work is split - expensive model training and candidate precomputation happen offline in batch, and the online path does only a cheap narrow-then-rank over a few hundred items, nudged by a thin real-time feature layer. Unlike a search system (the user hands you intent) here you must infer intent from behavior; unlike a simple cache (the answer is the same for everyone) here the answer is different for all 200M users and drifts as they watch. That asymmetry - precompute the heavy stuff offline, keep the online path to a two-stage funnel over a small candidate set, and layer real-time signals on top - is the whole problem.

Non-Functional Requirements (NFR)

  • Scale: ~200M subscribers, assume ~2-3 profiles each, so ~300-400M active profiles. Catalog ~100k titles. Hundreds of millions of home-screen loads per day; billions of interaction events per day.
  • Latency: home screen recommendations under ~100ms at p95 for the online serving path (the user is waiting, staring at a spinner). Candidate generation + ranking together must fit that budget. Batch/offline jobs have hours; the real-time update path targets seconds-to-minutes from a watch event to a shifted recommendation.
  • Availability: 99.9%+ on the read/serving path - if the home screen cannot personalize, you fall back to a non-personalized default (popular-in-your-country) rather than showing nothing. Graceful degradation is a first-class requirement, not an afterthought.
  • Consistency: eventual everywhere. A watch from ten seconds ago not yet reflected, a model that is an hour stale, a candidate list refreshed every few hours - all fine. Recommendations are inherently fuzzy; there is no “correct” answer to be strictly consistent about. The only thing that must not happen is stale-forever (a user’s behavior never changing anything).
  • Durability: raw interaction events are the crown jewels - they are the training data. An accepted event must not be lost; it lands on a durable log. Derived artifacts (embeddings, precomputed candidate lists, model files) are all regenerable from raw events + models, so they need availability, not durability.
  • The real budget is per-request scoring cost and freshness. It is acceptable for a model to be retrained nightly and for candidate lists to be refreshed every few hours; it is not acceptable to score 100k titles synchronously per request, nor to let “I just watched three horror films” take until tomorrow to register. The whole design precomputes aggressively offline and keeps a thin, cheap real-time layer for recency.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Users and read QPS (the serving path):

~300M active profiles. Say each loads/refreshes a personalized screen ~5x/day
   (open app, switch profile, scroll to a new row, come back later).
   300M * 5 = 1.5B recommendation requests/day.
   1.5B / 86,400 ≈ ~17,000 rec requests/sec (steady). Peak (prime-time, ~4x) ≈ ~70,000/sec.
Now the killer if done naively:
   70,000 req/sec * 100,000 titles to score = 7 * 10^9 title-scorings PER SECOND.
   At even a microsecond per scoring that is 7,000 CPU-seconds of work per wall second.
   => scoring the full catalog per request is physically impossible. Hence two-stage.
With candidate generation narrowing to ~500 candidates:
   70,000 * 500 = 3.5 * 10^7 scorings/sec on the RANKER. Three orders of magnitude smaller.
   That is a fleet you can actually build.

Interaction event volume (the write / training-data path):

Events: plays, pauses, seeks, thumbs, hovers, impressions, completions.
   Say each active profile generates ~200 interaction events/day (impressions dominate -
   every tile shown on the home grid is an impression event).
   300M * 200 = 6 * 10^10 = 60B events/day.
   60B / 86,400 ≈ ~700,000 events/sec steady. Peak ~3x ≈ ~2M events/sec.
No transactional DB absorbs 2M writes/sec. This is a partitioned event LOG (Kafka),
   consumed by both the real-time layer and the offline training pipeline.

Storage:

Raw events: 60B/day * ~300 bytes/event ≈ ~18 TB/day of raw interaction log.
   Retain hot (~90 days) for training features: ~1.6 PB. Older -> cold/compacted.
User embeddings: 300M profiles * a 256-dim float32 vector (256*4 = 1KB) ≈ 300 GB.
   Item embeddings: 100k titles * 1KB ≈ 100 MB (tiny; fits in RAM everywhere).
Precomputed candidate lists: 300M profiles * ~500 title_ids * 8 bytes ≈ ~1.2 TB.
   Refreshed every few hours by a batch job; a key-value store keyed by profile_id.
Model files: a handful of GB per model version; served from a model store, hot in RAM.

Bandwidth / compute for training:

Nightly (or few-hourly) retrain over ~1.6 PB of feature data on a Spark/GPU cluster.
   This is a large but BOUNDED batch job - it has hours, not milliseconds.
   The output is small: refreshed embeddings + a new ranker model file.
The expensive matrix work (embedding factorization, deep model training) is entirely
   offline. The online path never trains; it only does lookups + a light forward pass.

Cache sizing (the online hot set):

Precomputed candidate lists (~1.2 TB) + user embeddings (~300 GB) + item embeddings
   (~100 MB) + recent-behavior features. Shard the candidate/embedding stores across
   a fleet; keep item embeddings and the ranker model fully in RAM on every ranker node
   (they are small and read every request).

The takeaways: reads are ~17k-70k rec-requests/sec where scoring the full 100k catalog per request is impossible (7 billion scorings/sec) - forcing the two-stage narrow-then-rank funnel that cuts it to ~35M/sec on the ranker; writes are a ~2M-events/sec firehose that must be a partitioned log feeding both offline training and a real-time layer; storage is dominated by a ~1.6 PB hot event log (training fuel) plus small, regenerable embedding and candidate stores; and training is a big-but-bounded offline batch whose small outputs (embeddings + a model file) are what the online path serves. Those four - offline training, candidate generation, ranking, and the real-time update loop - are the interview.

High-Level Design (HLD)

The architecture splits along the fundamental seam of a recommender: the offline plane (ingest events, train models, precompute embeddings and candidate lists - expensive, batch, runs on its own schedule) is separate from the online serving plane (narrow to candidates, rank them, return - cheap, per-request, under 100ms), joined by a thin real-time layer that streams fresh behavior into features so the online path reacts within seconds. Events flow from clients into a durable log; offline jobs turn that log into models and precomputed artifacts; the online path reads those artifacts plus fresh features and produces the ranked screen.

   OFFLINE / TRAINING PLANE (batch, expensive, scheduled)
   ┌──────────┐  interaction  ┌──────────────┐   durable    ┌──────────────────┐
   │ Clients  │──events──────>│ Event Ingest │────log──────>│  Event Log        │
   │ (TV/app/ │  (plays,      │ (validate,   │              │  (Kafka, by       │
   │  web)    │  thumbs,      │  dedup, emit)│              │  profile/topic)   │
   └──────────┘  impressions) └──────────────┘              └────────┬─────────┘
                                                                     │ consume
                     ┌───────────────────────────────────────────────┤
                     ▼ (batch)                                        ▼ (stream)
            ┌──────────────────┐                          ┌────────────────────┐
            │ Feature / Training│  raw -> features         │ Real-time Feature   │
            │ Pipeline (Spark,  │  train models,           │ Updater (Flink):    │
            │ GPU): embeddings, │  factorize, learn        │ recent-watch state, │
            │ ranker model      │  ranker                  │ session features    │
            └───────┬───────────┘                          └─────────┬──────────┘
                    │ publish artifacts                              │ writes
        ┌───────────┼───────────────┬──────────────┐                 ▼
        ▼           ▼               ▼              ▼          ┌──────────────────┐
  ┌──────────┐ ┌──────────┐  ┌────────────┐ ┌──────────┐    │ Online Feature   │
  │ User/Item│ │ Candidate│  │  ANN index │ │  Ranker  │    │ Store (Redis/KV: │
  │ Embedding│ │  Lists   │  │ (vectors)  │ │  Model   │    │ recent behavior) │
  │  Store   │ │ (per prof)│ │            │ │  Store   │    └────────┬─────────┘
  └────┬─────┘ └────┬─────┘  └─────┬──────┘ └────┬─────┘             │
   ────┼───────────┼──────────────┼─────────────┼───────────────────┼──────────
   ONLINE / SERVING PLANE (per request, <100ms) │                    │
       │           │              │             │  read features     │
   ┌───▼───────────▼──────────────▼─────────────▼────────────────────▼──────┐
   │  Recommendation Service                                                  │
   │  1. CANDIDATE GEN: fetch precomputed list + ANN nearest to user emb +    │
   │     recent-session items  -> ~500 candidates                            │
   │  2. RANKER: score each candidate with the ranker model (user+item+ctx    │
   │     features) -> ordered list                                           │
   │  3. FILTER/DIVERSIFY: drop already-watched, dedup, spread genres, build  │
   │     rows -> final home screen                                           │
   └────────────────────────────────┬───────────────────────────────────────┘
                                     ▼
                            ┌──────────────┐   fallback: popular-in-country
                            │   Client      │   if any stage fails/times out
                            │ (home screen) │
                            └──────────────┘

Event / ingest flow (behavior into the system):

  1. Clients emit interaction events (play, pause, complete, abandon, thumbs, add-to-list, and crucially impressions - every tile shown) to the Event Ingest gate, which validates, dedups, and lands them on a durable partitioned log (Kafka), partitioned by profile_id.
  2. The log fans out to two consumers: the offline training pipeline (reads it in bulk for retraining) and the real-time feature updater (reads it as a stream for recency).

Offline flow (turn events into models and precomputed artifacts):

  1. The Feature/Training Pipeline (Spark + GPU) periodically reads the event log, builds training features, and trains: (a) embeddings - dense vectors for every user and every title such that “user near title” means “likely to enjoy” (matrix factorization / two-tower neural net); and (b) the ranker model - a gradient-boosted tree or deep net that scores a (user, item, context) tuple.
  2. It publishes artifacts: the User/Item Embedding Store, an ANN index over item vectors (for fast nearest-neighbor candidate lookup), a per-profile precomputed candidate list (batch-scored top-N), and the new ranker model to the model store. These are versioned and swapped atomically.

Online flow (a user opens the app, under 100ms):

  1. The Recommendation Service takes profile_id + context (device, time). Stage 1 - candidate generation: it fetches the precomputed candidate list, does an ANN lookup of items nearest the user’s embedding, and adds recent-session-based candidates - unioned and deduped to ~500 candidates. This is cheap: lookups + one vector search, no full-catalog scan.
  2. Stage 2 - ranking: it pulls per-user and real-time features from the online feature store and runs the ranker model over the ~500 candidates, producing a precise score each.
  3. Filter and diversify: drop already-watched, apply business rules, spread genres so the screen is not all one type, group into rows. Return the ranked home screen. If any stage times out, fall back to a non-personalized default.

Real-time update flow (recommendations shift as you watch):

  1. As the user watches, events stream to the real-time feature updater (Flink), which maintains recent-behavior state per profile (last N watched, current-session genres, just-completed title) in the online feature store within seconds.
  2. The next recommendation request reads those fresh features, so the ranker (and session-based candidates) reflect the just-watched content immediately - no waiting for the nightly retrain.

The key insight: all the expensive work - training, embedding, batch candidate scoring - is done offline and produces small, fast-to-read artifacts; the online path only narrows-then-ranks a few hundred items and layers on fresh session features. The offline plane makes recommendations good; the real-time layer makes them fresh; the two-stage funnel makes them fast. That separation is what lets serving hit 70k QPS under 100ms while models retrain on petabytes in the background.

Component Deep Dive

The hard parts, naive-first then evolved: (1) candidate generation - how to narrow 100k titles to a few hundred without scoring them all; (2) ranking - how to precisely order candidates for one user; (3) the offline training + embedding pipeline - how models and vectors get built and refreshed; (4) the real-time update loop - how recommendations shift within seconds of a watch. Plus cold start, which threads through all four.

1. Candidate generation: why full-catalog scoring fails, then two-stage retrieval

Naive approach: on each request, run the ranker model over all ~100,000 titles for the user, sort by score, return the top ones.

Where it breaks:

  • The arithmetic is fatal. From the BoE, 70k requests/sec * 100k titles = 7 billion model scorings per second. Even a cheap ranker is far too expensive to run 100k times per request inside a 100ms budget - a single request scoring 100k items with a real feature-rich model is already tens of milliseconds to seconds of CPU, times 70k concurrent. Impossible.
  • Most of the catalog is irrelevant to any given user. Scoring all 100k to find the ~40 you show is enormous waste; ~99.9% of titles have no chance of making the screen. You are paying to rank obvious garbage.
  • Feature fetch explodes. A good ranker needs rich per-(user,item) features; fetching and assembling those for 100k items per request would crush the feature store.

Evolution: two-stage funnel - cheap candidate generation, then expensive ranking on the survivors. Split into a fast, coarse retrieval step that narrows 100k to a few hundred using cheap operations, followed by the precise ranker (next section) over only those. The narrowing uses several complementary cheap sources, unioned:

100,000 titles
      │  candidate generation (cheap, no per-item model call)
      ▼
  ┌────────────────────────────────────────────────────────────┐
  │ (a) Precomputed batch list: top-N titles for this profile,   │
  │     scored offline nightly, read by profile_id  (~200)       │
  │ (b) ANN / embedding retrieval: k-nearest item vectors to the │
  │     user's embedding vector  (~150)                          │
  │ (c) Session / recency: "because you watched X" - items near  │
  │     the just-watched title's embedding  (~100)               │
  │ (d) Trending / popular-in-country (cold-start & diversity)   │
  │     (~50)                                                     │
  └────────────────────────────────────────────────────────────┘
      │  union + dedup + drop already-watched
      ▼
  ~500 candidates  ──> hand to the ranker

The workhorse is (b), approximate nearest neighbor (ANN) over embeddings. Every user and every title has a dense vector (from the offline pipeline) laid out so that dot-product/cosine similarity approximates “will enjoy.” Finding a user’s candidates becomes “find the item vectors nearest this user’s vector” - a vector similarity search, not a model scan. Doing this exactly (compare against all 100k) is still 100k dot products, but ANN indexes (HNSW, IVF, ScaNN) answer “approximate top-k nearest” in sub-millisecond, sub-linear time by pre-organizing the vectors into a graph/tree. Item vectors are tiny (100k * 1KB = 100MB) so the index sits in RAM.

Why multiple sources instead of just ANN: each covers a blind spot. The precomputed list (a) can use a heavier offline model than you could run online and captures long-term taste. ANN (b) is fresh to the current user embedding and finds “similar to your taste” broadly. Session (c) captures right now (“you just watched a thriller”). Trending (d) handles cold-start users (no embedding yet) and injects diversity/novelty so you are not trapped in a filter bubble. Union them, dedup, and you have a high-recall ~500-candidate set - the ranker’s job is precision on top of that recall.

Why this is correct at scale: candidate generation is cheap and precomputable. The batch list is a KV lookup; ANN is one sub-ms vector query; session candidates are one more ANN query seeded by the last-watched item. No per-item model forward pass, no 100k-item feature fetch. You have traded “perfectly score everything” for “cheaply retrieve a high-recall shortlist, then score that precisely” - the only shape that fits the latency budget.

2. Ranking: why one global order fails, then a per-user learned ranker

Naive approach: order candidates by a global score - e.g. overall popularity, or average rating - the same ranking logic for everyone.

Where it breaks:

  • It is not personalized. A globally popular action franchise ranks first for the person who only watches documentaries. The entire product premise - a different screen per user - is violated. Popularity is by definition the same for all users.
  • It ignores context. The same user wants different things on a phone at lunch vs a TV at 9pm; a static global order cannot bend to device, time, or the current session.
  • It cannot combine signals. Real relevance depends on dozens of interacting features (this user’s genre affinity, recency, completion likelihood, title freshness, time-of-day). A hand-tuned formula over a few of them is brittle and always wrong somewhere.

Evolution: a learned ranking model scoring each candidate as a (user, item, context) tuple. For each of the ~500 candidates, assemble a feature vector and run a trained model that outputs a relevance score (predicted probability the user plays and keeps watching), then sort.

For each candidate item, build features:
  USER features:   long-term genre affinities, watch-time patterns, embedding,
                   maturity/language prefs, device history
  ITEM features:   genre/tags, popularity, recency, embedding, avg completion rate
  CROSS features:  user-embedding · item-embedding, has-watched-similar,
                   past interaction with this franchise/actor
  CONTEXT features: device, time-of-day, day-of-week, current-session genres,
                    just-watched title (REAL-TIME, from the online feature store)
        │
        ▼
  Ranker model (gradient-boosted trees, or a deep net) -> P(play & engage)
        │
        ▼
  sort the ~500 candidates by score

The model is typically a gradient-boosted decision tree (GBDT, like XGBoost/LightGBM) or a deep neural ranker. GBDTs are the pragmatic default: fast to evaluate (a few hundred tree traversals per item - microseconds), robust to mixed feature types, and easy to serve. The key is that it learns the weighting of dozens of signals from actual play/abandon labels, so it captures “this user watches thrillers on weeknights but comedies on weekends” without anyone hand-coding it.

Why it fits the latency budget: it only runs over ~500 candidates, not 100k. 70k req/sec * 500 = 35M scorings/sec across the ranker fleet, each a microsecond-scale forward pass - a fleet you can size. The ranker model file is small and lives in RAM on every ranker node; item embeddings too. The only per-request I/O is fetching user + real-time features from the online feature store (one batched read), because item and cross features are largely precomputable or cheap.

Multi-objective and re-ranking. A single “P(play)” is not enough - you also care about completion, long-term retention, and diversity. Production rankers predict multiple objectives (play probability, expected watch time, thumbs-up likelihood) and combine them, then a re-ranking / diversification pass reshapes the sorted list so the screen is not ten near-identical thrillers: it enforces genre spread, novelty injection, and the row structure (“Because you watched X”, “Trending”, “New releases”). This is a business-rules + diversity layer on top of the learned scores - relevance from the model, variety from the re-ranker.

The result: a per-user, context-aware, learned ordering over a small candidate set, cheap enough to run 35M times a second, with a diversification pass so the screen feels varied rather than a monotone list of the single highest-scored genre.

3. Offline training and embeddings: why online learning fails, then batch training with fast artifacts

Naive approach: update the model live on every event - true online learning - and compute embeddings on the fly per request.

Where it breaks:

  • Training is heavy and cannot live on the request path. Factorizing a 300M-user x 100k-item interaction matrix, or training a deep net over petabytes of features, is hours of GPU-cluster work. You cannot do a fraction of that inside a 100ms serving request.
  • Per-request embedding computation is impossible. A user’s embedding depends on their whole history and the global model; recomputing it live per request would re-read gigabytes and re-run matrix math every time.
  • Naive online updates are unstable. Blindly nudging a global model on every raw event invites feedback loops, drift, and hard-to-debug regressions with no offline evaluation gate.

Evolution: train offline in batch, publish small fast-to-read artifacts, refresh on a schedule. The training pipeline is a bounded batch job that has hours; its outputs are what the online path serves.

Event Log (raw interactions, ~1.6 PB hot)
   │  (Spark) build training features: per-user histories, per-item stats,
   │  labels (played/abandoned/thumbs), negative sampling for impressions
   ▼
FEATURE STORE (offline)  ──────────────────────────────┐
   │                                                     │ (same feature
   ▼                                                     │  definitions
TRAIN (GPU cluster):                                     │  used online)
   * Embeddings: matrix factorization or a TWO-TOWER net │
     -> user vectors + item vectors (dot product ~ affinity)
   * Ranker: GBDT / deep net over the feature vectors
   │
   ▼  publish, versioned + atomically swapped
   ├─> User/Item Embedding Store (KV by id -> vector)
   ├─> ANN index rebuilt over item vectors
   ├─> Precomputed candidate lists (batch top-N per profile)
   └─> Ranker model file -> Model Store (loaded into ranker RAM)

Embeddings are the core artifact. Two common recipes:

  • Matrix factorization: factor the huge sparse user-item interaction matrix into low-rank user and item matrices; each row is an embedding. Cheap, strong baseline, captures collaborative-filtering structure (“people like you watched this”).
  • Two-tower neural net: a “user tower” maps user features -> a vector and an “item tower” maps item features -> a vector, trained so watched pairs have high dot product. More expressive, handles content features (helps cold start), and the item tower can embed a brand-new title from its metadata alone.

The train/serve feature skew trap is why the feature store is shared: the exact feature definitions used to train must be reproduced at serving time, or the model sees different inputs online than it learned on. A single feature store with one definition, materialized offline for training and online for serving, prevents skew.

Refresh cadence, layered: the heavy retrain (full re-factorization, ranker retrain) runs daily (or a few times a day). Embeddings for active users can be refreshed more often. Precomputed candidate lists refresh every few hours. None of this is real-time - and it does not need to be, because the real-time layer (next) handles recency. The offline plane is where quality comes from; it trades freshness (a model is up to a day stale) for the ability to use expensive, well-evaluated models. Every published artifact is versioned; a bad model is rolled back by pointing serving at the previous version, and new models ship behind an A/B experiment before full rollout.

4. Real-time updates: why nightly-only fails, then a streaming feature layer

Naive approach: recommendations only change when the nightly batch reruns - what you watch today shows up tomorrow.

Where it breaks:

  • The product feels dead. A user binges three episodes of a new series and their home screen does not react until tomorrow. The single most powerful signal - what you are doing right now - is ignored for hours. “Because you watched X” cannot appear until X is a day old.
  • Session intent is lost. Within one sitting a user’s intent is coherent (they are in a horror mood tonight); a batch-only system cannot exploit the current session at all.
  • Cold-start users get nothing. A brand-new user with no batch-computed artifacts stays generic until the next training run includes them.

Evolution: a thin streaming layer that updates features (not models) in near-real-time, which the ranker and session-based candidates read on the next request. The insight is that you do not need to retrain the model in real-time; you need the model’s inputs to be fresh. Keep training offline; make the features live.

Watch events ──stream──> Real-time Feature Updater (Flink), keyed by profile_id
   maintains per-profile ROLLING STATE, written to the Online Feature Store:
     * last N titles watched (and their embeddings)
     * current-session genre distribution
     * just-completed / just-abandoned title
     * short-window engagement (watch-time in last hour)
        │  (written within seconds of the event)
        ▼
   Online Feature Store (Redis / low-latency KV, keyed by profile_id)
        │
        ▼  next recommendation request reads these fresh features:
   * Ranker uses "just-watched", session genres as CONTEXT features
   * Candidate gen adds "because you watched X" = ANN nearest to X's embedding
   => the screen shifts within SECONDS of finishing an episode, no retrain needed.

Why this is the right split: models change slowly (retrained in batch), but features change fast (streamed). Finishing an episode does not require a new model - it requires the next forward pass to know you just finished it. The Flink job turns the raw event stream into a small, always-fresh per-profile feature blob; the online path reads it cheaply. This gives seconds-level responsiveness (react to the just-watched title) without any online training and its instability.

Two-speed freshness falls out naturally: fast path (streaming features, seconds) for “react to what I just watched”; slow path (batch retrain, hours-to-a-day) for “learn my evolving long-term taste and re-embed me.” The user experiences instant reactivity from the fast path while the slow path quietly improves the underlying model.

Cold start, handled across the stages:

  • New user (no history/embedding): candidate generation leans on trending/popular-in-country (d) and any onboarding signals (a few “pick titles you like” taps -> a rough initial embedding via the two-tower item tower). As they watch, the real-time layer immediately starts personalizing within the session, long before the nightly batch ever sees them.
  • New title (no viewers): the two-tower item tower embeds it from metadata (genre, cast, synopsis) alone, so it can be retrieved by content similarity to users who like similar things - content-based bootstrapping until behavioral signal accrues. Deliberate exploration (occasionally showing new titles to gather feedback) seeds the behavioral data. This is the classic explore/exploit trade-off: spend a little relevance now on exploration to avoid never-surfacing good new content.

API Design & Data Schema

Most traffic is recommendation reads and event writes; the APIs below cover the serving handshake, event ingest, and feedback. The heavy model artifacts move through internal pipelines, not these APIs.

REST / RPC endpoints

# --- Recommendation serving (the hot read) ---
GET /api/v1/recommendations?profile_id=p_88&context=tv&row_count=10
  -> the personalized home screen: ordered rows of ordered titles.
  Response 200: {
    "profile_id":"p_88",
    "model_version":"ranker_v231",          # for A/B + debugging
    "rows":[
      { "title":"Because you watched Dark",
        "items":[ {"title_id":"t_512","score":0.94}, {"title_id":"t_77","score":0.90} ] },
      { "title":"Trending Now",
        "items":[ {"title_id":"t_9","score":0.88}, ... ] }
    ]
  }
  # p95 < 100ms. On timeout/failure of any stage -> fallback rows (popular-in-country).

GET /api/v1/recommendations/similar?title_id=t_512&profile_id=p_88
  -> "more like this" - ANN nearest to t_512's embedding, ranked for the user.

# --- Interaction / event ingest (fire-and-forget, high volume) ---
POST /api/v1/events
  Body: { "profile_id":"p_88", "title_id":"t_512", "type":"play|pause|complete|
          abandon|impression|thumbs_up|thumbs_down|add_list",
          "position_s": 1320, "session_id":"s_5", "device":"tv", "ts": 1752300000 }
  -> 202 Accepted; lands on the event log. NO synchronous model/DB update.

# --- Explicit feedback (also flows as events, but a direct endpoint for the UI) ---
POST /api/v1/feedback   { "profile_id":"p_88", "title_id":"t_512", "signal":"thumbs_up" }
  -> 202; updates real-time state + is logged for training.

# --- Internal: model / artifact management (control plane, not user-facing) ---
POST /internal/models/promote  { "model_version":"ranker_v232", "traffic_pct": 5 }
  -> route 5% of ranking traffic to a new model behind an experiment.

Data store schemas

1. User / Item Embedding Store - low-latency KV (Redis / RocksDB / a vector DB), keyed by id. Read every request in candidate generation; tiny values; regenerable from the offline pipeline (needs availability, not durability). Item vectors also live fully in RAM on ranker/ANN nodes.

Key: user_emb:{profile_id}   -> [256 float32]  (1 KB)   # ~300 GB total, sharded
Key: item_emb:{title_id}     -> [256 float32]  (1 KB)   # ~100 MB total, in-RAM
   + an ANN INDEX over all item vectors (HNSW/IVF) rebuilt on each embedding refresh,
     answering top-k nearest in sub-ms.

2. Precomputed Candidate Lists - KV, keyed by profile_id. Batch-written top-N per profile every few hours; read once per request in candidate gen. Sharded by profile_id.

Key: cand:{profile_id} -> { "ver":"batch_20260712_06", "items":[ t_512, t_77, ... ] }
   (~500 title_ids * 8 bytes ≈ 4 KB/profile; ~1.2 TB total across 300M profiles)

3. Online Feature Store - low-latency KV (Redis), keyed by profile_id. Holds real-time per-profile features written by the Flink updater; read once per request (batched) by the ranker. Also serves materialized offline features for the same profile so train/serve definitions match.

Key: rtfeat:{profile_id} -> {
   "recent_watched": [ {t_512, emb_ref, completed:true}, ... ],  # last N
   "session_genres": {"horror":0.6, "thriller":0.4},
   "just_completed": "t_512",
   "watch_time_1h_s": 5400,
   "device_recent": "tv"
}   # TTL/rolling window; overwritten by the stream, seconds-fresh

4. Interaction Event Log - durable partitioned stream (Kafka), partitioned by profile_id. The source of truth and training fuel. Retained ~90 days hot for feature building, then compacted/archived to cold object storage. Not a queryable DB - it feeds the offline pipeline and the real-time updater.

Topic: interactions   (partition = hash(profile_id))
  { profile_id, title_id, type, position_s, session_id, device, ts }
  ~2M events/sec peak; replicated; replayable for retraining if feature defs change.

5. Model / Artifact Store - object storage + a version registry. Ranker model files, embedding snapshots, ANN index snapshots. Versioned; serving nodes load the active version into RAM; rollback = point at the previous version.

Why these choices: embeddings and candidate lists are read-every-request, tiny, regenerable - KV/in-RAM stores where a relational DB adds nothing and latency is everything. Real-time features are overwrite-heavy, key-point-read, ephemeral - Redis with rolling windows, not a durable transactional store. The event log is append-only, enormous, replayable - a partitioned log (Kafka), where the append-and-replay model is exactly right and SQL would be catastrophic. There is no SQL system on the hot path at all - the only relational-ish data (title metadata, profile settings) is small, cacheable, and read from a separate service. Each layer gets the store its access pattern demands: KV/vector for lookups, a log for the event firehose, object storage for big versioned artifacts - forcing one database to do all of it is the naive mistake.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Full-catalog scoring on the request path (breaks first, catastrophically). Scoring 100k titles per request is 7B scorings/sec - impossible. Fix: the entire two-stage funnel. Candidate generation narrows 100k to ~500 with cheap precomputed lists + sub-ms ANN vector search + session lookups (no per-item model call); the ranker runs the expensive model over only ~500. This cuts online work three orders of magnitude to ~35M scorings/sec, which a horizontally-scaled ranker fleet handles inside 100ms.

2. Serving fleet QPS and tail latency. 70k rec-requests/sec at peak, each doing a candidate union + ~500 ranker forward passes + feature fetch, under a p95 of 100ms. Fix: the Recommendation Service is stateless and horizontally scaled behind a load balancer; item embeddings + the ranker model are in RAM on every node (small), so the only per-request I/O is a batched feature read. Cache full home-screen results per profile for a short TTL (a user reloading within seconds can get the cached screen). Enforce per-stage timeouts with graceful fallback so a slow feature store never blows the whole request - degrade to the precomputed list or popular-in-country rather than miss the budget.

3. Event ingest write volume. ~2M events/sec at peak; no DB absorbs that. Fix: never write to a DB synchronously. Events land on a partitioned Kafka log (partition by profile_id), consumed asynchronously by the offline pipeline and the real-time updater. The write path is an append; all the heavy lifting is downstream and async.

4. Hot key / celebrity title or viral moment. A newly-dropped blockbuster is retrieved as a candidate for tens of millions of users at once; its item vector and metadata are read constantly, and its impression/play events concentrate on one Kafka partition. Fix on the read side: item embeddings and title metadata are replicated in RAM everywhere, so a hot title has no single hot node - every ranker node already holds it. Fix on the write side: a pathologically hot title_id in the event stream is handled by partitioning on profile_id (spreads events across partitions regardless of title) so no single title concentrates writes. Trending candidates are precomputed and cached, so the viral title is served from a warm list, not recomputed per request.

5. Real-time feature store pressure. Every request reads rtfeat:{profile_id}; the Flink job writes it constantly. Fix: shard the Redis fleet by profile_id; reads and writes for one profile hit one shard, spreading load uniformly across 300M keys (no natural hot key, since it is per-user). Batch the per-request read (recent features + user embedding in one round trip). The store is regenerable, so replicas need availability not durability, and a shard loss degrades to batch-only recency for those users.

6. Training pipeline cost and staleness. Retraining over ~1.6 PB is expensive and, if too slow, models drift stale. Fix: the heavy retrain is a bounded offline batch with hours to run - scale the GPU/Spark cluster to hit the daily (or few-hourly) cadence. Layer cadences: full retrain daily, active-user embeddings more often, candidate lists every few hours, and the real-time feature layer covers the seconds-level recency the batch cannot. Incremental training (warm-start from the previous model) instead of from-scratch cuts cost. Staleness is bounded and, crucially, masked by the real-time layer.

7. Feedback loop / filter bubble. The model recommends what it already thinks you like, you watch it, it learns you like it more - narrowing forever, and new/niche content never surfaces. Fix: deliberate exploration - the diversification/re-ranking pass injects novelty and occasionally surfaces under-explored titles (explore/exploit); trending and content-based candidates (source d and the two-tower item tower) inject items outside the behavioral bubble; new titles get an exploration budget so they gather feedback before they can be judged.

8. Shard keys, stated plainly. Event log -> profile_id (spreads the write firehose evenly, keeps all of one user’s events together for stream stateful processing, and dodges hot-title concentration). Embedding / candidate / real-time-feature stores -> profile_id or title_id as the natural point-lookup key (per-user and per-item reads spread uniformly across 300M/100k keys with no hot spot). ANN index -> replicated, not sharded (item vectors are 100MB, cheaper to copy everywhere than to shard). Sharding events by title_id would concentrate a viral title’s firehose on one partition; sharding by session_id would scatter a user’s history and break per-user stream state. The profile is the unit of personalization and the write key; the title is the unit of retrieval and lives replicated in RAM.

9. Single points of failure and graceful degradation. Serving, ingest, and ranker services are stateless and horizontally scaled behind load balancers. The Kafka log is replicated; Flink jobs checkpoint and resume. Embedding/candidate/feature stores are replicated (regenerable, so availability over durability). The critical resilience property is layered fallback: if the ranker times out, serve the candidate list unranked; if candidate gen fails, serve the precomputed batch list; if that is missing (cold user), serve popular-in-country. The home screen always renders something - a personalized screen is best-effort on top of a guaranteed non-personalized floor. No single component’s failure blanks the screen.

10. Multi-region. The serving plane runs region-local: embedding, candidate, and feature stores are replicated per region so a request never crosses regions for a lookup. The event log replicates cross-region into a central (or per-region) training pipeline. A user’s real-time features follow them to their nearest region via replication; a few seconds of cross-region lag on session features is acceptable per the NFR. Training can run centrally (all events) or per-region; models are published to all regions. Serving correctness never depends on any one region - a dead region’s users fail over to another with, at worst, slightly staler real-time features.

Wrap-Up

The trade-offs that define this design:

  • Two-stage funnel over full-catalog scoring. Scoring 100k titles per user per request is physically impossible, so the work splits into cheap high-recall candidate generation (precomputed lists + ANN vector search, ~500 items) and an expensive precise ranker over only the survivors - trading a slightly-imperfect candidate set for a 1000x cheaper online path that fits 100ms. Full-catalog ranking was rejected on arithmetic alone.
  • Offline training, online serving, split by cost. All the heavy work - embedding, factorization, ranker training - is done offline in bounded batch jobs that have hours, producing small fast-to-read artifacts (embeddings, candidate lists, a model file) that the online path only looks up and runs a light forward pass over. Online/real-time training was rejected as too heavy for the request path and too unstable.
  • Fresh features, not fresh models, for real-time. Recommendations shift within seconds of a watch not by retraining but by streaming behavior into a per-profile feature store that the next request reads - the model changes slowly (batch), its inputs change fast (stream). Nightly-only updates were rejected because ignoring what the user is doing right now is the biggest possible signal loss.
  • Embeddings as the universal currency. Users and titles share a vector space, so candidate retrieval is a nearest-neighbor search, “similar titles” is a vector lookup, and cold-start new titles are embedded from metadata via a two-tower model - one representation powers retrieval, similarity, and cold start.
  • Consistency spent almost nowhere; resilience spent on fallback. Everything is eventually consistent (stale models, lagging features, few-hourly candidate refresh are all fine); the durability that matters is only the raw event log (the training fuel). The engineering effort goes not into strong consistency but into layered graceful degradation so the screen always renders something personalized-or-popular, never blank.

One-line summary: a recommendation engine that trains embeddings and a ranker offline over a petabyte-scale event log, narrows 100k titles to ~500 candidates per request with precomputed lists plus sub-millisecond ANN vector search, ranks those precisely with a learned per-user context-aware model in under 100ms, and shifts the results within seconds of each watch by streaming fresh behavior into an online feature store - so the expensive learning happens offline, the online path stays a cheap narrow-then-rank funnel, and a thin real-time layer keeps 200M personalized home screens reacting to what each user just watched.