An online learning platform looks like “YouTube with a login” for about a minute, and then the differences bite. Video is only one plane of it: the actual product is a learner working through a structured course - watching a lecture, then taking a quiz that gets graded instantly, then submitting a programming assignment that runs against a test harness, then having every one of those actions bump a progress record that decides whether they have earned a certificate. The video is stateless and cacheable; everything wrapped around it is stateful, per-learner, write-heavy, and correctness-sensitive in a way a video counter never is. A wrong view count is a rounding error nobody notices. A wrong grade or a certificate issued to someone who did not finish is a support ticket and a trust problem.

And it does not arrive as a smooth stream. Real courses have scheduled start dates. When a flagship course opens, millions of learners enroll and hit lesson one in the same hour - a synchronized thundering herd on enrollment, on the first video’s cache, and on the first quiz’s grader. So the real problem is: serve video lectures cheaply from the edge, grade quizzes and assignments at spiky scale without melting the grader, track fine-grained per-learner progress as a write-heavy stream, issue certificates exactly once against auditable criteria, and absorb a coordinated enrollment stampede on course-start day. The video is the easy, solved part (I lean on the streaming design and spend the depth on the learning machinery). Let me build it properly.

Functional Requirements (FR)

In scope:

  • Enroll in a course. A learner enrolls (free audit or paid). Enrollment creates the learner-course relationship and unlocks the course content according to its schedule.
  • Stream video lectures. Adaptive-bitrate video-on-demand, played from the edge, with the player reporting playback position so progress persists across devices.
  • Take quizzes (auto-graded). In-video and standalone quizzes: multiple choice, checkbox, numeric, short answer. Graded instantly with immediate feedback, honoring attempt limits.
  • Submit assignments (auto-graded and peer-graded). Programming assignments run against a hidden test harness in a sandbox; open-ended work is peer-graded (each learner grades N peers against a rubric).
  • Track progress. Every completed lecture, passed quiz, and graded assignment updates a per-learner, per-course progress record. Resume-where-you-left-off, percent complete, and “requirements met” all read from it.
  • Issue certificates. When a learner meets the course’s completion criteria (all required items passed, minimum grade), issue a verifiable certificate exactly once, with a public verification URL.
  • Course catalog and content model. Courses contain modules, modules contain items (lecture / quiz / assignment / reading). Browse, search-by-id, and fetch the syllabus.

Explicitly out of scope (say it out loud so you own the scope):

  • The video transcode + CDN pipeline internals. Upload, segment-parallel transcoding, HLS/DASH packaging, and edge caching are a solved sub-system (see the video-streaming design); I consume its manifests and note where lecture playback hooks in, but I do not re-derive it here.
  • Recommendations and search ranking. A separate discovery service ranks the catalog; I expose the metadata it indexes.
  • Payments and billing. Enrollment can be paid; I treat the payment service as an external dependency that returns a paid/failed signal and stop there.
  • Live cohort features (discussion forums, live sessions, mentor chat). Real, but their own systems; mentioned only where they touch progress or grading.
  • Anti-cheat / proctoring depth. Plagiarism detection and remote proctoring are flagged as hook points, not designed end to end.

The decision that drives everything: this is a read-heavy delivery system (video, catalog) fused to a write-heavy, correctness-sensitive state machine (progress, grades, certificates), with a bursty, schedule-driven load profile. The video plane wants edge caching and eventual consistency; the learning plane wants durable, idempotent, auditable writes. Treating the whole thing like a video site (everything eventually consistent and cacheable) silently corrupts grades and certificates; treating it like a bank (everything strongly consistent and synchronous) cannot absorb the enrollment stampede. The design lives in getting that split right.

Non-Functional Requirements (NFR)

  • Scale: ~100M registered learners, assume ~10M daily active learners. Tens of thousands of courses. A flagship course open can enroll millions of learners in a single day and drive a peak of hundreds of thousands of enrollments and first-lesson hits per hour. Billions of progress events over the lifetime; millions of quiz submissions per day.
  • Latency: Video time-to-first-frame under ~1s p95 from a warm edge (delegated to the CDN). Catalog and syllabus reads under ~150ms p95. Quiz auto-grade under ~500ms p95 - it must feel instant. Programming-assignment grading is async (seconds to a couple of minutes) with a “grading…” state. Progress writes are fire-and-forget from the learner’s view (accepted in under ~100ms, durably persisted async).
  • Availability: 99.99% on the read/playback and catalog path - if learners cannot watch or browse, the product is down. Grading and certificate issuance can tolerate brief degradation (a submission queued for an extra minute is fine) but must never lose a submission or mis-issue a certificate.
  • Consistency: Eventual everywhere it is cheap and harmless - catalog, view counts, progress percentage as displayed. Strong / read-your-writes where correctness is visible to the learner: a quiz result must reflect the answers just submitted; a certificate must never be issued twice or issued without the criteria met. Grades are authoritative records, not analytics.
  • Durability: 11 nines on submissions (quiz answers, assignment artifacts), grades, and issued certificates - an accepted submission is never lost, and a certificate, once issued, is permanent and verifiable years later. Video masters delegate their durability to the streaming sub-system.
  • The real budget is two-sided. On the delivery side it is edge hit rate and TTFF, same as any video system. On the learning side it is write throughput on progress + grade correctness under burst. The whole design pushes video to the edge, makes progress an idempotent append-stream, keeps grading off the synchronous path where it is heavy, and treats certificate issuance as an exactly-once transaction against an auditable grade record.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Learners and baseline activity:

100M registered, ~10M DAU.
Say an active learner does one ~30-min "study session"/active day:
   ~1 lecture watched + ~1 quiz attempt + occasional assignment.
Lecture watch: 10M DAU * ~1 lecture = ~10M lecture starts/day
   10M / 86,400 ≈ ~115 lecture starts/sec steady. Peak ~5x ≈ ~575/sec.
   Each start = a manifest fetch + hundreds of segment GETs (CDN-served),
   so segment GETs are millions/sec globally - delegated to the CDN, same as YouTube.

Progress event volume (the write firehose):

Progress isn't one write per lecture - the player heartbeats position every ~10-15s
so "resume at 7:42" survives a crash, plus item-complete events.
   10M lectures/day * (~30 min / 15s) ≈ 10M * 120 = ~1.2B heartbeats/day
   1.2B / 86,400 ≈ ~14,000 progress writes/sec steady, peak ~5x ≈ ~70,000/sec.
No transactional row wants 70k targeted writes/sec on hot rows.
   => progress is an append/aggregate stream + last-write-wins state, not a hot UPDATE.

Quiz + assignment grading:

Say 3M quiz attempts/day + 300k programming-assignment submissions/day.
   Quiz: 3M / 86,400 ≈ ~35/sec steady, but SPIKY around course-start and deadlines
      -> peak easily ~1,000/sec. Auto-grade is cheap CPU (compare answers) -> fine online.
   Assignments: 300k/day ≈ ~3.5/sec steady, peak ~50/sec at a deadline.
      Each runs untrusted code in a sandbox: ~5-60s CPU each.
      50/sec * ~30s each = ~1,500 concurrent sandbox-seconds of work at peak
      => a queue-fed autoscaled grader fleet of hundreds of sandboxes, never inline.

Enrollment stampede (the schedule-driven spike):

A flagship course opens. 2M learners enroll on day one, heavily front-loaded:
   say 40% in the first 3 hours -> 800k enrollments / 3h ≈ ~75 enrollments/sec sustained,
   with minute-level peaks far higher as an email blast lands -> ~1,000/sec bursts.
Each new enrollee immediately hits lesson 1: the SAME first video and SAME first quiz.
   => one video's first segments and one quiz's grading become instant hot keys.
   The load is CORRELATED, not spread - that is what makes course-start special.

Storage:

Enrollments: 100M learners * avg ~5 courses = ~500M enrollment rows. Small, ~KB each.
Progress state: 500M enrollments * ~50 items/course = ~25B item-progress rows,
   ~100 bytes each => ~2.5 TB of progress state. Modest; the WRITE RATE is the issue,
   not the size. Heartbeats aggregate into last-position, they don't all persist forever.
Submissions: (3M quiz + 300k assignment)/day. Quiz answers ~1 KB, assignment artifacts
   up to a few MB (code, PDFs). ~300k * 2 MB = ~600 GB/day of assignment artifacts
   -> object storage. Quiz answers ~3 GB/day -> a submission log DB.
Certificates: cumulative, say ~50M issued lifetime, ~1 KB metadata each = ~50 GB. Tiny,
   but permanent and immutable - the crown-jewel append-only ledger.

The takeaways: the video/catalog read path is a millions-of-QPS CDN problem identical to any streaming site and is delegated. The new hard scale is on the write side: ~70k progress writes/sec that must not hammer hot rows, spiky grading (cheap-and-online for quizzes, heavy-and-async in sandboxes for assignments), and a correlated enrollment stampede that turns one course’s first lesson into a hot key on video, quiz, and enrollment simultaneously. Progress tracking, grading, certificate issuance, and the course-start spike are the interview.

High-Level Design (HLD)

The architecture splits along the same seam every learning platform has: a cacheable delivery plane (video lectures, catalog, syllabus - read-heavy, eventually consistent, edge-served) and a stateful learning plane (enrollment, progress, submissions, grading, certificates - write-heavy, durable, correctness-sensitive). Video is delegated to a streaming sub-system (transcode + HLS/DASH + CDN). Everything learner-stateful flows through services backed by the right store for its access pattern, with heavy or bursty work (assignment grading, progress aggregation, certificate issuance) pushed onto async queues.

   DELIVERY PLANE (read-heavy, cacheable, eventually consistent)
   ┌──────────┐   catalog/syllabus    ┌──────────────┐      ┌──────────────────┐
   │ Learner  │──────────────────────>│  Catalog Svc  │<────│  Course Content   │
   │ (web /   │   <- course tree       │  (cached,     │      │  DB (courses,     │
   │  mobile) │                        │  read-heavy)  │      │  modules, items)  │
   │          │   manifest + segments  └──────────────┘      └──────────────────┘
   │  ABR     │────────────> CDN EDGE (lecture video, delegated to streaming design)
   │  player  │<───────────── segments from nearest PoP (99%+ edge hit)
   └────┬─────┘
        │
   ─────┼───────────────────────────────────────────────────────────────────────
        │  LEARNING PLANE (write-heavy, durable, correctness-sensitive)
        ▼
   ┌──────────────┐  enroll   ┌───────────────┐        ┌──────────────────────┐
   │ API Gateway  │──────────>│ Enrollment Svc │───────>│ Enrollment DB (SQL,   │
   │ (auth, rate  │           │ (idempotent,   │        │ shard by learner_id)  │
   │  limit)      │           │  pay hook)     │        └──────────────────────┘
   └──┬───┬───┬───┘           └───────────────┘
      │   │   │
      │   │   │ progress heartbeat/complete (fire-and-forget)
      │   │   └────────────> ┌──────────────┐   ┌───────────────┐   ┌───────────────┐
      │   │                  │ Progress Gate │──>│ Event Stream   │──>│ Progress Store │
      │   │                  │ (dedup, LWW)  │   │ (Kafka, by     │   │ (KV, by        │
      │   │                  └──────────────┘   │  enrollment_id)│   │  enrollment_id)│
      │   │                                     └──────┬────────┘   └───────┬───────┘
      │   │ quiz submit (auto-grade, ONLINE)           │ item-passed events  │
      │   └──────────> ┌──────────────┐                ▼                     │
      │                │ Quiz/Grade   │──> Grade Store  ┌──────────────────┐ │
      │                │ Svc (compare │    (SQL, by     │ Completion       │<┘
      │                │  vs key)     │    enrollment)  │ Evaluator        │
      │                └──────────────┘                 │ (criteria met?)  │
      │                                                 └────────┬─────────┘
      │ assignment submit (heavy, ASYNC)                         │ criteria met
      ▼                                                          ▼
   ┌──────────────┐   ┌────────────┐   ┌──────────────┐   ┌──────────────────┐
   │ Submission   │──>│ Grade Queue │──>│ Grader Fleet  │   │ Certificate Svc   │
   │ Svc (store   │   │ (Kafka/SQS) │   │ (sandboxes,   │   │ (exactly-once     │
   │  artifact ->  │   └────────────┘   │ autoscaled)   │   │  issue, sign)     │
   │  blob store) │                     └──────┬───────┘   └────────┬─────────┘
   └──────────────┘                            │ grade                │ cert
                                        write Grade Store      Certificate Ledger
                                                              (append-only, signed)

Enrollment flow (learner joins a course):

  1. The client calls the Enrollment Service with a client-supplied idempotency key. For paid courses it first confirms payment via the external payment service; for free/audit it proceeds directly.
  2. Enrollment writes an enrollment row (learner_id, course_id, status, cohort/schedule) keyed idempotently - a retried request returns the same enrollment, never a duplicate. It seeds an empty progress record and returns the syllabus.
  3. The learner is now unlocked to the course content per its schedule (some courses gate modules by date; others are self-paced and open everything).

Lecture + progress flow (learner watches):

  1. The client fetches the syllabus from the Catalog Service (cached), then the lecture’s manifest URL, and the ABR player streams segments from the CDN edge - delivery is delegated to the streaming design.
  2. As it plays, the player fires progress heartbeats (position every ~10-15s) and an item-complete event to the Progress Gate, which dedups and lands them on the event stream. This is fire-and-forget: the UI does not block on persistence.
  3. A stream consumer folds heartbeats into a last-position (last-write-wins by timestamp) and records item completion in the Progress Store, then emits an item-passed event when a required item is completed.

Quiz flow (instant, online):

  1. The learner submits answers. The Quiz/Grade Service loads the answer key (cached), enforces attempt limits, grades by comparison, and writes an authoritative attempt record to the Grade Store. It returns the score and feedback synchronously in well under a second and emits an item-passed event if the pass threshold is met.

Assignment flow (heavy, async):

  1. The learner uploads an artifact (code, PDF). The Submission Service stores it in the blob store, writes a submission row with status grading, and enqueues a grade job.
  2. The Grader Fleet pulls the job, runs the artifact in an isolated sandbox against the hidden test harness (auto-graded) or routes it into the peer-grading queue (open-ended), writes the grade to the Grade Store, flips submission status to graded, and emits item-passed on a pass.

Certificate flow (exactly-once):

  1. Every item-passed event triggers the Completion Evaluator, which checks the course’s criteria against the Grade Store. When all required items are passed at the minimum grade, it calls the Certificate Service, which issues a signed certificate exactly once (idempotent on learner+course) into an append-only ledger and returns a public verification URL.

The key insight: the read path (video, catalog) is a cheap cacheable firehose and is delegated to the CDN; the write path (progress, grades, certificates) is where the real design lives, and each write class is routed to the treatment its correctness and volume demand - progress is a high-volume idempotent stream, quizzes are cheap online transactions, assignments are heavy async sandboxed jobs, and certificates are exactly-once ledger writes.

Component Deep Dive

The hard parts, naive-first then evolved: (1) progress tracking - a write-heavy per-learner stream that must survive 70k writes/sec without hot-row death; (2) grading - why quizzes and assignments need opposite treatments, and how to sandbox untrusted code safely at a deadline spike; (3) certificate issuance - exactly-once, verifiable, and auditable; (4) the course-start stampede - absorbing a correlated spike on enrollment, first video, and first quiz at once.

1. Progress tracking: why a hot UPDATE fails, then an idempotent stream

Naive approach: on every heartbeat and completion, UPDATE progress SET position=?, percent=? WHERE learner_id=? AND item_id=? synchronously on the request.

Where it breaks:

  • Write volume on hot rows. ~70k progress writes/sec at peak, and during a course-start the writes concentrate on the same course’s first lecture - a pile-up of updates on a narrow set of rows, each serialized behind a row lock. The hot row is the killer before total throughput even matters.
  • It couples the learner’s UI to a durable write. A heartbeat every 15s should never make the learner wait on a database round-trip; playback must not stutter because the progress DB is busy.
  • Out-of-order and duplicate events corrupt state. Mobile clients retry, buffer offline, and deliver events late. A naive last-UPDATE-wins can move the position backward when a delayed earlier heartbeat lands after a newer one.
  • Cross-device conflicts. The same learner on phone and laptop produces two streams; a blind overwrite loses one.

Evolution A: fire-and-forget through a gate onto an event stream. The client posts progress to a Progress Gate that accepts in under ~100ms (validate, dedup by event id, drop bot/replay noise) and lands the event on a Kafka stream partitioned by enrollment_id (learner+course). The UI never blocks on persistence. A stream consumer owns the fold into durable state. Now the request path is decoupled from the datastore, and all of one enrollment’s events are ordered on one partition so a single consumer task can reason about them without cross-node coordination.

Where it still breaks: we still have to fold a firehose into per-item state without moving position backward or double-counting, and without writing every one of 120 heartbeats per lecture to durable storage forever.

Evolution B: last-write-wins position + idempotent completion + aggregate-don’t-append.

Event: {enrollment_id, item_id, type: heartbeat|complete, position_s, ts, event_id}

Consumer (owns one enrollment_id partition):
  heartbeat -> update last_position for (enrollment,item) ONLY IF ts > stored ts
               (monotonic last-write-wins; a late earlier event is ignored)
            -> do NOT persist every heartbeat; keep newest position, coalesce writes
               (e.g. flush to store at most once per ~10s per item)
  complete  -> mark item completed IF not already (idempotent set, event_id dedup)
            -> emit "item-passed(enrollment,item)" exactly once

The moves that make it work:

  • Monotonic last-write-wins on position. Position is a value, not a delta - store the highest-timestamp position and ignore anything older. Out-of-order and duplicate heartbeats become harmless. Cross-device is resolved the same way (latest wins), which is exactly the “resume where you last were on any device” behavior learners expect.
  • Coalesce heartbeats; do not append them. 120 heartbeats per lecture collapse into a single evolving last_position row plus a boolean completed. Durable writes drop by ~100x versus persisting each event. The raw stream is retained short-term for replay, not as the store of record for position.
  • Completion is an idempotent set with event-id dedup. Marking an item complete twice is a no-op; the item-passed event that drives certificates is emitted exactly once per item because completion transitions once.
  • The Progress Store is a KV store keyed by enrollment_id. All of a learner’s per-item progress for a course co-locates under one key (a document / wide-column row), so “resume course,” “percent complete,” and “which required items remain” are single-key reads. Sharded by enrollment_id, writes spread across the whole keyspace instead of hot-spotting - except during a stampede, where many different learners’ first-lecture events still spread across many enrollment_ids (each learner is a distinct key), so the hot-key problem is on the video and quiz, not on progress. That is a deliberate win of keying by enrollment rather than by item.

The result: progress is accepted instantly, decoupled from durable storage, folded by a single per-enrollment consumer that cannot move backward or double-count, and coalesced so the write rate to durable storage is a fraction of the event rate. Percent-complete and item-passed fall out of the fold for free.

2. Grading: why one grader fits nobody, then split online vs sandboxed-async

Naive approach: one “grade submission” endpoint that synchronously grades everything - compare quiz answers, and for programming assignments, run the learner’s code inline in the request handler against the test harness.

Where it breaks:

  • Running untrusted code inline is a security catastrophe. A programming submission is arbitrary code. Executing it in your application process lets it read secrets, exhaust CPU/memory, fork-bomb, or exfiltrate data. It must run in a hardened, isolated sandbox with no network and strict resource limits - never in the app tier.
  • Heavy grading cannot be synchronous. A test harness can take seconds to a minute (compile, run test suites, time out infinite loops). Holding an HTTP request open that long, times a deadline spike of 50/sec, exhausts the request tier immediately.
  • Deadline spikes are correlated. Everyone submits in the last hour before a deadline. A synchronous grader has no way to shed or smooth that burst; it just falls over.
  • But making quizzes async is bad UX. A multiple-choice quiz is a cheap key comparison that should feel instant. Forcing it through a queue adds latency and a “grading…” spinner for no reason.

Evolution: route by grading cost. Quizzes online; assignments async in sandboxes; open-ended via peer grading.

Submission
  ├─ QUIZ (MCQ/numeric/short) ── ONLINE ── compare vs answer key (cached)
  │        enforce attempt limit, write attempt record, return score <500ms
  │
  ├─ PROGRAMMING ── ASYNC ── store artifact -> Grade Queue -> Grader Fleet:
  │        pull job -> spin ISOLATED SANDBOX (no net, cpu/mem/time caps,
  │        read-only rootfs, killed on timeout) -> run hidden test harness
  │        -> write grade -> status=graded -> emit item-passed on pass
  │
  └─ OPEN-ENDED (essay/project) ── PEER GRADING ──
           submission enters a peer-grade pool; each learner is assigned N peers'
           work + a rubric; grade = trimmed median of peer scores after M reviews;
           calibration submissions + staff spot-checks bound abuse.

The design:

  • Quiz grading stays synchronous and cheap. The answer key is small and cached; grading is a comparison plus rubric logic. Attempt limits are enforced with a conditional write (increment attempt count only if under the max). The attempt record is the authoritative grade. Sub-500ms, no queue.
  • Assignment grading is a queue-fed autoscaled sandbox fleet. The Submission Service persists the artifact to blob storage and writes status=grading before enqueuing, so a submission is durable the instant it is accepted even if the grader is backed up. The Grader Fleet autoscales on queue depth; each job runs in a single-use sandbox (container/microVM: no network, CPU/memory/pids/time limits, read-only root, killed on timeout) so untrusted code cannot escape or hog. Grades are idempotent per submission (re-running a job overwrites the same grade), so retries after a worker crash are safe.
  • Deadline spikes are absorbed by the queue, not the graders. A 50/sec deadline burst just deepens the queue; graders drain it at their own pace and learners see “grading, results shortly.” The queue is the shock absorber; correctness never depends on grading instantly.
  • Peer grading for the ungradeable-by-machine. Open-ended work fans into a peer-grade pool: each learner grades N peers against a rubric, the score is a trimmed median over M reviews to blunt outliers, and calibration items plus staff spot-checks bound gaming. This is its own small workflow (assignment state, reviewer assignment, aggregation) sitting behind the same submission model.
  • Grade Store is the authoritative ledger. Every attempt/grade is written to a SQL store sharded by enrollment_id (co-located with the learner-course), with the pass/fail decision derived from it. Grades are records, not analytics - strong consistency and durability, read-your-writes so a learner immediately sees the result they just earned.

The result: cheap grading stays instant and online; expensive, dangerous grading is isolated, async, autoscaled, and shock-absorbed by a queue; the ungradeable is peer-graded with abuse controls; and every grade lands in an authoritative store that certificates can trust.

3. Certificate issuance: why “issue on completion” fails, then exactly-once + verifiable ledger

Naive approach: when the last item is completed, the completion handler inserts a certificate row and emails a PDF.

Where it breaks:

  • Double issuance. The item-passed event that triggers evaluation can be delivered more than once (at-least-once streams), or two devices finish two last items near-simultaneously, or a retry fires twice. A blind insert issues two certificates - or worse, charges twice for a paid certificate.
  • Issuing against stale or partial state. If the evaluator reads a progress cache that has not caught up, it may issue before criteria are truly met, or check the wrong criteria version (a course whose passing bar changed mid-run).
  • Unverifiable certificates. A row in a DB is not a credential. Employers verify certificates years later via a public URL; the artifact must be tamper-evident and verifiable independently of a live lookup that might have changed.
  • No audit trail. “Why does this learner have a certificate?” must be answerable from records: which items, which grades, which criteria version.

Evolution: an idempotent completion transaction against the authoritative Grade Store, writing a signed, append-only certificate ledger.

On item-passed(enrollment, item):
  1. Completion Evaluator loads the course's CRITERIA (versioned) and the learner's
     GRADE STORE records (authoritative, not the progress cache).
  2. If NOT all required items passed at >= min grade -> stop.
  3. If met -> call Certificate Svc with idempotency key = hash(learner_id, course_id).
        Certificate Svc:
          - conditional insert into certificate ledger UNIQUE(learner_id, course_id)
            -> if row exists, return the EXISTING cert (exactly-once)
          - compute cert_id, snapshot {criteria_version, items, final_grade, issued_at}
          - SIGN the payload (private key) -> store signature in the ledger
          - return verification_url = /verify/{cert_id}

The moves that make it work:

  • Exactly-once via a unique constraint + idempotency key. The ledger has UNIQUE(learner_id, course_id); issuance is a conditional insert. A duplicate trigger, a replayed event, or a race collapses to one row - the second caller reads back the first certificate. No certificate is ever issued twice, and a paid certificate is charged once.
  • Evaluate against the authoritative Grade Store, not the progress cache. Progress is eventually consistent and fine for a percentage bar; issuance reads the authoritative, strongly-consistent grade records so it never issues on stale or partial state. The criteria are versioned and the certificate snapshots which version it satisfied - a mid-course rule change cannot retroactively invalidate or wrongly grant.
  • The certificate is signed and self-verifiable. The payload (learner, course, criteria version, final grade, issue date, cert_id) is cryptographically signed. Verification checks the signature, so the credential is tamper-evident and can be validated even offline; the public /verify/{cert_id} URL confirms it against the ledger for the online case.
  • Append-only, immutable ledger. Certificates are never updated or deleted - a revocation is a new appended “revoked” record referencing the original, preserving the full audit trail. This is the crown-jewel store: tiny, permanent, 11-nines durable, replicated widely.
  • Idempotent and replay-safe end to end. Because evaluation is a pure function of authoritative grades and issuance is exactly-once, the whole path can be safely re-run (event replay, reprocessing) without side effects. That is what lets the rest of the system be eventually consistent while certificates stay exactly correct.

The result: certificates are issued once, only when authoritative criteria are truly met, as signed tamper-evident credentials in an immutable auditable ledger - correctness the rest of the eventually-consistent system does not have to guarantee.

4. The course-start stampede: why uniform scaling fails, then absorb the correlated spike

Naive approach: assume load is smooth, autoscale each service on average CPU, and let enrollment write straight through to the DB.

Where it breaks:

  • The load is correlated, not smooth. When a flagship course opens (often on an email blast at a fixed hour), millions of learners hit the same endpoints in minutes: enroll, then the same first video, then the same first quiz. Average-CPU autoscaling reacts too slowly to a step-function spike and scales the wrong things uniformly.
  • Enrollment write hot spot. A burst of enrollments for one course_id concentrates writes if you shard or index naively by course_id, and duplicate submits (double-clicks, retries on a slow response) inflate it further.
  • First-lesson hot keys. Every new enrollee watches the same first video’s first segments and takes the same first quiz. That is a cache cold-start on one video and a synchronized grading spike on one quiz - textbook thundering herd on a narrow key set.
  • Downstream pile-up. The wave of first-lecture heartbeats and first-quiz item-passed events hits progress and completion evaluation in a correlated burst.

Evolution: shard away the enrollment hot spot, pre-warm the known-hot content, idempotent enrollment, and let queues absorb the rest.

Course-start playbook:
  ENROLLMENT: shard enrollment by learner_id (writes spread across all shards even
     for one course); idempotency key per (learner,course) collapses double-submits;
     pre-provision (don't rely on reactive autoscale for a KNOWN scheduled spike).
  FIRST VIDEO: PRE-WARM the first module's segments to edge PoPs before the start time
     (the spike is SCHEDULED - you know the video and the hour). Origin-shield tier
     collapses any edge misses so origin is filled once, not N times.
  FIRST QUIZ: answer key is cached and read-only; grading is cheap online compare,
     so it scales horizontally on stateless graders. Pre-scale ahead of the hour.
  DOWNSTREAM: progress + completion are queue-fed and per-enrollment partitioned, so
     the correlated event wave is buffered by Kafka and drained at consumer pace.

The design levers:

  • Shard enrollment by learner_id, not course_id. Even when a million learners join one course, each is a distinct learner_id, so enrollment writes spread uniformly across shards. Sharding by course_id would funnel the entire stampede onto one shard.
  • Idempotent enrollment. A per-(learner, course) idempotency key makes double-clicks and retries no-ops, so the effective write rate is one-per-learner, not one-per-click.
  • Pre-warm the known-hot content. The spike is scheduled - you know exactly which video and which quiz will be hot and when. Push the first module’s segments to the edge and pre-scale the stateless quiz graders before the start hour, converting a cold-start stampede into warm-cache steady serving. The origin-shield tier catches any residual misses so the origin fills each segment once.
  • Pre-provision capacity for a scheduled event. Reactive autoscaling lags a step function; for a known start time you provision ahead and let it scale down after. This is the one place you trade a little cost for not falling over at t=0.
  • Queues absorb the downstream wave. The correlated burst of first-lecture heartbeats and first-quiz passes lands on Kafka and is drained by per-enrollment consumers at their own pace - the same shock absorber that handles deadline spikes. Certificates are far downstream and see no first-day spike at all.

The result: the enrollment write spreads across shards and dedups to one-per-learner; the two genuinely hot keys (first video, first quiz) are pre-warmed and pre-scaled because the spike is scheduled and its hot set is known; and every downstream write is queue-buffered. The correlated stampede is defused by exploiting the one thing that makes it different from random load - it is predictable.

API Design & Data Schema

Most traffic is video segment GETs (CDN-served, not these APIs) and catalog reads; the APIs below handle enrollment, progress, grading, and the certificate lifecycle.

REST / RPC endpoints

# --- Catalog (read-heavy, cached) ---
GET  /api/v1/courses/{id}
  -> course tree: modules -> items (lecture|quiz|assignment|reading).
  200: { "course_id":"c_42","title":"...","modules":[
           {"module_id":"m_1","items":[
              {"item_id":"i_9","type":"lecture","manifest_url":"https://cdn/.../master.m3u8"},
              {"item_id":"i_10","type":"quiz","attempt_limit":3,"pass_pct":80} ]}]}

# --- Enrollment (idempotent) ---
POST /api/v1/enrollments
  Header: Idempotency-Key: <client-uuid>
  Body: { "course_id":"c_42" }         # payment confirmed upstream for paid courses
  -> creates (or returns existing) enrollment; seeds empty progress.
  201: { "enrollment_id":"e_7","course_id":"c_42","status":"active","schedule":"self_paced" }

# --- Progress (fire-and-forget) ---
POST /api/v1/progress
  Body: { "enrollment_id":"e_7","item_id":"i_9","type":"heartbeat",
          "position_s":462,"ts":1753600000,"event_id":"ev_113" }
  -> 202 Accepted; lands on the stream. NO synchronous DB write. LWW by ts.
GET  /api/v1/enrollments/{id}/progress
  -> 200: { "percent":54,"last_item":"i_9","last_position_s":462,
            "completed_items":["i_1",...],"required_remaining":["i_20","i_21"] }

# --- Quiz (online grading) ---
POST /api/v1/quizzes/{item_id}/attempts
  Body: { "enrollment_id":"e_7","answers":[{"q":"q1","a":"B"},...] }
  -> grades vs key, enforces attempt limit, writes authoritative attempt.
  200: { "attempt_id":"a_88","score_pct":90,"passed":true,"attempts_left":2,
         "feedback":[{"q":"q1","correct":true},...] }        # <500ms, synchronous

# --- Assignment (async grading) ---
POST /api/v1/assignments/{item_id}/submissions
  Body(multipart): artifact=<file>, enrollment_id=e_7
  -> stores artifact to blob, writes submission(status=grading), enqueues grade job.
  202: { "submission_id":"s_501","status":"grading" }
GET  /api/v1/submissions/{id}
  -> 200: { "submission_id":"s_501","status":"graded","score_pct":88,"passed":true,
            "report_url":"https://blob/.../report.json" }

# --- Certificate (exactly-once, verifiable) ---
GET  /api/v1/enrollments/{id}/certificate
  -> 200 if issued: { "cert_id":"cert_9f","final_grade":91,
                      "verification_url":"https://.../verify/cert_9f" }
  -> 409/404 if criteria not yet met.
GET  /verify/{cert_id}                     # PUBLIC, no auth
  -> 200: { "cert_id":"cert_9f","learner":"...","course":"...","final_grade":91,
            "criteria_version":3,"issued_at":"...","signature_valid":true }

Data store schemas

1. Course Content DB - SQL, read-heavy, cached, sharded by course_id. Structured, relational (course -> modules -> items), rarely written, read enormously. Front with a cache and read replicas; the syllabus is highly cacheable.

Table: courses      course_id PK, title, description, criteria_version, status
Table: modules      module_id PK, course_id FK/INDEX, ordinal, unlock_rule
Table: items        item_id PK, module_id FK/INDEX, type ENUM(lecture,quiz,assignment,reading),
                    manifest_key (lectures), pass_pct, attempt_limit, required BOOL
Table: quiz_keys    quiz_item_id PK, answers JSON, rubric JSON   -- cached, small

2. Enrollment DB - SQL, sharded by learner_id. The learner-course relationship; needs read-your-writes on the enroll commit and dedup. Sharding by learner_id spreads a single course’s stampede across all shards.

Table: enrollments
  enrollment_id  UUID    PK
  learner_id     BIGINT  SHARD KEY, INDEX
  course_id      BIGINT  INDEX (for course-level analytics)
  status         ENUM(active, completed, cancelled)
  schedule       ENUM(self_paced, cohort)   cohort_start_date  TIMESTAMP NULL
  idempotency_key TEXT   UNIQUE(learner_id, course_id)   -- dedup double-enroll
  created_at     TIMESTAMP

3. Progress Store - KV / wide-column, keyed by enrollment_id. High write, coalesced (last-position + completion set), single-key reads for resume/percent. Eventual consistency is fine here.

  Key: e_7  -> {
    course_id: c_42,
    items: { i_9:  {last_position_s:462, ts:1753600000, completed:false},
             i_10: {completed:true, completed_at:...},
             ... },
    percent: 54, updated_at: ...
  }
  # writes are LWW-by-ts, coalesced (<=1 durable write/~10s/item), event_id dedup

4. Grade Store - SQL, authoritative, sharded by enrollment_id. Attempts and grades; strong consistency, read-your-writes, durable. Certificates evaluate against this, never the progress cache.

Table: attempts        -- quiz + assignment grades
  attempt_id   UUID PK
  enrollment_id UUID  SHARD KEY, INDEX
  item_id      BIGINT INDEX
  kind         ENUM(quiz, assignment_auto, assignment_peer)
  score_pct    INT
  passed       BOOL
  attempt_no   INT     -- for attempt-limit enforcement (conditional increment)
  submission_key TEXT  -- blob key for assignment artifact/report
  graded_at    TIMESTAMP
  INDEX (enrollment_id, item_id)

5. Submission artifact + report blobs - object storage. Assignment uploads and grading reports; immutable, durable, keyed {enrollment_id}/{submission_id}/artifact. Kept for audit and re-grade.

6. Certificate Ledger - append-only, replicated, signed. The crown jewels. Tiny, permanent, 11-nines.

Table: certificates
  cert_id        UUID PK
  learner_id     BIGINT INDEX
  course_id      BIGINT INDEX
  UNIQUE(learner_id, course_id)            -- exactly-once issuance
  criteria_version INT
  final_grade    INT
  payload        JSON     -- snapshot of items/grades satisfied
  signature      BLOB     -- signed payload (tamper-evident)
  status         ENUM(issued, revoked)     -- revoke = APPEND a new record
  issued_at      TIMESTAMP

Why these choices: course content is relational, read-mostly, small - SQL with heavy caching and replicas. Enrollment is relational with a dedup/commit requirement - SQL sharded by learner_id to defuse the course stampede. Progress is write-heavy, coalesceable, tolerant of eventual consistency - a KV store keyed by enrollment_id, never a hot transactional UPDATE. Grades are authoritative records needing strong consistency - SQL sharded by enrollment_id. Certificates are exactly-once, immutable, verifiable - an append-only signed ledger with a unique constraint. Artifacts are large blobs - object storage. Each store gets the consistency and cost model its access pattern demands; the naive mistake is forcing progress and grades into the same store with the same guarantees.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Progress write firehose (breaks first under normal load). ~70k progress writes/sec, worsened by heartbeat frequency. Fix: never write per event. The Progress Gate accepts fire-and-forget onto a enrollment_id-partitioned stream; a per-enrollment consumer folds with monotonic last-write-wins and coalesces ~120 heartbeats/lecture into one evolving row (~100x fewer durable writes). event_id dedup and LWW make duplicate/out-of-order events harmless. Shard the Progress Store by enrollment_id so writes spread across the keyspace.

2. Course-start enrollment stampede (the scheduled spike). Fix: shard enrollment by learner_id (a single course’s flood spreads across all shards), idempotency-key per (learner,course) collapses double-submits, and pre-provision capacity for the known start hour rather than trusting reactive autoscale on a step function.

3. First-lesson hot keys (correlated with the stampede). Every new enrollee hits the same first video and first quiz. Fix: the spike is scheduled and its hot set is known, so pre-warm the first module’s segments to the edge and pre-scale the stateless quiz graders before start time; the origin-shield tier collapses any edge misses so origin fills each segment once (thundering-herd defense). The first quiz’s answer key is cached and read-only, so grading scales on stateless workers.

4. Assignment grading burst at deadlines (heavy + untrusted). Fix: grading is a queue-fed autoscaled sandbox fleet; the submission is persisted to blob + DB (status=grading) before enqueue, so it is durable instantly. The queue absorbs the deadline burst; graders drain at their pace behind a “grading…” state. Each job runs in a single-use isolated sandbox (no network, resource caps, killed on timeout) so untrusted code cannot escape or hog. Grades are idempotent per submission so crash-retries are safe.

5. Certificate double-issuance / mis-issuance. Fix: exactly-once via UNIQUE(learner_id, course_id) + idempotency key; evaluate against the authoritative Grade Store, not the eventually-consistent progress cache; snapshot the versioned criteria so mid-course rule changes cannot retroactively grant or revoke; sign the payload so the credential is tamper-evident; append-only ledger for a full audit trail.

6. Catalog read pressure. Every syllabus and lesson load reads course content. Fix: content is read-mostly and highly cacheable - front the Content DB with a cache (course tree, quiz keys) and read replicas; shard by course_id. Manifests themselves are served by the CDN, not the DB.

7. Hot keys, stated plainly. Enrollment -> learner_id (spreads a one-course stampede; idempotent per learner+course). Progress + Grade stores -> enrollment_id (all of one learner-course’s state co-locates; resume, percent, and grade reads are single-key; distinct learners spread across keys even for one course). Content DB -> course_id (a course and its modules/items are one read unit). Certificate ledger -> unique per (learner, course). Sharding progress by item_id would hot-spot the first lecture during a stampede; sharding enrollment by course_id would funnel the whole flood onto one shard. Keying by the learner-course unit is what makes the correlated spike spread.

8. Single points of failure. API/enrollment/quiz/submission services are stateless behind load balancers. SQL stores are quorum-replicated with failover. The Grade Store and Certificate Ledger are the durability crown jewels - replicated widely, append-only for certificates. Blob storage is erasure-coded across zones for 11-nines. Kafka is replicated; consumers checkpoint and resume, and because the fold is idempotent (LWW + event_id dedup) a replay does not corrupt progress or double-issue certificates. No single box whose loss loses a submission, a grade, or a certificate.

9. Multi-region. Video and catalog are global by construction (CDN + cached content, replicated cross-region). Learner-stateful data (enrollment, progress, grades) has a home region per learner with async cross-region replicas; a learner is served from their home region, and cross-device reads within a region are read-your-writes. Certificates replicate widely because verification is global and permanent. Grading sandboxes run region-local to the submission. Playback and catalog correctness never depend on one region; only fresh cross-region visibility of a brand-new enrollment may lag, which is acceptable.

Wrap-Up

The trade-offs that define this design:

  • Split the cacheable delivery plane from the stateful learning plane. Video and catalog are delegated to a CDN-backed, eventually-consistent read path; everything learner-stateful (progress, grades, certificates) is routed to the store and consistency model its correctness and volume demand. Treating the whole system like a video site corrupts grades; treating it like a bank cannot absorb the stampede.
  • Progress is a coalesced idempotent stream, never a hot UPDATE. Fire-and-forget onto an enrollment-partitioned stream, folded with monotonic last-write-wins and heartbeat coalescing, trades a little staleness for ~100x fewer durable writes and immunity to out-of-order, duplicate, and cross-device events. A synchronous per-heartbeat UPDATE was rejected because the hot row and the write volume both make it impossible.
  • Grade by cost, not uniformly. Cheap quiz grading stays instant and online; heavy, untrusted assignment grading is isolated in autoscaled sandboxes behind a queue that absorbs deadline spikes; the ungradeable-by-machine is peer-graded with abuse controls. Grading everything synchronously was rejected as both a security hole and a scaling dead end.
  • Certificates are exactly-once, verifiable, and auditable. Issued once via a unique constraint against the authoritative grade record, signed and tamper-evident, snapshotting versioned criteria, appended to an immutable ledger. This is the one place strong correctness is non-negotiable, and isolating it lets the rest of the system stay eventually consistent.
  • Defuse the stampede by exploiting that it is scheduled. Shard enrollment by learner_id, dedup with idempotency keys, and pre-warm/pre-scale the known-hot first lesson before the start hour, converting a correlated cold-start flood into warm steady serving; queues buffer every downstream wave.

One-line summary: an online learning platform that serves video lectures from the edge like any streaming site, but wraps them in a write-heavy stateful core - progress as a coalesced idempotent enrollment-partitioned stream, quizzes graded instantly online and assignments graded asynchronously in autoscaled sandboxes behind a shock-absorbing queue, certificates issued exactly once as signed verifiable ledger entries against authoritative grades, and the scheduled course-start stampede defused by learner-id sharding, idempotent enrollment, and pre-warming the known-hot first lesson - so the cheap cacheable reads scale on the CDN while the correctness-critical writes each get exactly the guarantees they need.