Everyone thinks a job scheduler is a while loop with a sleep. “Check the clock, if a job is due run it, sleep a second, repeat - cron already does this, it is fifty lines.” That mental model survives exactly until the interviewer asks the questions that turn it into a real system: you run three copies of the scheduler for availability, so when a job comes due, does it fire once or three times? The scheduler process was down for ten minutes during a deploy - the 2am backup job never fired, is it gone forever or does it run late? A job takes an hour and the worker running it crashes at minute 55 - does anyone notice, and does it restart from scratch or double-charge the customer? You have ten million jobs due at midnight because everyone set their cron to 0 0 * * * - does the whole system fall over at the top of the hour?

A distributed job scheduler is deceptive because the single-node cron demo works perfectly on one laptop and falls apart the moment you need availability (the scheduler cannot be a single point of failure, so you run several - and now they race), scale (millions of scheduled jobs, more than one machine can scan every second), reliability (a job that was due must eventually run even if everything was down when it came due), and correctness (a job that must not double-execute - send one email, issue one refund - has to actually run once). The whole design comes down to a few decisions that are not obvious: separating the scheduler (decides what is due) from the workers (actually run it) so you can scale execution independently, using leader election or partitioned ownership so a due job is claimed by exactly one scheduler and not fired N times, storing jobs in a durable store with a due-time index so a job survives a total outage and gets picked up late instead of lost, and being honest that exactly-once execution is not free - it is at-least-once delivery plus idempotent jobs, and you have to build the second half.

Let me do it properly.

Functional Requirements (FR)

In scope:

  • Schedule a one-off job at a future time. “Run this at 2026-07-12T02:00:00Z.” A run-once job with a fire time, a payload (what to do), and a target (which handler/queue). This is the core write.
  • Schedule a recurring job (cron). “Run this every day at 2am,” “every 5 minutes,” “the first Monday of the month.” A recurring job carries a cron expression / interval, and the scheduler computes each next fire time after the previous run.
  • Execute the job at (approximately) the right time. When a job is due, hand it to a worker that runs the associated task, with a target latency between due-time and start.
  • At-least-once execution as the baseline, with an idempotency path to effectively-once. No due job is silently dropped; a job may run more than once under failure, and the design gives job authors a clean way to make that harmless (idempotency keys).
  • Retries with backoff on failure. A job that fails (worker error, timeout, non-2xx from the target) is retried a configurable number of times with backoff, then dead-lettered.
  • Missed-run handling. If the scheduler or a partition was down when a job’s fire time passed, the job must still run when the system recovers (catch-up), subject to a policy: run every missed occurrence, run once, or skip.
  • Cancel / pause / update a job. Cancel a scheduled job before it fires; pause and resume a recurring job; change its schedule or payload.
  • Visibility. Query a job’s status and run history (scheduled, running, succeeded, failed, dead-lettered) - operators and callers need to see what happened.

Explicitly out of scope (say it so you own the scope):

  • The business logic of the jobs themselves. We deliver “it is time, run task X with payload Y” reliably and once-ish. What task X does (send an email, run a report, charge a card) lives in the caller’s handler. We own scheduling and dispatch, not the work.
  • Complex workflow orchestration / DAGs. Job B runs only after A and C succeed, fan-out/fan-in, conditional branches - that is a workflow engine (Airflow/Temporal-style) built on top of a scheduler. We provide the reliable timer and dispatch primitive; we note where a DAG layer hooks in but do not build it.
  • Sub-millisecond real-time scheduling. This is not a real-time OS scheduler or a low-latency trading timer. Target dispatch accuracy is on the order of a second or a few seconds, not microseconds.
  • Arbitrary long-running compute placement. We dispatch a job to a worker and track its lifecycle, but bin-packing heavy compute across a cluster with resource-aware placement is a scheduler-of-a-different-kind (Kubernetes/YARN). We assume workers pull work they can run.
  • Exactly-once side effects on non-cooperating external systems. We guarantee at-least-once dispatch and give you an idempotency mechanism; making a downstream call that is inherently non-idempotent happen exactly once requires that downstream to cooperate (dedup key, its own transaction). We describe the pattern but cannot make a foreign side effect exactly-once by ourselves.

The one decision that drives everything: a scheduler at scale is not one loop that both decides and runs - it is a durable store of jobs indexed by fire time, a set of scheduler nodes that partition ownership of that store so each due job is claimed exactly once, and a separate fleet of stateless workers that execute claimed jobs at-least-once. That separation - durable store, partitioned/elected ownership for the decide step, stateless scaled-out workers for the run step - is what buys availability, scale, reliability, and a clean correctness story all at once, and everything below is a consequence of it.

Non-Functional Requirements (NFR)

  • Scale: on the order of hundreds of millions of scheduled jobs live in the store, with a steady dispatch rate of tens of thousands of jobs/sec average and sharp spikes at round times (top of the minute, hour, midnight) that can be 10-100x the average. Job payloads are small (a few KB of metadata and a target reference), not the work itself.
  • Latency (dispatch accuracy): a due job starts within a p99 of a few seconds of its scheduled time under normal load, and within tens of seconds even during a top-of-the-hour spike. This is a “soon after due” system, not a hard real-time one - we optimize for never drop over never late.
  • Availability: 99.95%+ on scheduling (accepting new jobs) and on dispatch. No single node is a single point of failure. A scheduler node dying must not stall dispatch for more than the few seconds it takes to reassign its partition; a worker dying must not lose the job it held.
  • Reliability / durability: once a job is accepted, it is durable and will eventually run. A total outage means jobs run late, not never - missed fire times are caught up on recovery. Accepted jobs survive node and disk loss (replicated store). This is the property the whole design exists to protect.
  • Delivery / execution semantics: at-least-once execution by default - a due job runs one or more times, never zero. Effectively-once is available via idempotency keys and conditional state transitions. True machine-level exactly-once is acknowledged as impossible across failures; we engineer to make duplicates harmless.
  • Consistency: the job store is strongly consistent for the operations that matter - claiming a job (exactly one owner), transitioning its state, and computing the next fire time must not race. Read-your-writes on scheduling a job. Eventual consistency is fine only for the visibility/history read path.

Back-of-the-Envelope Estimation (BoE)

Let me ground the design in numbers.

Job volume and dispatch rate:

Live scheduled jobs:        200,000,000 (mix of one-off + recurring, next-fire indexed)
Average dispatch rate:      50,000 jobs/sec  (steady state)
Peak (top of minute/hour):  assume 20% of recurring jobs land on round times.
   Say 40M jobs fire "on the hour" bucket. If they all target :00:00 exactly,
   that is 40M in one second -> impossible to dispatch in 1s.
   So the spike MUST be smeared: dispatch over a 60s window -> ~670K/sec peak,
   still 13x the average. Jitter + windowed scan flatten it further (see below).

The first non-obvious number: the load is not the average, it is the spike, and the spike is self-inflicted because humans pick round times. A naive “fire everything due at exactly this instant” design dies at midnight. Design the scan to smear a due-time bucket across a window and add jitter, so 40M “midnight” jobs become a manageable stream, not a wall.

Scan / read load on the store:

Scan cadence:  every scheduler partition scans "jobs due in the next window"
               every ~1-5 seconds.
Due-soon working set: jobs due in the next 60s.
   At 50K/sec average -> 3M jobs in any 60s window (steady),
   far more in a spike window -> the scan must be an INDEXED range query
   on fire_time, never a full-table scan of 200M rows.
Index:  (partition_id, next_fire_time) so each scheduler scans only its shard's
        due-soon slice.

Storage:

Per job record: id, schedule/cron, payload ref, target, state, next_fire_time,
   attempt count, owner, timestamps ~ 1 KB packed.
200M jobs * 1 KB          = ~200 GB primary.
Replication factor 3       = ~600 GB. Trivial for a sharded store; the load is
   in the read/scan QPS and the write contention on claims, not raw bytes.

Run history: say we keep 30 days of runs, avg 50K/sec * 86,400 * 30
   = ~130B run records if we kept them all -> too much. Keep recent runs
   hot (days) and roll older history to cheap cold storage / sample it.

Worker fleet size:

Suppose the average job's dispatch+handoff work on a worker is light (it calls
   out to a target), but some jobs run for seconds to minutes on the worker.
If average in-flight duration is 2s and steady dispatch is 50K/sec:
   concurrent in-flight jobs = 50K/sec * 2s = 100,000 concurrent.
At ~500 concurrent jobs per worker (mostly IO-bound, waiting on targets):
   100,000 / 500 = ~200 workers steady.
Provision for the spike (670K/sec * 2s = 1.34M concurrent) with autoscaling
   and a queue that absorbs the burst -> workers scale on queue depth, not on
   the instantaneous fire rate.

The headline shape: hundreds of millions of durable jobs but only a few-hundred-GB store; the scarce resources are the indexed due-time scan QPS, contention on claiming a due job exactly once, and burst dispatch capacity at round times - not raw storage and not CPU. Size for the spike, make the scan an index range query, and put a queue between “it is due” and “run it” so bursts buffer instead of topple.

High-Level Design (HLD)

The system splits into four planes: an API tier that accepts and manages jobs, a durable job store (sharded, replicated) that holds every job indexed by next fire time and by id, a scheduler tier whose nodes each own a shard of the keyspace, scan their shard for due jobs, and enqueue them, and a worker tier of stateless executors that pull enqueued jobs, run them at-least-once, and report results. A task queue sits between scheduler and workers to decouple “it is due” from “it ran,” absorb spikes, and give retries a home. A coordination service (Raft/ZooKeeper-style) handles leader election and shard assignment.

   CALLERS                         ┌──────────────── COORDINATION (Raft quorum) ─────────────┐
  ┌────────┐                       │  - shard assignment: which scheduler owns which shard   │
  │ create │                       │  - leader election / node liveness (heartbeats)         │
  │ cancel │                       │  - failover: reassign a dead node's shards in seconds    │
  │ update │                       └───────────────┬──────────────────────────────────────────┘
  └───┬────┘                                       │ watch assignments
      │ 1. POST /jobs                               │
      ▼                                             ▼
 ┌─────────┐   2. write job     ┌──────────────────────────────────┐   3. each scheduler scans ITS shards:
 │  API    │──────────────────► │        JOB STORE (sharded)        │      "give me jobs in my shards
 │  tier   │  (id, schedule,    │  shard 0 | shard 1 | ... | shard K│       with next_fire_time <= now+W"
 └─────────┘   next_fire_time)  │  index: (shard, next_fire_time)   │      (indexed range query, per shard)
                                │  index: (id) for point lookups    │◄─────────────┐
                                └───────────────┬──────────────────┘              │
                                                │ 4. CLAIM due job (atomic CAS:   │
   ┌──────────── SCHEDULER TIER ────────────────┤    state scheduled->dispatched, │
   │  node S0 owns shards {0,3}                  │    set owner+lease)             │
   │  node S1 owns shards {1,4}   ── scan ───────┘                                 │
   │  node S2 owns shards {2,5}                                                     │
   │  each: scan due -> claim -> enqueue -> compute next_fire_time (if recurring) ──┘
   └───────────────┬───────────────────────────────────────────────────────────────┘
                   │ 5. enqueue task (topic per priority/type)
                   ▼
        ┌──────────────────────────┐        6. pull task, run handler / call target,
        │       TASK QUEUE          │           extend lease via heartbeat while running
        │  (durable, partitioned)   │◄──────────────────────────┐
        └───────────┬──────────────┘                            │
                    │ pull                                       │ 7. ack success  -> mark done
                    ▼                                            │    nack/timeout -> retry w/ backoff
          ┌────────────────── WORKER FLEET (stateless, autoscaled on queue depth) ──────────┐
          │   w0   w1   w2   ...   wN   -> execute job, report result, dead-letter on give-up│
          └──────────────────────────────────────────────────────────────────────────────────┘

Request flow - scheduling a job (the write path):

  1. A caller POST /jobs with a schedule (a fire time or a cron expression), a payload reference, and a target. The API validates it, computes the first next_fire_time (for cron, the next occurrence from now), and assigns a shard (shard = hash(job_id) % numShards).
  2. The API writes the job row to the job store in state scheduled, durably and replicated, indexed by (shard, next_fire_time) and by id. It acks the caller with the job id - read-your-writes, the job is now durable and will run.

Request flow - dispatching a due job (the decide path):

  1. Each scheduler node owns a set of shards (assigned by the coordination service). On a tick (every ~1s), for each shard it owns, it runs an indexed range query: “jobs in this shard with next_fire_time <= now + lookahead and state scheduled.” This reads only the due-soon slice, never the whole store.
  2. For each due job, the scheduler claims it with an atomic conditional update: state scheduled -> dispatched, owner = me, lease_until = now + T, conditioned on the row still being scheduled. The atomic compare-and-set is what makes the claim exactly-one - if two schedulers ever look at the same row (during a failover overlap), only one CAS wins.
  3. On a winning claim, the scheduler enqueues the task onto the durable task queue, and - if the job is recurring - computes the next fire time from the cron expression and writes a fresh scheduled row (or updates next_fire_time and resets state) so the next occurrence is armed. One-off jobs move to dispatched and are done being scheduled.

Request flow - executing a job (the run path):

  1. A worker pulls a task from the queue. It marks the run running, then executes the handler - typically calling the job’s target (an HTTP endpoint, an internal service, a function). While the job runs, the worker heartbeats to extend its lease so the system knows it is alive and holding the job.
  2. On success the worker acks the task and transitions the job/run to succeeded. On failure or timeout it nacks; the task is retried with backoff up to the max attempts, after which it is dead-lettered for inspection. If the worker dies mid-run, its lease expires and the task becomes visible again for another worker to pick up - this is where at-least-once comes from, and why job handlers must be idempotent.

The key architectural insight: the scheduler decides and the worker runs, and those are separate tiers connected by a durable queue. A single loop that both scans and executes cannot scale execution independently of scheduling, cannot absorb a spike (a slow job blocks the scan), and makes “run exactly once” tangled up with “who is scanning.” Splitting them lets the scheduler tier stay small and fast (just scan-and-claim), the worker tier scale out statelessly on queue depth, and the queue absorb bursts and own retries. The durable store plus the atomic claim is what makes a due job fire once despite multiple schedulers; the queue plus leases is what makes it execute at-least-once despite worker deaths.

Component Deep Dive

Four hard parts: (1) making a due job fire once, not once-per-scheduler-node - leader election and partitioned ownership; (2) executing at-least-once and getting to effectively-once - leases, retries, and idempotency; (3) never losing a due job when things were down - missed-run detection and catch-up; and (4) surviving the top-of-the-hour spike and scaling the workers. I will walk each from naive to scalable.

1. Firing Once, Not N Times: Leader Election and Partitioned Ownership

We run multiple scheduler nodes for availability. The instant we do, a job that comes due is visible to all of them, and the naive design fires it once per node.

Approach A: Every scheduler scans everything (the naive instinct)

Run 3 scheduler nodes, each scanning the whole store every second for due jobs and dispatching what it finds.

every node, every tick:
   for job in store where next_fire_time <= now and state == scheduled:
       enqueue(job)

Where it breaks: all three nodes see the same due job in the same tick and all three enqueue it - the job fires three times. For a job that sends an email or issues a refund, that is three emails and three refunds. You could try “only node with the lowest id dispatches,” but then you have one active node and two idle ones (no scale, and a failover gap), and any split-brain (two nodes both think they are lowest during a partition) reintroduces the double-fire. Scanning the whole store on every node also triples the read load for no benefit.

Approach B: Single leader (simple, but a scale ceiling)

Use the coordination service (Raft/ZooKeeper) to elect one leader among the scheduler nodes. Only the leader scans and dispatches; the others are hot standbys. If the leader dies (its lease/heartbeat lapses), the survivors elect a new one in seconds.

  • Correct on the fire-once question: exactly one node dispatches at a time, so a due job fires once. The atomic claim (below) still guards the brief overlap during a leader handover.
  • Where it strains: one leader is a throughput ceiling - a single node must scan the entire due-soon working set and enqueue all of it. At tens of thousands/sec average and hundreds of thousands/sec at spikes, one node cannot scan 200M rows’ worth of index and keep dispatch latency low. It works for small-to-medium schedulers and is worth stating as the simple baseline, but it does not reach our scale.

Approach C: Partitioned ownership - shard the keyspace, each shard has one owner

Split the job keyspace into K shards (say 256, far more than nodes, for smooth rebalancing). The coordination service assigns each shard to exactly one scheduler node (a partitioned-leader model: one leader per shard, not one leader for the whole system). Each node scans and dispatches only its own shards.

shard = hash(job_id) % 256
coordination service holds: shard -> owner_node, with leases + heartbeats.

node S0 owns shards {0, 3, 6, ...}:  scan those shards' (shard, next_fire_time) index
node S1 owns shards {1, 4, 7, ...}:  scan those
...
adding a node -> rebalance some shards to it (scale out the scan)
losing a node -> its shards reassigned to survivors within a heartbeat timeout
  • Fire-once and scale, together. Each shard has exactly one owner, so a due job is scanned and dispatched by one node - no double-fire - while the total scan work is spread across all nodes. Add scheduler nodes, get more shards-per-node parallelism, scan more jobs. This is the design that reaches our scale.
  • The claim is still atomic, as a safety net. Shard ownership can briefly overlap during failover: a node is declared dead by the coordinator but is actually alive (a GC pause, a network blip) and dispatches one more tick while the new owner also starts. To make that harmless, the dispatch is a conditional claim: UPDATE job SET state='dispatched', owner=me WHERE id=? AND state='scheduled'. Only one CAS wins; the loser sees the row already dispatched and skips. Ownership gets you efficiency (scan only your shard) and usually single-dispatch; the atomic claim gets you correctness even during the overlap. Never rely on ownership alone for correctness - fencing is what actually prevents the double-fire.
  • Fencing tokens for the truly paranoid. A zombie leader can still hold a stale claim. Attach a monotonic fencing token (epoch) to each ownership grant; the store rejects a write from an owner whose epoch is older than the latest granted for that shard. This kills the “delayed write from a node that already lost the shard” class of bug that plain leases miss.

Partitioned ownership plus an atomic conditional claim is the answer to “fire once at scale”: ownership spreads the scan and makes single-dispatch the common case, and the CAS (with fencing) guarantees exactly-one dispatch even when ownership momentarily overlaps.

2. Executing At-Least-Once and Getting to Effectively-Once

The scheduler enqueued the task exactly once. Now a worker must run it, and the worker can die mid-run. We need the job to survive that, which forces at-least-once, which forces idempotency.

Approach A: Fire-and-forget dispatch (naive)

The scheduler enqueues (or directly calls the target) and marks the job done. Whatever happens on the worker happens.

Where it breaks: if the worker crashes after pulling the task but before finishing, the job is marked done but never actually ran - silent loss, the exact failure the whole system exists to prevent. If instead we mark done only on the scheduler’s enqueue and the queue loses the message, same result. Fire-and-forget cannot promise “it ran.”

Approach B: Visibility leases + explicit ack, with retries and dead-lettering

Model execution as a lease with an explicit ack, the same pattern a durable queue uses:

worker pulls task -> task becomes INVISIBLE to other workers for lease_T seconds
worker runs the job, HEARTBEATS to extend the lease if it needs longer
  success -> ACK -> task deleted, run marked succeeded
  failure -> NACK -> task requeued, attempt++, visible again after backoff
  worker dies / lease expires with no ack -> task becomes visible again -> redelivered
  • No lost jobs. Because a task is only removed on an explicit ack, a worker death (no ack) simply means the lease expires and the task is redelivered to another worker. Nothing is dropped. This is why the system is at-least-once and not at-most-once.
  • Retries with backoff + jitter. A failing job (target returned 500, timed out) is nacked and retried after exponential backoff (base * 2^attempt) with jitter to avoid a thundering retry herd. After max_attempts it goes to a dead-letter queue for a human/alert, rather than retrying forever. State the policy explicitly: retries are bounded, and dead-lettering is a first-class outcome, not a black hole.
  • Lease length is a trade. Too short and a legitimately long job’s lease expires and it gets double-run while the first copy is still going; too long and a crashed worker’s job sits stuck until the long lease expires (slow recovery). The fix is heartbeat-extended leases: start with a modest lease and let a live worker push it out as it makes progress, so long jobs are fine and dead jobs recover fast. Guard against a job that runs forever with a hard max.

The at-least-once vs exactly-once reality

At-least-once means a job can run more than once (redelivery after a lease expiry, a retry after a timeout where the work actually succeeded but the ack was lost). True exactly-once execution across process/network failure is impossible - there is always a window where you cannot tell “did it run and I lost the ack” from “it never ran,” and you must choose to re-run (at-least-once) or not (at-most-once, risks loss). So the design commits to at-least-once and makes duplicates harmless:

  • Idempotency key per run. Each dispatch of a job carries a stable idempotency key (for a recurring job, deterministic from job_id + scheduled_fire_time; for one-off, the run id). The job handler - or a dedup layer in front of it - records “I have processed key K” and makes the second attempt a no-op that returns the first result. Now “it ran twice” becomes “it ran once and the duplicate was a no-op.” This is effectively-once: at-least-once transport plus idempotent execution.
  • Conditional state transitions. The job/run state machine only advances on a CAS: running -> succeeded succeeds once; a duplicate worker trying the same transition sees it already succeeded and drops its result. This keeps the scheduler’s view exactly-once even when the handler ran twice.
  • The honest caveat about external side effects. If the job’s effect is on a cooperating system (your own DB), the idempotency key on the write gives you effectively-once cleanly. If the effect is a non-idempotent external call (a payment API with no idempotency support), the scheduler cannot make it exactly-once alone - you need that API to accept an idempotency key, or you wrap the call in your own dedup ledger (charged(key)? check-then-act in a transaction). Say it plainly: “exactly-once execution” in practice means at-least-once dispatch plus an idempotent job, and the scheduler’s job is to supply a stable idempotency key and never drop or silently double-commit the scheduling side.

Leases with explicit acks make execution at-least-once (never lost, sometimes repeated); idempotency keys and conditional transitions turn at-least-once into effectively-once. Exactly-once is built by the job author on top of a stable key, not handed to them free by the transport.

3. Missed Runs: Never Losing a Job That Was Due While You Were Down

The scheduler tier was down (deploy, outage, a shard had no owner for two minutes). Jobs whose fire time passed during that window must not vanish.

Approach A: In-memory timer wheel (the naive fast path)

Load upcoming jobs into an in-memory timer wheel / min-heap keyed by fire time; a thread pops jobs as their time arrives and dispatches them. Fast and low-latency.

Where it breaks: it is memory-only. If the process restarts or crashes, every armed timer is gone - and worse, you do not even know which jobs you missed, because “the timer never fired” leaves no trace. A job due at 2:00:30 during a 2:00-2:03 outage is simply never dispatched. A timer wheel is a fine latency optimization on top of a durable store, but it can never be the source of truth, because a missed timer is invisible.

Approach B: Durable due-time index as source of truth + catch-up scan

Make the durable store the source of truth, with the fire time persisted per job. The scheduler’s scan is a range query next_fire_time <= now on the stored index - so a job that came due while the scheduler was down is simply still there, still scheduled, with a fire time now in the past, and the very next scan after recovery picks it up.

scan each tick:
   SELECT * FROM jobs
   WHERE shard IN (my_shards) AND state = 'scheduled'
     AND next_fire_time <= now
   ORDER BY next_fire_time
   LIMIT batch;

Because the query is <= now (not == now), it inherently catches everything overdue, whether it came due one second ago or during a ten-minute outage. Missed-run handling is not a special code path - it falls out of “scan for overdue, from a durable index.” The timer wheel from Approach A can still front this: prime the wheel from the store for the next N seconds for low-latency dispatch, but the store’s overdue scan is the backstop that guarantees nothing is lost.

The catch-up policy (this is the real design decision). When a recurring job missed several occurrences (a 30-minute outage for an every-5-minutes job = 6 missed runs), what should happen? Make it an explicit per-job policy:

Policy Behavior on recovery Use when
fire_all (backfill) Run every missed occurrence, catching up all 6 Each occurrence has independent value (e.g. per-interval data rollups you must not skip)
fire_once (coalesce) Run once now, then resume the normal schedule The job is “refresh latest state” - running it 6 times is wasteful, once is enough
skip (drop missed) Ignore missed occurrences, run only the next future one Stale runs are useless (a “send now” reminder whose time passed)
  • Guard against the backfill stampede. A fire_all policy after a long outage can dump thousands of catch-up runs at once. Bound catch-up: cap the number of backfilled occurrences (e.g. at most last N), and smear them over a window rather than firing all at the same instant.
  • A staleness deadline. Attach an optional deadline to a job (“do not run if more than X late”); an overdue scan that finds a job past its deadline dead-letters it instead of running it - useful for time-sensitive jobs where a late run is worse than no run.

The move that makes missed-runs a non-problem: persist the fire time and scan for <= now, so overdue jobs are just jobs the scan hasn’t reached yet. The only real decision left is the policy for how many missed occurrences of a recurring job to replay, which you expose per job.

4. Surviving the Spike and Scaling the Workers

Humans schedule on round times, so load is spiky, and the work runs on the worker fleet, which must scale to the burst without dropping or falling hopelessly behind.

Approach A: Dispatch everything the instant it is due, workers = fixed pool (naive)

At 00:00:00, scan finds 40M due jobs, enqueue all 40M immediately; a fixed pool of workers drains them.

Where it breaks: two failures. First, the scan and enqueue of 40M rows in one second overwhelms the store and the queue - a self-inflicted DDoS at the top of the hour. Second, a fixed worker pool sized for the average is buried by a 13-100x spike; either it falls minutes behind (blowing the latency target) or, if sized for the peak, it sits 90% idle the rest of the time (wasteful). Instantaneous dispatch to a fixed fleet cannot handle a bursty load.

Approach B: Smear the spike, queue-buffer the burst, autoscale workers on lag

Three mechanisms together tame the spike:

  • Smear the due-time bucket with a scan window + jitter. The scheduler scans a lookahead window (jobs due in the next W seconds) and dispatches them spread across that window, not all at the instant now == fire_time. Add small per-job jitter to fire times at schedule time (or at dispatch) so “everyone at exactly :00:00” becomes “spread across :00:00-:00:05.” 40M midnight jobs over a 60s smear is ~670K/sec instead of 40M in one instant - a stream the store and queue can sustain. For jobs where the exact second matters, opt them out of jitter; most jobs do not care about sub-5-second precision.
  • A durable queue absorbs the burst. Between scheduler and workers sits the task queue (partitioned, durable - a Kafka/SQS-style log). Even smeared, the burst outruns the workers briefly; the queue buffers it. Workers pull at their own sustainable rate; the queue depth rises during the spike and drains after. The queue converts a spike in dispatch into a bounded latency increase, not a dropped job. Partition the queue (by shard or job type) so pull throughput scales with worker count.
  • Autoscale workers on queue depth / lag, not on CPU. The right scaling signal is queue backlog and oldest-message age (dispatch lag), because that directly measures “are we falling behind on due jobs.” Scale the stateless worker fleet up when backlog/lag crosses a threshold and down when it drains. Because workers are stateless (they hold a job only via a lease, and a death just requeues it), scaling is safe - add or kill workers freely. Separate queues/pools by priority (a high queue for latency-critical jobs, a bulk queue for backfills) so a flood of low-priority catch-up work cannot starve time-sensitive jobs.

Worker isolation and poison jobs. One more scaling hazard: a poison job that crashes or hangs whatever worker runs it can, on redelivery, take down worker after worker. Bound it: per-job timeouts (a job that exceeds its max is killed and its lease released), an attempt cap that dead-letters after N failures (so a crash-looping job stops after N, not forever), and running job handlers in isolated execution (separate process / sandbox / container) so a hang or OOM kills one job’s slot, not the whole worker.

Smearing turns a wall of simultaneous fires into a sustainable stream; a durable queue turns the remaining burst into buffered latency instead of loss; autoscaling stateless workers on lag matches capacity to demand; and per-job timeouts + attempt caps + isolation keep one bad job from cascading through the fleet.

API Design & Data Schema

Scheduling / management API (control plane)

POST /jobs                                  # schedule a job
  { "type": "one_off" | "recurring",
    "run_at": "2026-07-12T02:00:00Z",        # one_off: absolute time
    "cron": "0 2 * * *",                      # recurring: cron expr (with timezone)
    "timezone": "Asia/Kolkata",
    "target": { "kind": "http", "url": "https://svc/internal/backup",
                "method": "POST", "headers": {...} },
    "payload_ref": "s3://payloads/abc",       # small ref, not the work itself
    "max_attempts": 5,
    "backoff": { "base_ms": 1000, "factor": 2, "max_ms": 60000 },
    "misfire_policy": "fire_once",            # fire_all | fire_once | skip
    "deadline_ms": 300000,                    # optional: skip if >5min late
    "idempotency_key": "backup-2026-07-12" }  # optional caller-supplied
  -> 201 { "job_id": "job_9f2a", "next_fire_time": "2026-07-12T02:00:00Z" }

GET    /jobs/{id}          -> job config + current state + next_fire_time
GET    /jobs/{id}/runs     -> paginated run history (scheduled/running/succeeded/failed)
PATCH  /jobs/{id}          -> update schedule/payload (recompute next_fire_time)
POST   /jobs/{id}/pause    -> recurring job stops arming next occurrences
POST   /jobs/{id}/resume   -> re-arm from now
DELETE /jobs/{id}          -> cancel (only if not already dispatched/running)

Internal worker API (data plane, usually via the queue client, not HTTP)

PULL   -> lease a task: { run_id, job_id, payload_ref, target, attempt,
                          idempotency_key, lease_until }
          task becomes invisible to others until lease_until
HEARTBEAT(run_id) -> extend lease_until (call periodically while running)
ACK(run_id, result)  -> mark run succeeded; CAS running->succeeded
NACK(run_id, error)  -> requeue with backoff; attempt++; dead-letter if attempt>max

Data model: the job store

The core is a sharded, strongly-consistent store with two access patterns: point lookup by id (for management) and an indexed range scan by (shard, next_fire_time) (for the due scan). This is the schema that matters.

jobs (one row per scheduled job):

job_id            STRING   PRIMARY KEY
shard             INT      # hash(job_id) % numShards; owned by one scheduler
type              ENUM     # one_off | recurring
cron              STRING   # for recurring; null for one_off
timezone          STRING
next_fire_time    TIMESTAMP  # THE hot column - drives the due scan
state             ENUM     # scheduled | dispatched | paused | cancelled | completed
target            JSON
payload_ref       STRING
max_attempts      INT
backoff           JSON
misfire_policy    ENUM     # fire_all | fire_once | skip
deadline_ms       INT      # nullable
idempotency_base  STRING   # for deriving per-run keys
version           BIGINT   # optimistic-concurrency / fencing on updates
created_at, updated_at TIMESTAMP

INDEX idx_due (shard, next_fire_time, state)   # the due-scan index (critical)
INDEX idx_id  (job_id)                          # point lookups

job_runs (one row per execution attempt, the history + dedup ledger):

run_id            STRING   PRIMARY KEY   # also the idempotency key surface
job_id            STRING   INDEX
scheduled_time    TIMESTAMP              # the fire time this run is for
idempotency_key   STRING   UNIQUE        # (job_id + scheduled_time) -> dedup
state             ENUM     # queued | running | succeeded | failed | dead_letter
attempt           INT
owner_worker      STRING
lease_until       TIMESTAMP
started_at, finished_at TIMESTAMP
error             TEXT

UNIQUE (idempotency_key)   # a second run of the same occurrence is rejected here

Coordination state (in the Raft/ZooKeeper quorum, not the job store):

shard_assignment:  shard_id -> { owner_node, epoch (fencing token), lease_until }
node_registry:     node_id  -> { host, last_heartbeat, status }

SQL vs NoSQL - the choice and why. The job store wants strong consistency (the claim CAS and state transitions must not race), secondary indexes (the (shard, next_fire_time) range scan), and horizontal sharding (hundreds of millions of rows, high scan QPS). That points to either a sharded relational store (Postgres/MySQL sharded by shard, or a NewSQL system like CockroachDB/Spanner) or a strongly-consistent distributed store with secondary indexes (DynamoDB with a GSI on (shard, next_fire_time), Cassandra with care). The deciding factor is the atomic claim: you need a real conditional-write (UPDATE ... WHERE state='scheduled' / a DynamoDB conditional expression / a Postgres SELECT ... FOR UPDATE SKIP LOCKED). A store without a strong conditional write cannot safely claim a job, so an eventually-consistent NoSQL store is the wrong pick for the job/claim data. Run history (job_runs) is append-mostly and can live in a cheaper, weaker store or roll to cold storage - it does not need the same guarantees.

SELECT ... FOR UPDATE SKIP LOCKED deserves a callout: on a relational store it is a clean, battle-tested way to have multiple scheduler workers pull disjoint due jobs concurrently without double-claiming, because each locks and skips rows another has already grabbed. It is often the simplest correct claim mechanism at moderate scale, before you need shard-partitioned ownership.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. The due scan turns into a full-table scan. With 200M jobs, scanning for “what is due” without the right index reads everything every tick and melts the store. Fix: the (shard, next_fire_time, state) index so each scheduler runs a cheap range query over only its shards’ due-soon slice. The scan reads thousands of rows, not hundreds of millions. This index is the single most important performance decision.

2. A due job fires N times (multiple schedulers). Availability requires multiple scheduler nodes; naively they all dispatch the same job. Fix: partitioned ownership (one owner per shard via the coordination service) so each job is scanned by one node, plus an atomic conditional claim (state scheduled -> dispatched CAS) with a fencing token as the correctness backstop during failover overlap. Ownership for efficiency, CAS+fencing for correctness.

3. The single scheduler leader is a throughput ceiling. One node cannot scan the whole due working set at spike. Fix: shard the keyspace (K shards » nodes) and spread ownership across all scheduler nodes; scan work scales with node count. Rebalance shards on membership change.

4. Top-of-the-hour spike (self-inflicted DDoS). Millions of jobs at exactly :00:00 overwhelm scan, store, and queue. Fix: scan a lookahead window and smear dispatch across it, add jitter to fire times so round-time jobs spread over a few seconds, and put a durable queue between scheduler and workers to buffer the residual burst into bounded latency rather than loss.

5. Missed runs during an outage. A memory-only timer loses everything on restart and cannot even tell what it missed. Fix: durable fire-time index as source of truth, scan for next_fire_time <= now so overdue jobs are caught automatically; per-job misfire policy (fire_all / fire_once / skip) with a cap on backfilled occurrences and a smear so catch-up does not stampede.

6. Worker dies mid-job / job lost. A crashed worker holding a long job. Fix: visibility leases with heartbeats + explicit ack - no ack means the lease expires and the task is redelivered (at-least-once); heartbeats let long jobs extend their lease so they are not double-run, and a hard max stops a runaway. Requeue-on-expiry is the no-loss guarantee.

7. Duplicate executions. Redelivery and lost-ack retries mean a job can run twice. Fix: idempotency keys (job_id + scheduled_time) with a unique constraint / dedup ledger so the second execution is a no-op, plus conditional state transitions (running -> succeeded CAS) so the scheduler’s view stays exactly-once. Effectively-once = at-least-once + idempotent job.

8. Worker fleet can’t keep up / is idle. Fixed pool is buried at spike, wasteful at trough. Fix: autoscale stateless workers on queue depth and oldest-message age (dispatch lag), the signal that directly measures falling behind; priority queues/pools so backfill/bulk work cannot starve latency-critical jobs.

9. Poison job cascades. A job that crashes or hangs every worker it touches. Fix: per-job timeouts (kill and release), attempt caps -> dead-letter (stop after N, not forever), and isolated execution (sandbox/process/container per job) so one bad job kills a slot, not a worker.

10. Coordination service as a single point of failure. If leader election / shard assignment dies, no failover can happen. Fix: run coordination as a replicated Raft quorum (3 or 5 nodes) that tolerates a minority failure and elects its own leader; keep its state tiny (shard assignments, node liveness, epochs) so it stays fast and available. Schedulers cache their current assignment so a brief coordinator blip does not stop in-flight dispatch.

11. Hot shard. One shard’s jobs disproportionately due at once, hotspotting its owner. Fix: enough shards that any one is a small slice; hash by job_id (not by a skew-prone attribute) so due-times spread; if a shard is persistently hot, split it and reassign. The smear/jitter also flattens intra-shard spikes.

12. Run-history storage explosion. Tens of thousands of runs/sec times 30 days is enormous. Fix: keep recent runs hot, roll older history to cheap cold storage, sample or aggregate high-volume job histories, and TTL the dedup ledger entries once a run’s occurrence is safely in the past (the idempotency key only needs to outlive the redelivery window).

Wrap-Up

The trade-offs that define this design:

  • Decide and run are separate tiers. A small scheduler tier scans a durable store and claims due jobs; a large, stateless worker fleet executes them, connected by a durable queue. We trade the simplicity of one loop for independent scaling, spike absorption, and a clean place to put retries. This separation is the whole design.
  • Ownership for efficiency, atomic claim for correctness. Partitioned shard ownership spreads the scan and makes single-dispatch the common case, but the atomic conditional claim (with a fencing token) is what actually guarantees a due job fires once, even when ownership overlaps during failover. Never trust ownership alone for correctness.
  • At-least-once is the honest baseline; effectively-once is built on top. Leases with explicit acks mean a due job is never lost and sometimes repeated; idempotency keys and conditional transitions make the repeat harmless. True machine-level exactly-once across failure is impossible, so we supply a stable idempotency key and let the job dedup - “exactly-once execution” is at-least-once dispatch plus an idempotent job.
  • Missed runs are not a special case. Persisting the fire time and scanning for <= now makes overdue jobs just jobs the scan hasn’t reached; the only real decision is the per-job misfire policy (fire all missed, fire once, or skip), bounded so catch-up does not stampede.
  • The spike is the load, and it is self-inflicted. Humans schedule on round times, so we smear the due-time bucket with a lookahead window and jitter, buffer the residual burst in a durable queue, and autoscale stateless workers on dispatch lag - turning a wall of simultaneous fires into bounded latency instead of dropped jobs.

One-line summary: a durable, sharded job store indexed by fire time, a scheduler tier that partitions shard ownership and atomically claims each due job exactly once (fencing-guarded), a durable queue that decouples and buffers dispatch, and a stateless autoscaled worker fleet that executes at-least-once under heartbeat-extended leases with retries and dead-lettering - where missed runs fall out of scanning for overdue and effectively-once falls out of a stable idempotency key on an inherently at-least-once execution path.