Everyone thinks PagerDuty is a glorified SMS sender. “A monitoring tool posts an alert, you look up who is on call, and you text them. If they do not reply, you text the next person.” Then the interviewer starts asking the questions that turn it into a real system: how do you know who is on call right now when the schedule is a weekly rotation with a holiday override layered on top and the engineer is in a different timezone? When you say “escalate if not acknowledged within 5 minutes,” where does that 5-minute timer live so that it still fires if the machine holding it crashes? If the same outage trips 400 alerts in ten seconds, how do you page one human once instead of 400 times? And the thing that actually matters: the entire product is a promise that when production breaks at 3am, exactly one awake human gets woken up and, if they do not answer, the timer does not silently die - it escalates. A dropped page is the one bug this system is not allowed to have.

The naive “look up the on-call person and send an SMS” demo works for one team with a fixed rotation and falls apart the moment you have real schedules and real failure modes. The whole design comes down to four hard problems: resolving who is on call at an instant through layered rotations, overrides, and holidays across timezones; running millions of durable escalation timers that survive process death and fire within a second of their deadline; delivering notifications reliably across phone, SMS, and Slack with retries, provider failover, and acknowledgement tracking; and deduplicating and grouping a storm of alerts so one incident is one page. All of it while the system stays more available than the services it watches, because a monitoring tool that is down during an outage is worse than useless.

Let me do it properly.

Functional Requirements (FR)

In scope:

  • Ingest alerts (events). Accept events from monitoring tools (Datadog, Prometheus Alertmanager, CloudWatch, a custom POST /events). Each event has a dedup_key, a service, a severity, and a payload. A trigger event opens or updates an incident; a resolve event closes one. This is the write firehose.
  • Deduplicate into incidents. Many events map to one incident. A flapping check that fires 200 times, or 400 hosts all reporting the same failed dependency, must collapse into a single incident so a human is paged once, not 400 times.
  • Route via escalation policies. Each service has an escalation policy: an ordered list of levels, each level targeting one or more on-call schedules or specific users, with a timeout (“if not acknowledged in N minutes, go to the next level”). Routing an incident means walking this policy.
  • Resolve on-call schedules. A schedule answers “who is on call for this rotation right now.” Schedules support recurring rotations (weekly, daily, custom handoff times), multiple layers stacked with precedence, overrides (cover for a colleague for 3 hours), and holidays / restrictions (no paging weekends on the secondary layer). Timezone-correct.
  • Notify across channels. Deliver the page over phone call, SMS, push notification, email, and Slack, in a per-user configurable order with its own delays (“push immediately, then SMS after 1 min, then a phone call after 3 min”). Track delivery and acknowledgement.
  • Escalate on non-acknowledgement. If nobody acknowledges within the level’s timeout, escalate to the next level. If the policy is exhausted, loop or notify a fallback. This is the core behavior - durable, reliable escalation timers.
  • Acknowledge / resolve / reassign. A responder can ack (stops escalation, “I have got it”), resolve (closes the incident), reassign, or snooze. State transitions drive the escalation loop.

Explicitly out of scope (say it so you own the scope):

  • The monitoring / alerting engine itself. Deciding that “error rate > 1% for 5m” is breached is the job of the metrics system upstream; we consume the alert it emits. (That is the handoff the metrics and monitoring design ends on - it fires the alert, we route and escalate it.)
  • Rich incident management / postmortems. Timeline reconstruction, postmortem docs, stakeholder status pages, and analytics are a large product surface on top. We build the paging core and note where they attach.
  • ChatOps automation / runbooks. Auto-remediation and runbook execution are downstream integrations. We deliver the page and record the response.
  • Billing, org/team management, SSO. Standard SaaS plumbing, not the interesting part of this system.

The one decision that drives everything: this is a low-throughput, extremely-high-reliability system whose defining feature is a durable timer that must fire even when the machine holding it dies, wired to an idempotent notification pipeline and a correct point-in-time schedule resolver. Volume is trivial (thousands of pages per minute globally, not millions per second) - the entire difficulty is that losing or delaying a single escalation is a production incident for a customer. So the design is dominated by durable scheduled work, exactly-once-ish delivery semantics, and correct temporal schedule math, not by raw scale.

Non-Functional Requirements (NFR)

  • Scale: modest throughput, huge fan-in reliability. Say 50,000 customer organizations, millions of monitored services, and an event ingest rate on the order of 5,000-20,000 events/sec globally, spiking during large outages (a cloud region goes down and every customer’s alerts fire at once - the load is anti-correlated with calm). Actual pages sent are far fewer after dedup - low thousands per minute. At any moment there may be hundreds of thousands to low millions of pending escalation timers waiting to fire.
  • Latency: the number that matters is event-to-first-notification p99 under ~5 seconds - when production breaks, the page should be on its way in seconds. Escalation timers must fire within ~1 second of their deadline - a “5 minute” timeout that fires at 5m40s erodes trust and, worse, delays getting a human on a live incident.
  • Availability: target 99.95%+ on the ingest and escalation paths, higher than most of the services being monitored. The governing principle: the system that tells you production is down must be more reliable than production. If we are down, outages go unnoticed and unpaged - the worst failure mode in the product.
  • Consistency: the escalation state machine needs strong consistency on incident state - two nodes must not both decide to escalate the same incident, and an ack must reliably stop escalation. This is the opposite of the monitoring system’s eventual-consistency relaxation: here a race means either a double-page or, catastrophically, a missed escalation. Schedule edits can be eventually consistent (a rotation change propagating in a second is fine).
  • Durability: incidents, escalation state, and pending timers must be durable and survive node and datacenter failure. A timer lost to a crash is a page that never escalates - unacceptable. Everything on the critical path is persisted before it is acted on.
  • Delivery reliability is a first-class NFR. A page must be delivered even if a telephony provider is degraded (failover across providers), and must be de-duplicated per incident so a retry never double-pages. The system tracks per-notification delivery state end to end.

Back-of-the-Envelope Estimation (BoE)

Let me ground it in numbers - and the point of this BoE is that throughput is small, so the sizing driver is timers and reliability, not QPS.

Traffic:

Customer orgs:                50,000
Monitored services:           ~2,000,000
Raw events/sec (steady):      ~10,000/sec
Raw events/sec (outage spike):~50,000/sec   (region down -> everyone fires)

After deduplication:
Open incidents at any time:   ~100,000 - 500,000
New incidents/sec:            ~200-500/sec   (most events fold into existing incidents)
Pages actually delivered:     low thousands/minute

Ingest is a few tens of thousands of events per second at peak - a single well-tuned service tier handles that. Dedup collapses the firehose by roughly 20-50x into actual incidents, because during a real outage the vast majority of events are duplicates of a handful of root causes. The scale challenge is not QPS; it is the state and the timers behind those incidents.

Pending timers - the real sizing driver:

Open incidents:                       ~500,000 (peak)
Escalation timers per incident:       1 active at a time (current level timeout)
  + per-user notification-step timers (push now, SMS +1m, call +3m)
    -> up to ~3-5 scheduled notifications per active target
Pending scheduled jobs at peak:       ~1-3 million timers waiting to fire
Timer firing precision required:      within ~1 second of deadline

This is the crux: at peak we hold on the order of a million-plus durable timers, each of which must fire within a second of its deadline and survive a process crash. You cannot do this with in-memory setTimeout (dies with the process) or a naive SELECT ... WHERE fire_at <= now() polled every minute (imprecise, and a thundering scan). The timer store is the component to engineer.

Storage:

Incident record:        ~2 KB (state, refs, dedup_key, timeline pointer)
Incident events log:    ~20 events/incident * ~0.5 KB = ~10 KB/incident
Incidents/day:          ~500/sec * 86,400 ≈ 43M/day (peak-ish); steady far less
   say ~10M incidents/day * 12 KB ≈ 120 GB/day of incident + event data
Retention:              hot 90 days, then archive to object storage
Schedules/policies:     tiny - 50K orgs * a few KB each ≈ single-digit GB, cache-resident

Incident and event-log data is the bulk of storage and it is append-heavy and time-ordered - a good fit for a partitioned store with TTL. Schedules and escalation policies are tiny and read on every routing decision, so they live in cache and a relational store, not a scan.

Notification fan-out and third-party cost:

Pages/minute:            ~2,000 (steady) -> ~10,000 during a big outage
Channels per page:       push + SMS + voice + Slack, retried
Outbound provider calls: ~4-6 per escalation step
   -> ~10,000 pages/min * 5 ≈ 50,000 provider API calls/min at peak

Notification volume is low in absolute terms but each call hits a third-party provider (Twilio for SMS/voice, APNs/FCM for push, Slack API) that has its own rate limits, latency, and outages. The design must treat providers as unreliable dependencies: queue, retry, and fail over. The scarce resources are durable-timer throughput, strongly-consistent incident state, and provider reliability - not CPU or bandwidth.

High-Level Design (HLD)

The system is an event-ingestion front end that deduplicates events into incidents, an escalation engine that walks policies driven by a durable timer service, a schedule resolver that answers “who is on call now,” and a notification pipeline that fans out to channels through unreliable providers and feeds acknowledgements back in.

        EVENT INGEST (monitoring tools, ~10-50k/sec, must not be lost)
  ┌──────────────────────────────────────────────────────────────────────┐
  │  Datadog / Prometheus / CloudWatch / API  --POST /events-->  Gateway   │
  └───────────────────────────────────────┬────────────────────────────────┘
                                           │  validated event
                                 ┌─────────▼──────────┐
                                 │  Ingest Service    │  (stateless, fleet)
                                 │  - auth + validate │
                                 │  - compute dedup   │
                                 │    key -> incident │
                                 └─────────┬──────────┘
                                           │ append (durable log, ordered per service)
                            ┌──────────────▼───────────────┐
                            │   KAFKA (event log)           │  absorbs spikes,
                            │   partitioned by service_id   │  ordered per incident
                            └───┬───────────────────────┬───┘
                                │                       │
                  ┌─────────────▼─────────┐   ┌─────────▼──────────────┐
                  │  Dedup / Incident Svc  │   │  Incident Store (SQL)  │
                  │  - upsert incident by  │◄──┤  incidents, state,     │
                  │    dedup_key (idempot.)│   │  strongly consistent   │
                  │  - trigger/ack/resolve │   └────────────────────────┘
                  └───────────┬───────────┘
                              │ "new incident -> route it"
                  ┌───────────▼────────────────────────────────────────┐
                  │            ESCALATION ENGINE                        │
                  │  - load escalation policy for the service           │
                  │  - resolve current level -> targets                 │
                  │  - ask Schedule Resolver "who is on call now?"      │
                  │  - enqueue notifications; ARM escalation timer      │
                  └───┬──────────────────────────┬─────────────────────┘
                      │ "fire at T+Nmin"          │ notify targets
          ┌───────────▼───────────┐   ┌───────────▼───────────────────┐
          │  DURABLE TIMER SVC    │   │   Schedule Resolver           │
          │  - persisted timers   │   │  - rotations + layers         │
          │  - fires within ~1s   │   │  - overrides + holidays       │
          │  - on fire -> callback│   │  - timezone-correct, cached   │
          │    escalation engine  │   └───────────────────────────────┘
          └───────────┬───────────┘
                      │ timer fired: not acked -> next level
          ┌───────────▼───────────────────────────────────────────────┐
          │            NOTIFICATION PIPELINE                           │
          │  per-user contact rules -> channel steps (push,SMS,call)   │
          │  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐           │
          │  │ Push    │ │ SMS     │ │ Voice   │ │ Slack   │  workers  │
          │  │ (APNs/  │ │(Twilio) │ │(Twilio) │ │ (API)   │  retry +  │
          │  │  FCM)   │ │         │ │         │ │         │  failover │
          │  └─────────┘ └─────────┘ └─────────┘ └─────────┘           │
          └───────────┬────────────────────────────────▲───────────────┘
                      │ delivered                        │ ACK (reply "4",
                      ▼                                   │  press 4, Slack btn,
              on-call engineer's phone  ─── acknowledges ─┘  app tap)
                                                          │
                                        ┌─────────────────▼──────────┐
                                        │  ACK Ingest -> Incident Svc │
                                        │  ack -> CANCEL escalation   │
                                        │         timer, stop paging  │
                                        └────────────────────────────┘

Request flow - an event becoming a page:

  1. A monitoring tool POSTs an event to the Ingest Gateway. The stateless Ingest Service authenticates the integration key, validates the payload, and computes the dedup key (from the event’s dedup_key/alert_key field, or a hash of service + summary). It appends the event to a durable log (Kafka) partitioned by service_id so all events for one incident are ordered on one partition. It acks the producer fast - the event is now durable.
  2. The Dedup / Incident Service consumes the log and does an idempotent upsert keyed by (service_id, dedup_key): if an open incident with that key exists, this event is folded in (append to its event log, maybe re-trigger); if not, a new incident is created in the strongly-consistent Incident Store. Only a genuinely new incident proceeds to routing - this is where the 20-50x collapse happens.
  3. For a new incident, the Escalation Engine loads the service’s escalation policy, takes level 0, and for each target that is a schedule it calls the Schedule Resolver: “who is on call for schedule S at this instant?” The resolver evaluates rotations, layers, overrides, and holidays and returns concrete user IDs.
  4. The engine enqueues notifications for those users into the Notification Pipeline and, critically, arms a durable escalation timer in the Timer Service set to fire at now + level.timeout. The incident is now triggered, level 0, waiting.
  5. The Notification Pipeline looks up each user’s contact rules (their personal channel order and delays) and dispatches through channel workers to the providers, tracking delivery state and retrying with failover.
  6. If the engineer acknowledges (replies to the SMS, presses a digit on the call, taps the push, clicks the Slack button) before the timer fires, the ACK flows back to the Incident Service, which cancels the escalation timer and moves the incident to acknowledged. Paging stops.
  7. If the timer fires first (no ack in time), the Timer Service invokes the Escalation Engine’s callback. The engine advances to the next policy level, resolves its targets, notifies them, and arms a fresh timer. This loop continues until acknowledged, resolved, or the policy is exhausted (then it loops or hits a fallback).

Request flow - resolve: a resolve event (or a human resolving in the UI) closes the incident, cancels any pending timer, and stops all escalation. Idempotent: resolving an already-resolved incident is a no-op.

The key architectural insight: the escalation loop is a durable state machine driven by cancelable, persisted timers, and every step is idempotent so that a crash-and-retry never double-pages and never drops a page. Ingest is decoupled from routing by the event log so a spike never blocks a monitoring tool; incident state lives in a strongly-consistent store so escalation decisions never race; timers live in a dedicated durable service so they fire regardless of which process created them; and the notification pipeline treats providers as unreliable and retries around them. Throughput is easy - correctness of the timer and the ack under failure is the entire game.

Component Deep Dive

Four hard parts: (1) the durable escalation timer that must fire even when nodes die, (2) resolving who is on call right now from layered rotations, overrides, and holidays, (3) delivering notifications reliably across unreliable providers without double-paging, and (4) deduplicating a storm of events into one incident. I will walk each from naive to scalable.

1. Durable Escalation Timers: setTimeout -> A Persisted, Sharded Timer Service

The core behavior is “escalate if not acknowledged in N minutes.” That N-minute wait is a scheduled job, and there may be a million of them pending. Where it lives decides whether the product works.

Approach A: In-memory timers on the escalation node (the naive instinct)

When the engine routes an incident, it calls setTimeout(escalate, N * 60_000) in the process, holding the incident in memory.

onNewIncident(inc):
   notify(level0)
   inc.timer = setTimeout(() => escalate(inc), inc.level.timeout)   # in RAM

Where it breaks: the timer lives in one process’s memory. If that process crashes or is redeployed - and processes crash and deploy constantly - every pending timer it held vanishes. The incidents it was tracking silently never escalate. That is the exact failure the product exists to prevent: a page that goes out, nobody acks, and it just dies. There is no durability, no failover, and no way to reconstruct which incidents were waiting. It works in a demo where nothing crashes and falls over the first time you deploy during an active incident. In-memory timers are disqualified for anything whose firing must survive.

Approach B: Poll a database WHERE fire_at <= now() (the next naive instinct)

Persist each timer as a row (incident_id, fire_at, status) and run a worker every minute: SELECT ... WHERE fire_at <= now() AND status='pending'.

Where it breaks: two problems. Precision: polling every minute means a “5 minute” timer fires anywhere from 5:00 to 6:00 - unacceptably imprecise for a live incident, and polling every second instead means a relentless full-index scan hammering the database. Concurrency: multiple workers running the same SELECT will both grab the same due row and double-escalate unless you add careful locking (SELECT ... FOR UPDATE SKIP LOCKED), and at high timer counts that scan-and-lock becomes a bottleneck and a hot spot. Naive polling is either imprecise or a database-crushing scan, and racy on top.

Approach C: A dedicated durable timer service - persisted, sharded, time-bucketed

Build a first-class Timer Service whose only job is “persist a callback to fire at time T, fire it within ~1s, exactly once, and let it be canceled.” Ideas that make it work:

  • Persist first, then schedule. Every timer is written durably (its own partitioned table or log) before it is acknowledged to the caller. A timer is (timer_id, fire_at, callback_ref, incident_id, epoch, status). Durability is non-negotiable - this is the state that must survive a crash.
  • Two-tier: time buckets on disk + an in-memory near-term wheel. Timers are bucketed by fire_at into coarse buckets (e.g. per-second or per-minute). A worker owning a shard loads only the near-future buckets (say the next 1-2 minutes) into an in-memory timing wheel / min-heap, which fires them with sub-second precision. Far-future timers just sit on disk and are loaded as their bucket approaches. This gives precision without scanning the whole table - you only ever look at the near horizon.
  • Sharded by a hash of incident_id, with clear ownership. The timer space is partitioned across worker shards; each shard exclusively owns its partitions (via a lease from a coordinator like ZooKeeper/etcd, or Kafka partition assignment). Exclusive ownership means exactly one worker will fire a given timer - no SKIP LOCKED races. If a worker dies, its lease expires and another worker takes over its partitions and reloads the near-term buckets from disk - the timers survive and keep firing.
  • Cancellation via status + epoch, not deletion races. An ack must cancel the timer. Rather than racing to delete a row that might be firing, we mark it canceled and stamp the incident with an epoch/generation number. When a timer fires, the callback checks the incident’s current epoch against the timer’s epoch; if they differ (the incident was acked, resolved, or already escalated), the fire is a no-op. This makes firing idempotent and safe against duplicates - a timer that fires twice, or fires just as an ack lands, does the right thing.
  • Firing is at-least-once, made effectively-once by the epoch check. We accept that under failover a timer might fire more than once (a worker fires, dies before recording it, another worker re-fires). The downstream escalation step is idempotent (guarded by the epoch and by incident state), so a duplicate fire never double-escalates.
arm(incident, level):
   epoch = incident.epoch                       # current generation
   persist Timer(id, fire_at=now+level.timeout, incident_id, epoch, status=pending)
   # worker for this shard will load it when its bucket nears

onTimerFire(timer):
   inc = load(timer.incident_id)                # strongly consistent read
   if inc.epoch != timer.epoch: return          # stale: acked/resolved/moved -> no-op
   if inc.state != 'triggered':   return        # defensive
   inc.epoch++                                   # invalidate any other timer for this inc
   advanceToNextLevel(inc)                       # notify next level, arm new timer
   persist(inc)                                  # commit new epoch + level atomically

ack(incident, user):
   inc.epoch++            # invalidates the pending escalation timer's epoch
   inc.state = 'acknowledged'
   persist(inc)          # the armed timer will now no-op when it fires

The mindset shift: an escalation timeout is not a setTimeout, it is a durable scheduled job in a purpose-built service that survives crashes, fires with sub-second precision by only holding the near horizon in memory, and is made exactly-once by an epoch check rather than by fragile delete/lock races. This is the single most important component - PagerDuty lives or dies on whether the escalation fires. (This same durable-timer pattern powers delayed-message queues, subscription renewals, and workflow timeouts - it is a reusable primitive.)

2. Who Is On Call Right Now: Resolving Layered Schedules, Overrides, and Holidays

Routing needs concrete users, but a schedule is not a static list - it is a set of recurrence rules with overrides on top, evaluated at an instant, in the right timezone.

Approach A: Store a static “current on-call” user per schedule (naive)

A column schedule.current_oncall_user_id, updated by a cron at each handoff.

Where it breaks: it cannot express any real schedule. A weekly rotation with a Monday 10am handoff, a secondary layer underneath, a colleague override covering Thursday 2-5pm, and a holiday rule that skips weekend paging on one layer - none of it fits in a single column. The cron that flips it is a single point of failure (miss the handoff and the wrong person is paged, or nobody is), it cannot answer historical “who was on call last Tuesday at 3am” for an incident timeline, and it cannot handle timezones (a “9am handoff” is a different UTC instant in IST vs PST, and shifts across DST). A static current-value throws away the fact that on-call is a function of time.

Approach B: Schedules as layered rules, resolved as a pure function of time

Model a schedule as ordered layers of rotations, with overrides applied on top, and resolve it by evaluating that function for a given timestamp:

  • A rotation is a recurrence rule. Store (start_time, rotation_length, handoff_time, [ordered users], restrictions, timezone). For any instant t, “who is on call” is pure arithmetic: elapsed = t - start_time, slot = floor(elapsed / rotation_length) mod len(users), adjusted to the handoff boundary in the schedule’s timezone. No cron, no stored current value - the answer is computed from the rule. This also makes historical queries free: pass a past t and get who was on call then.
  • Layers stacked by precedence. A schedule has ordered layers (a primary weekly rotation, a secondary, a business-hours layer). Each layer has restrictions (time windows it is active - e.g. “this layer only 9am-5pm weekdays”). Resolving the schedule evaluates layers top-down; the highest-precedence layer that is active and not restricted at t wins.
  • Overrides are a top priority layer. An override is (schedule_id, user_id, start, end) meaning “for this window, this user replaces whoever the rotation says.” Overrides are checked first - if t falls in an override window, the override user is on call, full stop. This is how “cover for me 2-5pm” works without touching the rotation.
  • Holidays and restrictions. Company holidays and per-layer restrictions are just time predicates in the resolution function: if a layer is restricted at t (weekend, holiday, outside its window), skip it and fall to the next layer. Configurable per schedule.
  • Timezone-correct, DST-aware. All handoff and restriction times are stored with the schedule’s IANA timezone and evaluated with a real timezone library, so a “Monday 9am handoff” is correct across DST transitions and for users in other zones.
resolveOnCall(schedule, t):
   for ov in schedule.overrides:                 # overrides win
        if ov.start <= t < ov.end: return ov.user
   for layer in schedule.layers (highest precedence first):
        if layer.restricted_at(t): continue      # holiday/weekend/out-of-window
        rot = layer.rotation
        elapsed = t - rot.start                   # all in schedule.timezone
        slot = floor(elapsed / rot.length) % len(rot.users)
        return rot.users[slot]
   return schedule.fallback_user                  # nobody scheduled -> account default
  • Caching + precomputation for latency. Resolving is cheap, but it is on the paging hot path and the escalation policy may reference several schedules. We cache the resolved on-call per schedule with a short TTL and precompute the next handoff time so the cache entry can be invalidated exactly at the boundary. Schedule edits publish an invalidation. Reads are effectively O(1) from cache; the function is the source of truth.
  • A final schedule and shift preview. Because resolution is a pure function of t, we can render the flattened “final schedule” (overrides + layers collapsed) for any window for the UI, and answer “who is on call in 2 hours” for pre-notifying an upcoming shift.

The mindset shift: on-call is a pure function of time over layered rules, not a stored current value. That makes it correct across timezones and DST, free to query historically, robust with no handoff-cron single point of failure, and cheap to cache because you know exactly when the answer changes.

3. Reliable Notification Across Unreliable Providers

Once we know the user, we must actually reach them - and every delivery channel goes through a third party (Twilio, APNs, FCM, Slack) that has rate limits, latency, and outages.

Approach A: Synchronously call the provider inline in the escalation engine (naive)

When escalating, the engine directly calls twilio.sendSMS(...) and waits for the response.

Where it breaks: the provider is a blocking, unreliable dependency in the critical path. If Twilio is slow or down, the escalation engine blocks or throws, and the whole paging loop stalls - exactly when you need it. A transient 500 from the provider means the page is simply lost (no retry). There is no failover to a second provider, no delivery tracking, and a retry of the whole escalation might send a duplicate SMS. Coupling paging to a synchronous third-party call makes the system only as reliable as its least reliable provider.

Approach B: An async, idempotent notification pipeline with retries and failover

Decouple notification into its own pipeline with durable per-notification state:

  • Enqueue, do not call inline. The escalation engine writes notification jobs to a durable queue (one per channel step) and returns immediately. Channel-specific workers (SMS, voice, push, Slack) consume and call providers. The escalation loop never blocks on a provider.
  • Per-user contact rules drive the channel steps. A user configures their own escalation of channels: “push immediately; if not acked in 1 min, SMS; if not acked in 3 min, phone call.” These generate a sequence of scheduled notification jobs (using the same Timer Service) so the user is progressively nudged harder - all cancelable the instant they ack.
  • Idempotent sends keyed by (incident, user, channel, step). Each notification job has a unique idempotency key. A worker records sent before/at the provider call and checks it on retry, so a retried or duplicated job never sends a second SMS for the same step. This is what makes at-least-once queue delivery safe.
  • Retries with backoff, then provider failover. A failed provider call retries with exponential backoff a few times; if the primary provider is failing broadly (error-rate breaker open), workers fail over to a secondary provider (a second SMS/voice vendor). No single provider outage blocks paging.
  • Delivery + acknowledgement tracking, closing the loop. Every notification has a lifecycle: queued -> sent -> delivered (provider callback) -> acked. Providers post delivery receipts via webhooks; the user acks via an inbound path - reply to the SMS, press a digit on the voice call (DTMF), tap the push, or click the Slack button. The inbound ACK handler maps the response back to the incident and triggers the epoch-bump cancel from part 1, stopping every pending notification step for that user and the escalation timer.
  • Rate-limit and provider-quota awareness. Workers respect per-provider rate limits (token buckets) and shard sends so a burst of 10,000 pages during a big outage is smoothed into the providers’ allowed rate rather than getting the account throttled or blocked.
escalateToUser(incident, user):
   steps = user.contact_rules            # [push@0, sms@1m, call@3m]
   for step in steps:
        key = idem_key(incident.id, user.id, step.channel, step.index)
        armTimer(fire_at = now + step.delay,
                 callback = enqueueNotify(key, incident, user, step.channel))

worker(job):                             # per channel
   if store.get(job.key).status in {sent, acked}: return   # idempotent
   ok = provider.send(job)  or  failover_provider.send(job)
   store.mark(job.key, sent if ok else failed_retryable)

onAck(incident, user):                   # inbound: SMS reply / DTMF / push / Slack
   cancelAllNotificationTimers(incident, user)
   incident.epoch++ ; incident.state = acknowledged     # stops escalation (part 1)

The mindset shift: treat every delivery channel as an unreliable dependency behind a durable, idempotent, retrying, failover-capable pipeline - never a synchronous call in the escalation path. Idempotency keys make at-least-once queues safe from double-paging, per-user contact rules progressively escalate channels, and the inbound-ack path is what actually stops the machine.

4. Deduplication: One Incident From a Storm of Events

During a real outage the ingest firehose is mostly duplicates. If each event pages, you page hundreds of times for one problem and the on-call learns to ignore the pager.

Approach A: Create an incident per event, page per incident (naive)

Every POST /events makes a new incident and routes it.

Where it breaks: a flapping check firing every 10 seconds creates a new page every 10 seconds; a shared-dependency failure that trips 400 host alerts creates 400 incidents and 400 pages for one root cause. This is the “page storm” that destroys trust in the pager - the single fastest way to make on-call engineers mute notifications, which defeats the entire product. Per-event paging is not viable.

Approach B: Deduplicate by key into a single incident, with alert grouping on top

  • Dedup key -> one open incident. Every event carries (or we derive) a dedup_key (a.k.a. alert key / incident_key). The Incident Service keys open incidents by (service_id, dedup_key) with a unique constraint: the first trigger creates the incident; subsequent triggers with the same key fold in (append to the event log, bump a counter, maybe re-alert if configured) but do not create a new incident or a new page. A resolve with that key closes it. This upsert is the 20-50x collapse from the BoE.
  • Idempotent upsert, concurrency-safe. Two events with the same key arriving concurrently must not create two incidents - the unique constraint (or a compare-and-set on the key) ensures exactly one wins and the other folds in. Per-service partitioning of the event log also serializes events for one incident onto one consumer, avoiding the race entirely.
  • Auto-resolve and flap suppression. If the monitoring source sends resolve, we close automatically. For flapping (trigger/resolve/trigger rapidly), a short suppression window collapses the churn so one flapping check is one incident, not fifty.
  • Alert grouping above dedup (intelligent grouping). Dedup handles identical alerts; grouping handles related ones - many distinct alerts that are one incident (all services in a failing region). Group by shared labels (cluster, region, service dependency) or a time-window heuristic (“many alerts for one service in 5 minutes -> one grouped incident”), so a broad failure is one page describing N alerts, not N pages. This mirrors the alert-manager grouping the monitoring system does upstream, applied here at the paging layer.
  • Priority and transient-suppression. Low-severity events can be configured to only create incidents (not page) unless they escalate; this keeps the actual paging volume to genuine, actionable incidents.

The mindset shift: paging volume must track the number of real problems, not the number of events, so dedup (identical -> one incident) and grouping (related -> one incident) sit in front of routing. A trustworthy pager only fires for things a human should act on - the noise is collapsed before anyone is woken up.

API Design & Data Schema

Event ingest API (the write plane)

POST /v1/events                                  # from monitoring integrations
  Authorization: Integration-Key <key>
  {
    "routing_key": "svc_checkout_prod",           # -> service
    "event_action": "trigger",                    # trigger | acknowledge | resolve
    "dedup_key": "cpu-high-host-42",              # dedup identity (optional; derived if absent)
    "payload": { "summary": "CPU > 95% for 5m",
                 "severity": "critical",
                 "source": "host-42", "custom_details": {...} }
  }
  202 Accepted  { "dedup_key": "cpu-high-host-42", "status": "triggered" }
  # 202 after durable append to the event log, before routing completes.

Incident + response API (the control plane)

GET  /v1/incidents?status=triggered,acknowledged&service_id=...
GET  /v1/incidents/{id}                          # state, timeline, current escalation level
POST /v1/incidents/{id}/acknowledge  { "user_id": "u_9" }   # stops escalation
POST /v1/incidents/{id}/resolve      { "user_id": "u_9" }
POST /v1/incidents/{id}/reassign     { "to_user_id": "u_3" }
POST /v1/incidents/{id}/snooze       { "duration": "1h" }
# Inbound ack from providers (SMS reply / voice DTMF / Slack action) hit internal
# webhooks that map the response -> incident -> POST .../acknowledge.
POST /webhooks/twilio/inbound                     # "reply 4 to ack" / DTMF
POST /webhooks/slack/actions                      # ack/resolve button

Schedule + policy API (config plane)

POST /v1/escalation_policies
  { "name":"Checkout Prod",
    "levels":[
      { "targets":[{"type":"schedule","id":"sch_primary"}],   "timeout":"5m" },
      { "targets":[{"type":"schedule","id":"sch_secondary"}], "timeout":"10m" },
      { "targets":[{"type":"user","id":"u_lead"}],            "timeout":"15m" } ],
    "on_exhaustion":"repeat" }                    # repeat | notify_admins

POST /v1/schedules
  { "name":"Primary", "timezone":"Asia/Kolkata",
    "layers":[ { "rotation_length":"7d", "handoff":"Mon 10:00",
                 "users":["u_1","u_2","u_3"], "restrictions":[] } ] }
POST /v1/schedules/{id}/overrides
  { "user_id":"u_5", "start":"2026-07-24T14:00:00+05:30", "end":"...T17:00:00+05:30" }
GET  /v1/schedules/{id}/oncall?at=2026-07-20T03:00:00Z   -> resolved user(s)

Data model - split by consistency need

Strongly-consistent core (SQL - Postgres): incidents, escalation state, timers. This is the state escalation decisions race on; it needs transactions, unique constraints, and strong reads.

incidents(
   incident_id PK, service_id, dedup_key,
   state ENUM(triggered|acknowledged|resolved),
   escalation_policy_id, current_level INT,
   epoch BIGINT,                       -- generation; bumped on ack/resolve/escalate
   assigned_user_id, created_at, updated_at,
   UNIQUE(service_id, dedup_key) WHERE state != 'resolved'   -- dedup guarantee
)
   INDEX(service_id, state)            -- open incidents per service
   INDEX(updated_at)                   -- timelines / cleanup

timers(
   timer_id PK, incident_id FK, fire_at TIMESTAMP,
   epoch BIGINT,                       -- must match incident.epoch to fire
   kind ENUM(escalation|notify_step), callback_ref,
   status ENUM(pending|fired|canceled),
   shard_id INT                        -- hash(incident_id) -> owner worker
)
   INDEX(shard_id, fire_at) WHERE status='pending'   -- near-horizon load per shard

notifications(
   notif_id PK, incident_id, user_id, channel ENUM(push|sms|voice|email|slack),
   step_index INT, idem_key UNIQUE,    -- (incident,user,channel,step) -> no double-send
   provider, status ENUM(queued|sent|delivered|failed|acked),
   provider_msg_id, created_at )

Config store (SQL, cache-fronted): policies, schedules, layers, overrides, users, contact rules. Tiny, read on every route, so it lives behind a cache keyed by schedule/policy id with invalidation on edit.

escalation_policies(policy_id PK, org_id, on_exhaustion)
policy_levels(policy_id FK, level_index, timeout_seconds)
policy_targets(policy_id FK, level_index, target_type, target_id)
schedules(schedule_id PK, org_id, timezone)
schedule_layers(schedule_id FK, layer_index, rotation_length,
                handoff_anchor, restrictions_json)
layer_users(schedule_id FK, layer_index, order_index, user_id)
overrides(override_id PK, schedule_id FK, user_id, start_ts, end_ts)
users(user_id PK, org_id, name, timezone)
contact_rules(user_id FK, step_index, channel, delay_seconds)   -- personal escalation
contact_methods(user_id FK, channel, address)                    -- phone, device token, ...

Event / timeline log (append-only, partitioned NoSQL or Kafka-backed store): every event and state transition per incident, high-volume and time-ordered, TTL’d to archive.

incident_events(incident_id PARTITION KEY, ts CLUSTERING KEY,
                event_action, actor, channel, detail_json)   -- append-only timeline

Why this split: the escalation core (incidents, timers, dedup) needs strong consistency, unique constraints, and transactions, so it is SQL; the config (schedules, policies) is tiny and read-heavy, so it is cached SQL; the event timeline is append-only and high-volume, so it is a partitioned log store with TTL. SQL wins the core because the whole system hinges on never racing two escalations or losing the dedup uniqueness - exactly the guarantees a relational store with constraints and transactions gives you, and exactly what a monitoring firehose does not need but this control plane does.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Lost or imprecise escalation timers (the number-one killer). In-memory timers die with the process; polled-DB timers are imprecise and racy. A lost timer is a page that never escalates - the cardinal sin. Fix: a dedicated durable Timer Service - persist before scheduling, shard by hash(incident_id) with exclusive lease-based ownership so exactly one worker fires each timer, hold only the near-term horizon in an in-memory timing wheel for sub-second precision, and make firing idempotent via an epoch check so a failover re-fire is a no-op. Timers survive crashes and fire on time.

2. Double-paging under retries and failover. At-least-once queues, timer re-fires, and concurrent acks can each cause a duplicate page. Fix: idempotency everywhere - notification jobs keyed by (incident,user,channel,step), escalation guarded by the incident epoch, dedup guarded by a unique constraint. Every action is safe to retry.

3. Missed or racing escalation decisions. Two nodes both deciding to escalate the same incident, or an ack landing exactly as a timer fires. Fix: strongly-consistent incident state in SQL with the epoch/generation guard - the fire re-reads state and no-ops if the epoch moved. Per-service event-log partitioning serializes events for one incident onto one consumer.

4. Page storms from duplicate/related alerts. One outage tripping hundreds of alerts pages hundreds of times and destroys pager trust. Fix: dedup by key into one incident (unique constraint, fold-in) plus grouping of related alerts by shared labels/time window. Paging volume tracks real problems, not event count.

5. Provider (Twilio/APNs/Slack) latency or outage in the critical path. A synchronous provider call stalls the whole escalation loop. Fix: an async notification pipeline - enqueue and return, per-channel workers, retries with backoff and failover to a secondary provider, delivery-receipt webhooks. No single provider outage blocks paging; respect provider rate limits with token buckets.

6. Schedule resolution as a single point of failure / stale. A handoff cron that flips a static current-on-call can miss and page the wrong person; timezone/DST bugs page nobody. Fix: on-call as a pure function of time over layered rotations + overrides + holidays, timezone-correct, cached with invalidation exactly at the next handoff. No cron SPOF; historical queries are free.

7. Ingest spike during a large outage. A region failing makes every customer’s alerts fire at once (load anti-correlated with calm). Fix: decouple ingest from routing with a durable event log (Kafka) that absorbs the spike; stateless ingest scales horizontally; dedup collapses the firehose 20-50x before it reaches the escalation engine. Reserve headroom for the peak.

8. Timer store hot shard. One giant customer or one service with a flood of incidents concentrates timers on one shard. Fix: shard by hash(incident_id) (not by customer or service) so timers spread evenly; rebalance shards by moving partition leases. The near-horizon load is bounded per shard.

9. Escalation engine or region failure. The engine node holding active loops dies, or a datacenter goes down. Fix: the engine is stateless - all state (incident, epoch, timers) is in the durable stores, so any engine instance can pick up any incident, and timer ownership fails over via lease. Multi-region replication of the core stores with failover, because this system must outlive the outages it reports.

10. Its own availability during the worst moment. Monitoring load peaks exactly when infra is failing; if the pager is down, outages go unnoticed. Fix: make the escalation and ingest paths more available than the monitored systems - multi-region, replicated stores, isolated from non-critical paths (analytics/UI can degrade, the pager cannot), and reserve capacity for the outage spike. Degrade dashboards before you ever degrade a page.

Wrap-Up

The trade-offs that define this design:

  • A dedicated durable timer service over in-memory or polled timers. Escalation timeouts are persisted, sharded scheduled jobs that survive crashes and fire within a second by holding only the near horizon in memory, made exactly-once by an epoch check rather than fragile locks. This is the component the whole product hinges on - a lost timer is a page that never escalates.
  • Strong consistency for the escalation core, relaxed elsewhere. Incidents, dedup, and timers live in SQL with unique constraints, transactions, and an epoch guard so two nodes never race an escalation and an ack reliably stops paging - the exact opposite of the eventual-consistency relaxation a metrics firehose can afford. Schedules and the event timeline get cheaper stores.
  • On-call as a pure function of time. Rotations, layers, overrides, and holidays are resolved by evaluating rules at an instant, timezone- and DST-correct, cached with invalidation at each handoff - no handoff-cron single point of failure and free historical lookups, instead of a brittle stored current-on-call value.
  • Idempotency as the load-bearing principle. At-least-once queues, timer re-fires, and provider retries are all made safe by idempotency keys and epoch generations, so the system can retry aggressively for reliability without ever double-paging.
  • An async, failover-capable notification pipeline. Every channel is an unreliable third party behind a durable, retrying, provider-failover queue with delivery and ack tracking - never a synchronous call in the escalation path - and dedup plus grouping keep paging volume tied to real problems so the pager stays trustworthy.

One-line summary: a stateless ingest tier deduplicating an event firehose into strongly-consistent incidents, an escalation engine that walks policies and resolves on-call as a pure function of layered rotations/overrides/holidays, driven by a purpose-built durable timer service that fires sub-second and survives crashes via epoch-guarded exactly-once semantics, feeding an idempotent async notification pipeline with provider failover - trading raw throughput concerns for the one guarantee that matters, that when production breaks the right human is paged and, if they do not answer, the escalation always fires.