You type a tracking number into an app and a timeline appears: “Picked up in Pune”, “Departed Mumbai hub”, “Arrived at Delhi facility”, “Out for delivery”, “Delivered”. Underneath that tidy list is a parcel that got scanned by a dozen different devices across a week - a pickup scanner in a van, a conveyor belt sorter, a handheld at a hub, a doorstep delivery app - each of them firing an event into the system from a different city, sometimes out of order, sometimes twice, sometimes hours late because the scanner was offline in a warehouse basement. The product looks like a list. The system is a distributed event pipeline feeding a state machine, and the interviewer will spend the whole hour on the gap between those two things.

The hard part the problem foregrounds is this: 100 million parcels, each generating several scan events a day, flowing through an event-sourced state machine, while you push real-time status changes to customers and keep every parcel’s full location history queryable at low latency. That is three different workloads fighting each other - a high-volume append-only ingest, a low-latency fan-out to connected clients, and a point-lookup read path over history. Conflate them and the design collapses. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Ingest scan events. Scanners, sorters, delivery apps, and partner carriers emit events like (tracking_id, scan_type, facility_id, timestamp, device_id). The system ingests them durably at high volume.
  • Maintain each parcel’s state via a state machine. From the sequence of scan events, derive the current canonical status of the parcel: CREATED -> PICKED_UP -> IN_TRANSIT -> AT_HUB -> OUT_FOR_DELIVERY -> DELIVERED (plus EXCEPTION, RETURNED, LOST). Transitions are driven by events and governed by rules.
  • Serve current status. Given a tracking id, return the parcel’s current status and latest location fast.
  • Serve full history. Return the ordered list of every scan/status event for a parcel - the timeline the customer sees.
  • Push real-time updates. When a parcel’s status changes, notify subscribed customers (app push, SMS/email, webhook to merchants) within seconds.
  • Handle out-of-order, duplicate, and late events. A hub scan can arrive before the preceding pickup scan; the same scan can be delivered twice; an offline scanner can flush a batch hours late. The derived state must stay correct regardless.
  • Expose estimated delivery (ETA). Surface a predicted delivery window; recompute it as the parcel progresses.
  • Merchant/carrier APIs. Let merchants register parcels and subscribe to status webhooks; let partner carriers push events for legs they operate.

Explicitly out of scope (say this so you own the scope):

  • Route optimization and vehicle dispatch. Deciding which truck carries which parcel on which road is a separate logistics-planning system. I consume scan events; I do not plan routes.
  • The physical scanning hardware and label printing. Barcode symbology, scanner firmware, label generation - hardware and provisioning. I treat a scanner as a client that emits events.
  • Billing, pricing, and payments. Shipping cost, invoicing, COD settlement are a payments problem. I track the parcel, not the money.
  • Warehouse inventory / WMS. What is on which shelf is a warehouse management concern. I track parcels in motion.
  • ETA model training. I consume a prediction service; I do not design the ML that produces the time estimate, only where it plugs in.

The one decision that drives everything: the scan events are the source of truth, and the parcel’s status is a derived projection of its event log, not an independently mutated field. The moment you treat “status” as a column you overwrite with the latest scan, out-of-order and duplicate events corrupt it and you can never reconstruct what actually happened. Event sourcing is not a buzzword here; it is the only thing that survives the messiness of real-world scans.

Non-Functional Requirements (NFR)

  • Scale: ~100M active parcels in flight at any time, each generating ~5-10 scan events over its lifetime, concentrated over a few days. ~500M-1B scan events ingested per day. ~100M+ status reads/day, spiking hard (everyone checks their package in the evening).
  • Latency: current-status read p99 < 100ms; full-history read p99 < 200ms; real-time push from scan-ingest to customer notification p95 < 5s, ideally sub-second. Ingest must accept an event in < 50ms and durably buffer it.
  • Availability: ingest is the critical path and must be 99.99%+ - a dropped scan is a parcel that goes dark. Read and push can degrade more gracefully (stale-but-available beats unavailable).
  • Consistency: the parcel’s status is eventually consistent but monotonic and correct - it must never go backwards (a late “in transit” event must not un-deliver a delivered parcel) and must converge to the right state regardless of event order. The event log itself is durable and append-only; losing a scan is the one thing we cannot do.
  • Durability: every scan event is a durable, immutable record retained for the parcel’s lifetime plus an audit window (disputes, “where is my package” investigations, SLA claims). Zero tolerance for lost events.
  • Idempotency: every event carries a stable id; re-delivery must not double-apply. Ingest and projection are both idempotent.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they set the architecture.

Event ingest rate:

100M active parcels, ~8 scan events each over ~4 day life.
Events/day ~= 100M * 8 / 4 ~= 200M/day steady, but new parcels keep entering;
realistic working number ~500M scan events/day.
= 500,000,000 / 86,400 s ~= 5,800 events/sec average.
Peak factor ~3-4x (hubs sort at night, multiple shift cutoffs align)
Peak ~= 20,000-25,000 events/sec.

So ingest is a ~20-25k events/sec append-only write - high volume, but each event is tiny and the write is a durable append, not a contended update. That shape points straight at a log/queue, not a relational row update.

Read rate:

100M parcels, say 1.5 status checks each per day on average (people refresh)
-> 150M reads/day = ~1,700 reads/sec average, peak ~8,000-10,000/sec.
Reads are point lookups by tracking_id: "current status" (hot) and
"full history" (colder). Current-status reads dominate ~10:1.

Read:write is roughly balanced in volume but opposite in shape - writes are an append stream, reads are keyed point lookups. That tells you the storage for ingest and the storage for serving reads should probably be different engines.

Storage:

Scan event record ~= 300 bytes (tracking_id, event_id, scan_type,
  facility_id, geo, device_id, timestamps, carrier_leg, raw_payload).
500M/day * 300B = 150 GB/day of raw events.
-> ~55 TB/year of event log, before replication (x3 -> ~165 TB/year).
Parcel current-state records: 100M active * ~500B = 50 GB, fits in memory
  across a cache cluster.
History materialized per parcel: ~8 events * 300B = 2.4 KB/parcel;
  100M active -> ~240 GB hot, trivially shardable.

Event log is large but boring - append-only, partitionable by time, cold after the parcel is delivered + retention window. Current-state is small enough to keep hot in memory.

Real-time push fan-out:

Not every parcel has a live subscriber. Say 20M parcels have an app open or
a webhook registered at peak. Status changes per parcel ~= 8 over its life
-> ~100M status-change notifications/day = ~1,200/sec avg, peak ~5,000/sec.
Concurrent WebSocket/long-poll connections at evening peak: a few million.

Bandwidth:

Ingest: 25k events/sec * 300B ~= 7.5 MB/sec. Trivial.
Reads: 10k/sec * ~2KB history payload ~= 20 MB/sec. Trivial.
This system is not bandwidth-bound; it is ingest-durability-bound and
fan-out-connection-bound.

The takeaways: a high-volume append-only event ingest (~25k/sec) that must never lose an event, a small hot current-state set (~50GB) that lives in memory, large boring append-only history storage, and a few-million-connection real-time fan-out. The design driver is keeping those three workloads on separate rails so one does not starve another, with an idempotent event-sourced state machine in the middle.

High-Level Design (HLD)

Four planes. The ingest plane accepts scan events from scanners and carriers, dedups and durably logs them. The projection plane consumes the event log, runs each parcel through its state machine, and maintains the current-state store and the per-parcel history. The serving plane answers status and history reads from read-optimized stores. The push plane fans out status changes to subscribed customers in real time. The event log in the middle is the source of truth and the decoupling buffer between ingest and everything downstream.

  Scanners / Sorters / Delivery apps / Partner carriers
        │ scan event (tracking_id, type, facility, ts, device, event_id)
        ▼
  ┌─────────────────────────────┐
  │  INGEST API / GATEWAY       │   validate, authenticate device,
  │  - dedup on event_id        │   stamp receive_ts, drop malformed
  │  - durable append           │
  └──────────────┬──────────────┘
                 │ append (partitioned by tracking_id)
        ┌────────▼─────────────────────────────┐
        │  EVENT LOG (Kafka-style, partitioned) │  ← source of truth
        │  partition = hash(tracking_id)        │  append-only, replicated
        └───────┬───────────────┬───────────────┘
                │               │
    ┌───────────▼────┐   ┌──────▼────────────────┐
    │ PROJECTION     │   │ ARCHIVER              │
    │ WORKERS        │   │ - raw events -> cold   │
    │ - per-parcel   │   │   object store / WH    │
    │   state machine│   └────────────────────────┘
    │ - fold events  │
    │ - idempotent   │
    └───┬────────┬───┘
        │        │ on transition -> emit status-change
        │        ▼
        │   ┌──────────────────────┐
        │   │ STATUS-CHANGE TOPIC  │──► PUSH SERVICE ──► WebSocket/APNs/FCM
        │   └──────────────────────┘    (fan-out)     ──► webhook to merchants
        │                                              ──► SMS/email
        ▼
  ┌──────────────────────────┐     ┌──────────────────────────┐
  │ CURRENT-STATE STORE      │     │ HISTORY STORE            │
  │ (KV, in-mem + durable)   │     │ (wide-column, by tid)    │
  │ tracking_id -> status,   │     │ tracking_id -> [events]  │
  │   last_facility, eta     │     │ ordered, append-only     │
  └───────────┬──────────────┘     └───────────┬──────────────┘
              │                                 │
        ┌─────▼─────────────────────────────────▼─────┐
        │  READ / SERVING API  (status + history)      │
        │  cache -> current-state -> history           │
        └──────────────────────┬───────────────────────┘
                               ▼
                         Customer apps / web

Ingest path (durable, idempotent, fast):

  1. A scanner emits a scan event with a client-generated stable event_id (e.g. device_id + monotonic_seq or a UUID). The ingest gateway authenticates the device, validates the payload, and dedups on event_id (recent-set check).
  2. The event is appended to the event log, partitioned by hash(tracking_id) so all events for a parcel land in the same partition in arrival order. The gateway acks only after the durable append. This is the critical path: accept-and-durably-log in < 50ms.

Projection path (the state machine):

  1. Projection workers consume the log partition-by-partition. For each parcel, they fold the event sequence through the state machine to compute the canonical current status, reconciling out-of-order and duplicate events by event semantics and timestamps (not arrival order).
  2. On a real transition, the worker updates the current-state store, appends the derived status event to the history store, and emits a status-change message to the status-change topic. All idempotent on (tracking_id, event_id).

Read path:

  1. The serving API answers “current status” from the current-state store (fronted by cache) in < 100ms and “full history” from the history store in < 200ms. Neither read touches the event log or the projection workers.

Push path:

  1. The push service consumes the status-change topic and fans out to whoever is subscribed for that tracking_id: live app connections (via a connection-routing layer), registered merchant webhooks, and SMS/email. Delivered best-effort with retries; the current-state store is the fallback source of truth if a push is missed.

The key insight: events are appended once and durably on the critical path; everything else - state, history, reads, push - is a downstream idempotent consumer of that log, so no slow consumer can ever slow ingest, and the state is always reconstructable by replaying the log.

Component Deep Dive

The hard parts, naive-first then evolved: (1) the event-sourced state machine under out-of-order and duplicate events, (2) low-latency history and current-status reads, (3) real-time push fan-out to millions of connections, and (4) idempotent, lossless ingest.

1. The event-sourced state machine under messy events

This is the heart of the problem. Everything else is plumbing around getting the derived state right.

Naive approach: store a status column and overwrite it on every scan.

on scan_event e:
    UPDATE parcels SET status = e.derived_status,
           last_facility = e.facility, updated_at = now()
    WHERE tracking_id = e.tracking_id

Where it breaks:

  • Out-of-order events corrupt the status. Scanners and carriers push over flaky networks. A parcel’s “OUT_FOR_DELIVERY” scan can arrive before the “AT_HUB” scan it logically follows (different devices, different uplinks, batched flushes). Last-write-wins on arrival order sets the status to AT_HUB after it was already OUT_FOR_DELIVERY. The parcel visibly goes backwards, which is the one thing a tracking timeline must never do.
  • Duplicates double-apply. A scanner retries on timeout; the same event lands twice. With side-effecting transitions (emit notification, increment a counter) the customer gets two “Delivered” texts.
  • No history, no audit. You overwrote the previous status. When a customer disputes “it says delivered but I never got it”, there is no record of what scans actually fired. You cannot reconstruct anything.
  • Late events rewrite settled parcels. An offline scanner flushes a day-old “in transit” batch; it overwrites a parcel that was already delivered.

First evolution: keep the events, derive the state. Stop storing status as a mutable field. Store the append-only event log as the source of truth and compute status as a fold (left reduce) over the events:

status(parcel) = reduce(state_machine, CREATED, events_sorted_by_event_time)

Now you never lose history, and recomputing from scratch always gives the right answer. But a naive re-fold of the entire event list on every new event is O(n) per event and does not by itself solve ordering - you still have to decide what “sorted” and “valid transition” mean.

The answer: an idempotent, order-tolerant state machine with explicit transition rules and a monotonic rank.

  • Each state has a monotonic rank. CREATED=0, PICKED_UP=1, IN_TRANSIT=2, AT_HUB=3, OUT_FOR_DELIVERY=4, DELIVERED=5. Terminal/branch states (RETURNED, LOST, EXCEPTION) are modeled explicitly with their own transition rules. The current status is derived as the event implying the highest-rank reachable state consistent with the transition rules, not the last-arrived event. This makes the projection monotonic: a late lower-rank event (IN_TRANSIT arriving after DELIVERED) is recorded in history but does not regress the current status.
  • Order by event-time, not arrival-time. Each event carries the device’s scan timestamp (event_ts). The fold sorts by event_ts (tie-break by facility rank / event_id) so the derived timeline reflects physical reality, not network luck. Clock skew between devices is bounded by the rank rules - ranks, not raw timestamps, decide the canonical current state, so a mis-set scanner clock cannot un-deliver a parcel.
  • Idempotent transitions keyed on event_id. The projection keeps, per parcel, the set of applied event_ids (compact - a parcel has ~8 events). Re-delivery of an applied event is a no-op. Side effects (emit status-change) fire only on a real rank change, so duplicates never double-notify.
  • Incremental state, not full re-fold. The current-state store holds a small materialized snapshot per parcel: {rank, status, last_facility, applied_event_ids, event_count}. A new event updates the snapshot incrementally - apply the transition if it is valid and not seen, bump rank only if higher. Full re-fold from the log is reserved for repair/rebuild, not the steady path.
  • Exceptions and branches are first-class. An EXCEPTION (damaged, misrouted, customs hold) or RETURN_INITIATED is a modeled transition with its own rules, not an error. The state machine encodes which transitions are legal from which state (you cannot go DELIVERED -> OUT_FOR_DELIVERY, but you can DELIVERED -> RETURN_INITIATED). Illegal transitions are logged as anomalies and sent to an exceptions queue, never silently applied.
apply(snapshot, event):
    if event.event_id in snapshot.applied_event_ids: return snapshot   # idempotent
    target = STATE_OF[event.scan_type]
    if not legal_transition(snapshot.status, target): 
        record_anomaly(event); mark_applied(event); return snapshot
    snapshot.applied_event_ids.add(event.event_id)
    snapshot.event_count += 1
    if rank(target) > rank(snapshot.status):        # monotonic forward only
        snapshot.status = target
        snapshot.last_facility = event.facility
        emit_status_change(parcel, target)          # side effect on real change
    append_history(parcel, event)                   # history keeps EVERY event
    return snapshot

The mental model: the event log is immutable truth; the status is a monotonic projection derived by a rule-based state machine that tolerates any arrival order and any duplication because it keys on event_id and advances only by rank. History keeps every event (even the late/illegal ones, flagged); the current status only ever moves forward through legal transitions.

2. Low-latency history and current-status reads

Naive approach: on read, query the event log and fold the events on the fly.

GET status -> read all events for tracking_id from the log, fold, return.

Where it breaks:

  • The event log is partitioned for append throughput, not point reads. Scanning a partition to find one parcel’s events is slow and scales with partition size. At 10k reads/sec this melts the log.
  • Folding on every read wastes CPU. The same fold is recomputed for every one of the millions of daily checks of the same parcel.
  • History read is a range scan the log cannot serve cheaply. The customer-facing timeline is “all events for this id, ordered” - a query the log’s partition-by-time layout does not answer well.

The fix: precompute both projections into read-optimized stores, serve from them, cache the hot current-status.

  • Current-state store = keyed KV, in-memory-backed. tracking_id -> {status, last_facility, eta, updated_at}. Point lookup by key, served in single-digit ms. Updated by the projection worker on each transition. This is the ~50GB hot set - it lives in an in-memory KV cluster (Redis-style) with durable backing, sharded by tracking_id.
  • History store = wide-column, clustered by event-time. tracking_id is the partition key, event_ts the clustering key, so “all events for this parcel in order” is a single partition range scan - exactly what Cassandra/Bigtable-style wide-column stores do best. Append-only; a new event is an append to the row’s partition, never a rewrite.
  • Cache the hot current-status. The evening-refresh spike hits a small set of in-flight parcels repeatedly. A read-through cache (short TTL, invalidated on status change) absorbs it so most reads never touch the KV store either. Status-change events invalidate the cache key proactively, so the cache is never more than one push behind.
  • Separate read scaling from write scaling. Because reads hit the current-state KV and history store - not the log, not the workers - the read path scales by adding read replicas and cache nodes independently of ingest. A read spike cannot slow a scan; a scan spike cannot slow a read.
GET /track/{tracking_id}           -> cache -> current-state KV      (<100ms)
GET /track/{tracking_id}/history   -> history store range scan       (<200ms)

The principle: never derive state on the read path. Precompute the current status and the ordered history into stores whose physical layout matches the query (point lookup, ordered range scan), and front the hottest with cache. Reads are O(1) / O(events) lookups, fully decoupled from ingest and projection.

3. Real-time push fan-out to millions of connections

Naive approach: clients poll GET /track/{id} every few seconds.

Where it breaks:

  • Polling is mostly wasted work. A parcel changes status ~8 times over 4 days. Polling every 10s is ~34,000 reads per parcel for 8 real updates - a ~4000:1 waste ratio. At 20M active watchers that is a crushing, pointless read load.
  • Latency is bounded by poll interval. “Delivered” shows up up to a poll-interval late; tighten the interval and the waste explodes.
  • Thundering herd on events. Everyone polling at the evening peak hammers the read path in sync.

The fix: push on real transitions through a dedicated fan-out plane, with polling as the fallback.

  • Status-change topic as the fan-out source. The projection worker emits a status-change message only on a real rank change (section 1). The push plane consumes this topic - it never watches the raw scan log, so it only ever processes meaningful changes (~5k/sec peak, not 25k/sec).
  • Connection-routing layer for live clients. Apps hold a WebSocket (or long-poll) connection to a pool of stateless gateway nodes. A subscription registry (tracking_id -> {connection_ids}, in a fast KV) maps a parcel to the gateway nodes holding interested connections. On a status change, the push service looks up subscribers and routes the message only to the gateways that need it. Connections are sticky to a gateway; the registry is updated on connect/disconnect.
  • Mobile push via APNs/FCM for backgrounded apps. When the app is not foregrounded there is no live socket; the push service sends through APNs/FCM using the device token stored at subscribe time. SMS/email for users who opted in.
  • Merchant webhooks with at-least-once + retry. Merchants register a webhook URL per shipment or account. The push service POSTs the status change; on non-2xx it retries with backoff from a durable queue, and dedups on event_id so the merchant can be idempotent. A dead-letter queue catches persistently failing endpoints.
  • The current-state store is the reconciliation backstop. Push is best-effort; a missed push is not a lost update because the next time the client opens the app it reads authoritative current status from the KV store. Push optimizes latency; the store guarantees correctness.
status change (real transition) ->
  status-change topic ->
    push service:
      live sockets  -> subscription registry -> gateway nodes -> clients
      background    -> APNs / FCM
      merchants     -> webhook POST (retry + DLQ, idempotent on event_id)
      opted-in      -> SMS / email
  fallback: client reconnect reads current-state KV (authoritative)

The principle: push only on real state changes through a dedicated plane that fans out by subscription, and treat the current-state store as the source of truth that heals any missed push. Polling becomes a degraded fallback, not the mechanism.

4. Idempotent, lossless ingest

Naive approach: scanner POSTs to the API, which writes directly to the database and returns 200.

Where it breaks:

  • A slow or down projection/DB drops scans. If ingest writes synchronously to the state DB and the DB is slow, the scanner times out and the scan is lost - a parcel goes dark. Ingest availability becomes hostage to the slowest downstream component.
  • No dedup under retries. Scanners retry aggressively on flaky warehouse networks; the same scan arrives many times.
  • Burst sorting overwhelms synchronous writes. A hub sorter rips through thousands of parcels in a burst; synchronous per-event DB writes cannot absorb a 4x spike.

The fix: durable append to a log, decoupled from projection, idempotent on event_id.

  • Accept-and-log, ack fast. The ingest gateway’s only job on the critical path is to validate, dedup, and durably append to the event log, then ack. Appending to a partitioned replicated log is cheap and absorbs bursts; the ack happens in < 50ms once the append is durable (replicated to quorum). Projection happens asynchronously downstream, so a slow projector buffers in the log instead of dropping scans.
  • Client-stable event_id for idempotency. Each scanner generates event_id = device_id + monotonic_seq (stable across retries). The gateway checks a recent-events set (short-TTL dedup cache) and the log is keyed so the projection also dedups on event_id - belt and suspenders. A retried scan is dropped at the gateway or no-ops at the projection.
  • Partition by tracking_id. partition = hash(tracking_id) % N keeps all of a parcel’s events in one partition in arrival order, so the projection worker for that partition sees them as a single ordered stream per parcel and never needs cross-partition coordination to fold one parcel’s state.
  • Durable offline buffering at the scanner. Warehouse basements lose connectivity. Scanners buffer events locally and flush when the link returns; because events carry event_ts and stable event_id, a late flush is applied correctly (ordered by event-time, deduped by id) without corrupting already-advanced state.
  • Backpressure, never drop. If a partition’s projection falls behind, the log retains the backlog and the projector catches up; ingest never blocks or drops because a consumer is slow. The log is the shock absorber between a bursty producer and a steady consumer.

The principle: the critical path is a durable, idempotent append to a partitioned log, and nothing downstream can make ingest lose or double-count a scan. Projection, state, and push all drain the log at their own pace; the log guarantees durability and ordering-per-parcel; the event_id guarantees exactly-once effect.

API Design & Data Schema

Scanners and carriers push events; customers and merchants read and subscribe.

Ingest (scanners / partner carriers)

POST /ingest/v1/scan
  headers: { device_token / carrier_api_key, signature }
  body: { event_id, tracking_id, scan_type, facility_id, geo:{lat,lng},
          event_ts, device_id, raw_payload }
  -> 202 { accepted:true, event_id }        # durably logged, async projected
  -> 200 { accepted:true, duplicate:true }   # idempotent replay, no-op
  -> 400 { error:"malformed" }

POST /ingest/v1/scan/batch                   # offline scanner flush
  body: { events:[ {event_id, tracking_id, scan_type, ...}, ... ] }
  -> 202 { accepted:N, duplicates:M }

Customer-facing (read)

GET /api/v1/track/{tracking_id}
  -> 200 { tracking_id, status:"OUT_FOR_DELIVERY",
           last_facility:"Delhi-Okhla-Hub", eta:"2026-11-15T18:00+05:30",
           updated_at:.. }

GET /api/v1/track/{tracking_id}/history
  -> 200 { tracking_id, events:[
             { ts, status:"PICKED_UP", facility:"Pune-Hinjewadi" },
             { ts, status:"IN_TRANSIT", facility:"Mumbai-Bhiwandi" },
             { ts, status:"AT_HUB", facility:"Delhi-Okhla-Hub" }, ... ] }

Subscriptions (apps / merchants)

# live app connection
WS   /api/v1/subscribe        -> { subscribe:[tracking_id,...] }
                              <- push { tracking_id, status, facility, ts }

# merchant webhook registration
POST /api/v1/webhooks
  body: { merchant_id, tracking_id | "*", callback_url, secret }
  -> 201 { webhook_id }
  # delivery: POST callback_url { event_id, tracking_id, status, ts, signature }
  #           at-least-once, retried with backoff, idempotent on event_id

Data stores - the right tool per plane

1. Event log - partitioned, replicated append-only log (Kafka-style). Source of truth. The ingest write is a high-volume durable append with per-parcel ordering - exactly a log’s strength. Partitioned by hash(tracking_id), replicated x3, retained for the online window then archived.

event log  (partition = hash(tracking_id) % N, append-only)
  event_id        (dedup key)
  tracking_id
  scan_type       PICKUP | HUB_IN | HUB_OUT | OUT_FOR_DELIVERY | DELIVERED
                  | EXCEPTION | RETURN | ...
  facility_id, geo
  event_ts        (device scan time - used for ordering)
  receive_ts      (gateway arrival time)
  device_id, carrier_leg, raw_payload

2. Current-state store - in-memory KV (Redis-style), durable-backed, sharded by tracking_id. Point lookups by tracking_id, ~50GB hot, served < 10ms. Updated by projection workers; fronted by a read-through cache. NoSQL KV is right - it is a keyed snapshot, no joins, no transactions across parcels.

current_state  (KV, shard by tracking_id)
  tracking_id         PK
  status              enum
  rank                int        -- monotonic guard
  last_facility
  eta
  applied_event_ids   compact set   -- idempotency within a parcel
  event_count, updated_at

3. History store - wide-column (Cassandra/Bigtable-style), partition by tracking_id, cluster by event_ts. The timeline read is an ordered range scan within one partition - the native query shape for wide-column. Append-only; scales horizontally by tracking_id.

history  (wide-column)
  partition key: tracking_id
  clustering key: event_ts
  columns: status, scan_type, facility_id, geo, event_id, flags(late/illegal)

4. Subscription registry - fast KV. tracking_id -> {connection_ids / gateway_nodes} and tracking_id -> {webhook_ids, device_tokens}. Hot, ephemeral for live connections, durable for webhook registrations.

5. Cold archive - object store / warehouse, partitioned by date. Raw events for delivered parcels past the hot window, for audit, disputes, and analytics. Write-once, read-by-range - cheap columnar storage.

Why a log for ingest, KV for current state, wide-column for history, object store for archive: the planes have opposite needs. Ingest is a bursty durable append with per-key ordering - a partitioned log. Current state is a tiny high-read keyed snapshot - in-memory KV. History is an unbounded ordered-by-time per-key list - wide-column range scan. Archive is write-once read-rarely - cold object storage. There is no single “tracking database”; each plane gets the engine whose physical layout matches its access pattern. The relational option is deliberately avoided for the hot path - no cross-parcel transactions are needed, and a row-update model would reintroduce the mutable-status corruption that event sourcing exists to prevent.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. Synchronous ingest coupling to the state DB (breaks first). Writing scans straight into the state store makes ingest availability hostage to the slowest downstream component, and a DB hiccup drops scans. Fix: durable append to a partitioned log on the critical path, project asynchronously. The log is the shock absorber; a slow projector buffers backlog instead of losing events. This is the single most important decoupling.

2. Hot partition / hot parcel. hash(tracking_id) spreads parcels evenly, but a viral tracking id (a celebrity’s package, a widely-shared shipment) can concentrate reads. Fix: reads never hit the log - they hit the cache and current-state KV, which are replicated so a hot key is served from many replicas. Write-side, a single parcel only fires ~8 events in its life, so there is no write hot-parcel problem; partition skew is handled by enough partitions and consistent hashing.

3. Projection worker falling behind under a sort burst. A hub sorting at night pushes a 4x spike into a few partitions; a single consumer per partition lags. Fix: scale projection workers with partition count (one consumer group, N partitions, workers added up to N). Per-parcel ordering is preserved because all of a parcel’s events are in one partition. The log retains backlog; p95 push may degrade to seconds during a burst but no event is lost and the state converges. Over-partition up front so you can add workers without repartitioning.

4. Out-of-order / late events corrupting state. Covered in depth in deep-dive 1. Fix: monotonic rank + event-time ordering + idempotent apply on event_id. The current status advances only forward through legal transitions; late/illegal events are recorded in history (flagged) but never regress the live status. Clock skew is defanged because ranks, not raw timestamps, decide the canonical state.

5. Read spike at evening peak. Everyone checks their package after work - 8-10k reads/sec, concentrated on in-flight parcels. Fix: read-through cache in front of the current-state KV, invalidated proactively on status change so it is never stale by more than one push. The hot set is small and repeatedly read - near-100% cache hit on current-status. History (colder) is served from the wide-column store’s replicas.

6. Polling waste / push fan-out load. Naive polling is ~4000:1 wasted reads; push to millions of connections is its own scaling problem. Fix: push only on real transitions through a dedicated plane keyed by a subscription registry, with APNs/FCM for backgrounded apps and the current-state store as the heal-on-reconnect backstop. Gateways are stateless and horizontally scaled; the registry routes a change only to gateways holding interested connections.

7. Event log as a single point of failure / throughput ceiling. Everything flows through it. Fix: the log is partitioned and replicated (x3, quorum acks), multi-broker, and multi-region for the ingest tier. It is designed for exactly this append throughput; 25k/sec is well within a modest cluster. Partition count sets the ceiling - provision generously and it scales linearly.

8. Storage growth (~55TB/year of raw events before replication). Fix: retain only the hot window on the log and KV; a parcel delivered past its retention moves to cold object-store archive partitioned by date. Current-state and history keep only active + recently-delivered parcels hot. Append-only everywhere means archival is a clean partition move, no in-place rewrites.

9. Merchant webhook endpoints that are slow or down. Fix: webhook delivery is at-least-once from a durable queue with exponential backoff and a dead-letter queue; a failing merchant endpoint cannot block the push plane or other merchants. Payloads carry event_id so merchants dedup. Persistent failures surface for the merchant to fix without affecting anyone else.

10. Exactly-once effect under retries across all layers. Fix: stable event_id enforced idempotent at the gateway (dedup cache), the projection (applied-set), and the webhook consumer. Duplicate scans, retried pushes, and replayed webhooks all collapse to one effect. Correctness depends on the projection’s durable applied-set; earlier dedup layers just cut traffic.

11. State rebuild / corruption recovery. If the current-state KV is lost or a projection bug writes bad state, you must recover. Fix: because the event log is the immutable source of truth, you replay the log through the (fixed) state machine to rebuild the current-state and history projections deterministically. This is the superpower of event sourcing - the derived stores are disposable and reconstructable; only the log must be durable. Rebuild runs per-partition in parallel.

12. Multi-region and data locality. Scanners and customers span regions. Fix: run ingest regional (scanners write to the nearest log cluster), replicate the current-state KV and history to the regions that serve those customers, and route reads to the nearest replica. Because per-parcel state only ever advances by rank and is idempotent on event_id, cross-region replication can be asynchronous without breaking correctness - a region seeing events slightly late still converges to the same monotonic state.

Wrap-Up

The trade-offs that define this design:

  • Event sourcing, not mutable status. The scan events are the immutable source of truth; the parcel’s status is a derived projection. This is what makes out-of-order, duplicate, and late events survivable, makes the full timeline auditable, and makes every derived store disposable and rebuildable by replaying the log.
  • Monotonic, rule-based state machine. Status advances only forward through legal transitions, ordered by event-time and guarded by rank, idempotent on event_id. A parcel never goes backwards and never double-notifies, no matter how messy the event stream is - ranks beat raw clocks, so scanner clock skew cannot un-deliver a package.
  • Four separated planes on one log. Ingest (durable append), projection (state machine), serving (precomputed read stores), and push (subscription fan-out) each scale independently off the shared event log, so a read spike, a sort burst, or a slow webhook never starves the others.
  • Push for latency, the state store for truth. Real-time updates go out on real transitions through a dedicated fan-out plane; a missed push self-heals because the client re-reads authoritative current status from the KV store on reconnect. Polling is a degraded fallback, not the mechanism.
  • The right engine per access pattern. A partitioned log for bursty ordered ingest, in-memory KV for hot point-lookup current status, wide-column for ordered-by-time history range scans, cold object store for archive. No single database; each plane gets the layout its query shape wants.

One-line summary: a package tracker where scan events are durably appended to a per-parcel-partitioned log as the source of truth, folded by an idempotent monotonic state machine into a hot current-state KV and a wide-column history store served at low latency, with real-time status changes fanned out to subscribers through a dedicated push plane - and the whole derived state rebuildable at any time by replaying the immutable event log.