Design a Spam / Fraud Detection Pipeline - System Design

Everyone thinks fraud detection is a set of if statements. “Block logins from Nigeria, flag transactions over $5000, rate-limit signups per IP, done.” Then the interviewer piles on the constraints that make it a real system: it is not a rulebook run once a night, it is a synchronous gate in front of every login, every payment, every signup, and every message send - so it must return a verdict in the tens of milliseconds before the user’s request proceeds; it is not thousands of events, it is billions of events per day; the signal it needs (this card was used in three countries in the last hour, this device fingerprint has 40 accounts, this IP just failed 200 logins) lives in history, so scoring means joining live events against a huge, constantly-updating memory; and the fraudsters adapt, so the model that caught them yesterday is stale today, which is why the model has to be retrained daily on fresh labels and hot-swapped without downtime.

A spam/fraud pipeline is deceptive because a handful of hardcoded rules demos beautifully on a slide and then falls apart the moment you have adaptive adversaries, a hard sub-100ms budget on the request path, billions of events, and a feature that needs the last hour of a user’s history computed before the request times out. The whole design comes down to four decisions: how you score synchronously in under 100ms without ever letting the scorer become a single point of failure that takes down login (a fast, fail-open inference tier), how you compute behavioral features over history (a card’s velocity, an IP’s failure rate) fresh enough to matter but fast enough to read in a few milliseconds (an online feature store fed by a streaming aggregator), how you combine cheap deterministic rules with an expensive ML model so the common good case is cheap and only the ambiguous case pays for the model (a layered rules-then-model cascade), and how you keep the model current against adapting fraud without training-serving skew (a daily retraining loop with a shared feature definition and shadow evaluation). All of it while the p99 stays under 100ms and a scorer outage never blocks a legitimate login.

Let me do it properly.

Functional Requirements (FR)

In scope:

• Score every incoming event for fraud/spam probability. For each event - a login attempt, a transaction, a signup, an outbound message - return a risk score in [0, 1] and a decision (allow, challenge, block, review) synchronously, before the action completes. This is the core gate.
• Use behavioral and historical signal, not just the event. The score must reflect context: how many transactions this card made in the last hour, whether this device has spun up 50 accounts, whether this IP is failing logins in a burst. That means joining the live event against aggregated history.
• Layered decisioning - rules plus model. Support fast deterministic rules (allow/deny lists, hard velocity caps, known-bad fingerprints) and a probabilistic ML model for the grey area, combined into one verdict with a clear precedence.
• Feedback and labels. Ingest ground-truth labels - chargebacks, user “this is spam” reports, manual review outcomes, confirmed account takeovers - and feed them back so the model learns. Labels arrive late (a chargeback lands weeks after the transaction).
• Daily retraining and safe rollout. Retrain the model on fresh labels daily, evaluate it offline and in shadow, and roll it out without downtime and without regressing on caught fraud.
• Analyst tooling. A queue of review-flagged events for human analysts, with the features and score that drove the decision, and the ability to label the outcome (which becomes training data).

Explicitly out of scope (say it so you own the scope):

• Case management / chargeback dispute workflow. The downstream ops process of fighting a chargeback with the bank is a separate operational system. We produce the risk signal and the review queue; we do not run the dispute.
• Payment authorization / settlement itself. Moving the money is the payment system’s job. We are a gate in front of it that returns allow/block; we do not debit accounts. (See a payments design for that half.)
• The model architecture bake-off. Whether the model is gradient-boosted trees, a logistic model, or a graph neural net is a data-science decision. We design the pipeline that trains, serves, and feeds any such model, and we assume a tree-based model (fast, interpretable, strong on tabular fraud features) as the default.
• Identity verification / KYC. Document checks and government-ID matching at onboarding are a separate vertical. We consume their outputs as features; we do not run them.
• Full graph fraud-ring detection. Linking accounts into rings via a graph database is a powerful adjacent system; we note where it plugs in as a feature source but do not design the graph store in depth.

The one decision that drives everything: fraud scoring is a synchronous, low-latency gate that must join a live event against a large, fast-changing history and run an adapting model - so the design is dominated by a hard latency budget, an online feature store, a rules-then-model cascade, and a daily retraining loop, not by any single clever model. Everything below follows from that.

Non-Functional Requirements (NFR)

• Scale: on the order of 5B scored events per day across all event types (logins, transactions, signups, messages) at a large platform. That averages ~58K events/sec, but traffic is spiky (business hours, sale events, credential-stuffing attacks that arrive as a wall), so peak is ~300K events/sec. Fraud is rare - typically well under 1% of events are actually fraudulent - so the model lives in an extreme class-imbalance world.
• Latency: the score must return inside the caller’s budget. Target p99 under 100ms for the whole score call, which in practice means a feature-fetch budget of ~10-20ms and a model-inference budget of ~10-30ms, with headroom. The synchronous path is on the critical path of login/checkout, so every millisecond is user-facing friction.
• Availability: the scorer sits in front of revenue-critical flows, so its availability contract is subtle: it must be highly available (99.99%+), and critically it must fail open for the good path. If the scoring service is down, we do not block every login on the planet - we fall back to rules-only or allow-with-flag, because blocking all legitimate users is a worse outage than briefly letting some fraud through. (High-assurance flows like a large money movement may fail closed to review instead - a per-flow policy.)
• Consistency: features are eventually consistent - a velocity counter being a second or two behind is acceptable and unavoidable. The decision must be deterministic and auditable: given the same event and the same feature snapshot and model version, we must be able to reproduce exactly why we allowed or blocked, for compliance and dispute.
• Durability: every scored event, its feature snapshot, its score, and its decision are logged immutably - this is both the audit trail (why did we block this customer) and the training data (the features exactly as the model saw them). Losing this log means losing our ability to explain decisions and to train without skew.
• Adaptivity: the adversary adapts, so model freshness is a first-class requirement - retrain daily, and be able to push emergency rules within minutes when an attack pattern is spotted.

Back-of-the-Envelope Estimation (BoE)

Let me ground the design in numbers.

Traffic:

Scored events/day:            5,000,000,000   (5B)
Average QPS = 5B / 86,400 sec  ≈ 58,000 QPS   (misleading average)
Traffic is spiky - business hours + sale events + attack bursts:
Peak scoring QPS (say ~5x avg) ≈ 300,000 QPS   (the number we size for)

Fraud base rate:  < 1% of events. Of 5B events, maybe tens of millions
                  are actually fraud. Extreme class imbalance -> the model
                  optimizes for recall on a tiny positive class, and raw
                  accuracy is a useless metric (99% "accuracy" = block nothing).

The daily average of ~58K QPS is a trap. Credential-stuffing and card-testing attacks do not arrive smoothly - they arrive as a coordinated wall of hundreds of thousands of attempts in minutes. We size the scoring tier and the feature store for ~300K events/sec, not 58K, and we design the rules layer to shed that wall cheaply before it reaches the model.

Read vs write on the scoring path:

Every scored event is BOTH a read and a write:
  READ:  fetch ~100-300 features for this entity set (user, card, device, IP)
         from the online feature store.  300K events/sec * ~200 feature reads
         (batched into a few multi-key lookups) -> the feature store must
         serve on the order of tens of millions of key-values/sec.
  WRITE: append the event + feature snapshot + score + decision to the log,
         and update the streaming aggregates (velocity counters) for next time.
         300K writes/sec into the event log, plus counter updates.

Feature-store read latency is the tightest budget: it is on the p99 path.

The scoring path is read-dominated per event (fetch many features) but every event also writes back to update the counters that future events will read - so the feature store is simultaneously a very-high-QPS read store and a high-write-rate aggregate store. That dual role, under a 10-20ms read budget, is the hardest infrastructure constraint in the design.

Storage:

Immutable scored-event log (features snapshot + score + decision):
  ~2 KB/event (a few hundred features + metadata, packed).
  5B/day * 2 KB  ≈ 10 TB/day of scored-event records.
  Retain ~90 days hot for training + audit, tier the rest to cold object storage.
  90 days ≈ ~900 TB hot. Fat but sequential and cheap in columnar form.

Online feature store (the hot, low-latency state):
  Keyed by entity: user_id, card_id, device_id, ip.
  Say hundreds of millions of active entities * ~1 KB of features each
  ≈ low hundreds of GB to a few TB of HOT key-value state. Fits in a
  sharded in-memory / SSD-backed KV cluster (Redis / Aerospike / Cassandra).

Model artifacts: a tree-model is tens of MB. Trivial to store and to ship
  to every scorer. Versioned; keep the last N for rollback and audit.

The split is the same shape as any good pipeline: a fat immutable log of everything we scored (for training and audit) and a thin, hot feature store the scorer actually reads on the request path. The log is big and cheap; the online state is small and expensive (it must be fast).

Bandwidth and compute:

Feature fetch: 300K events/sec, each pulling ~200 features batched into a
  handful of multi-get calls -> the feature store cluster fans this across
  many shards. The scarce resource is p99 read latency, not raw bandwidth.

Model inference: a tree model scores in ~1-10ms on CPU. 300K/sec across the
  scorer fleet is modest CPU - the model is cheap; the FEATURE FETCH is the
  cost. (A deep net would flip this and need GPU batching; we choose trees
  partly to keep inference cheap and synchronous.)

Training: daily job over ~90 days of labeled events. Batch, offline, hours
  of cluster time. Not on any latency budget - it just must finish daily.

Inference is cheap; the expensive things are fetching many fresh features within the latency budget and keeping those features consistent between training and serving. Size for those.

High-Level Design (HLD)

The system has three planes that share one contract. The serving plane is a synchronous, low-latency gate: it fetches features, runs rules then the model, returns a decision, and logs everything. The streaming plane consumes the same events asynchronously and updates the behavioral aggregates (velocity counters) that the serving plane reads next time. The training plane consumes the immutable log plus late-arriving labels, retrains daily, and ships a new model. The glue that stops the whole thing from silently rotting is a single shared feature definition used by both serving and training, so the model is trained on the same features it sees in production.

                        SYNCHRONOUS SCORING PATH  (p99 < 100ms, fail-open)
   ┌───────────────────────────────────────────────────────────────────────────┐
   │  Client action (login / txn / signup / message)                            │
   │        │  POST /v1/score  {event}                                          │
   │   ┌────▼─────────┐                                                          │
   │   │  API Gateway │  auth, rate-limit                                        │
   │   └────┬─────────┘                                                          │
   │   ┌────▼─────────────────┐    (1) rules first (cheap, deterministic)        │
   │   │  Scoring Service     │───► Rules Engine  (allow/deny lists, hard caps,   │
   │   │  (stateless fleet)   │      known-bad fingerprints)  -- may short-circuit│
   │   │                      │    (2) fetch features (batched multi-get)         │
   │   │                      │◄──► Online Feature Store  (Redis/Aerospike, KV)   │
   │   │                      │    (3) run ML model on features                   │
   │   │                      │───► Model Runtime (in-proc tree model, versioned) │
   │   │                      │    (4) combine -> score + decision                │
   │   └────┬─────────────────┘                                                  │
   │        │  return {score, decision, reason, model_version}                   │
   │        │  (5) async: emit scored-event to Kafka  (never blocks the response)│
   └────────┼──────────────────────────────────────────────────────────────────┘
            │
      ┌─────▼──────────────────────┐   scored-events topic
      │   KAFKA (event backbone)   │◄──────────────── labels topic (chargebacks,
      │   scored-events | labels   │                   user reports, review outcomes)
      └───┬───────────────────┬────┘
   stream │ (features)        │ archive (training + audit)
   ┌──────▼───────────┐   ┌───▼──────────────────────┐
   │ Streaming Feature│   │  Immutable Event Log      │
   │ Aggregator(Flink)│   │  -> Object store (S3),    │
   │ - velocity counts│   │     columnar, partitioned │
   │ - windowed aggs  │   └───┬──────────────────────┘
   │ - upsert online  │       │  daily batch
   └──────┬───────────┘   ┌───▼──────────────────────┐
          │ writes back   │  Training Pipeline        │
   ┌──────▼───────────┐   │  - join events + labels   │
   │  Online Feature  │◄──┤  - build feature set from │
   │  Store (hot KV)  │   │    SHARED definitions     │
   └──────────────────┘   │  - train, evaluate, shadow│
                          └───┬──────────────────────┘
                    new model │  (offline metrics + shadow pass)
                       ┌──────▼───────────┐
                       │  Model Registry  │ --> hot-swap into Scoring fleet
                       │  (versioned)     │     (canary -> full rollout)
                       └──────────────────┘
                          ▲
                   ┌──────┴────────┐  review queue
                   │ Analyst Tool  │  (score, features, reason) -> label -> labels topic
                   └───────────────┘

Request flow - scoring an event (the synchronous hot path):

1. The client action calls POST /v1/score with the event (type, actor IDs, amount, device fingerprint, IP, timestamp). The API gateway authenticates and hands it to a stateless Scoring Service instance.
2. Rules first. The scorer runs the cheap deterministic Rules Engine: allow/deny lists, hard velocity caps, known-bad device/IP fingerprints, and hard business constraints. A hit here can short-circuit immediately - a blocklisted card is blocked without ever fetching features or running the model, which is what lets the rules layer absorb an attack wall cheaply.
3. Fetch features. For the ambiguous majority, the scorer fetches the features it needs from the Online Feature Store, batched into a few multi-key lookups across the entities involved (user, card, device, IP). This is the tightest latency budget on the path.
4. Run the model. The scorer runs the versioned ML model in-process on the fetched features, producing a probability in [0, 1].
5. Combine and decide. Rules and model score combine by a defined precedence into a final score + decision (allow / challenge / block / review), and the response returns - all inside the p99 budget.
6. Log asynchronously. After responding (never before), the scorer emits the scored-event - the event, the exact feature snapshot it used, the score, the decision, the model version, and the triggering rules - to Kafka. This is fire-and-forget so logging never adds latency to the user’s request.

Request flow - the asynchronous paths:

1. Feature update (streaming plane): the Streaming Feature Aggregator (Flink) consumes the event stream and maintains windowed behavioral aggregates - transactions per card in the last 1h/24h, failed logins per IP in the last 5m, accounts per device - and upserts them into the Online Feature Store. So the very act of scoring this event updates the counters the next event will read. This is the feedback loop that makes velocity features work.
2. Label ingestion: ground-truth labels arrive late and asynchronously - chargebacks from the payment processor (weeks later), user spam reports, analyst review outcomes, confirmed ATOs - onto the labels topic. They are joined to the original scored-event by ID.
3. Training plane: a daily batch job reads the immutable event log (features exactly as served) joined to the labels, builds the training set from the same shared feature definitions the scorer uses, trains a new model, evaluates it offline and in shadow, and publishes it to the Model Registry. The Scoring fleet hot-swaps to the new version via canary then full rollout.

The key architectural insight: the features the model sees at serving time and the features it is trained on must be produced by the same definition, or the model silently mis-scores production. That single constraint - avoiding training-serving skew - is why there is a shared feature store and shared feature code rather than two hand-written feature pipelines, and it is the thing most naive designs get wrong.

Component Deep Dive

Four hard parts: (1) computing behavioral features over history fresh and fast (the feature store), (2) scoring synchronously under 100ms without becoming a single point of failure (the serving tier), (3) combining rules and a model so the common case is cheap and the decision is auditable (the cascade), and (4) keeping the model fresh against an adapting adversary without training-serving skew (the retraining loop). I will walk each from naive to scalable.

1. Behavioral Features: Query-on-Read -> Streaming Feature Store

The model’s power comes from context: “this card made 12 transactions in the last hour” or “this IP has failed 300 logins in 5 minutes.” Computing that on every request is the hard part.

Approach A: Query the transaction database at scoring time (the naive instinct)

When scoring, run queries against the primary datastore to compute the features on the fly:

-- velocity: transactions for this card in the last hour
SELECT COUNT(*) FROM transactions
WHERE card_id = ? AND ts > now() - interval '1 hour';
-- failed logins for this IP in the last 5 minutes
SELECT COUNT(*) FROM logins
WHERE ip = ? AND success = false AND ts > now() - interval '5 minutes';

Where it breaks: you cannot run these aggregate scans inside a 20ms budget at 300K QPS. Each score would fire several range-scan queries against the production OLTP database, adding tens to hundreds of milliseconds and hammering the very database that serves real traffic. During a card-testing attack - exactly when you most need velocity features - the attack multiplies the query load and the scoring path takes the database down with it. Computing history-over-time by scanning history at read time does not fit a synchronous latency budget. Wrong shape.

Approach B: Precompute features with a streaming aggregator, read them from a hot KV store

Split “compute the aggregate” (continuous, async, off the request path) from “read the aggregate” (a single fast key lookup on the request path). A streaming aggregator (Flink) consumes the event stream and maintains the counters incrementally; the scorer just reads the current value:

Streaming aggregator (continuous):
  keyBy(card_id).window(sliding 1h).aggregate(count, sum(amount))
  keyBy(ip).window(sliding 5m).aggregate(count where !success)
  keyBy(device_id).aggregate(distinct account_ids)
     -> upsert into Online Feature Store:  feature:card:{id} -> {txn_1h: 12, ...}

Scorer (per request):
  multiget( feature:user:{u}, feature:card:{c}, feature:device:{d}, feature:ip:{ip} )
     -> ~1-5ms, one round trip per entity, batched

• The online feature store is a sharded, low-latency KV store (Redis / Aerospike / a Cassandra tier) keyed by entity ID. Reads are single-key lookups the scorer batches into a few multi-gets - milliseconds, not scans. It holds the current values of every feature, updated by the stream, not the raw history.
• Windowed aggregates, incrementally maintained. The aggregator keeps only the running counters per key (a sliding window of counts/sums), not the events - so state is bounded and updates are cheap. Sliding or hopping windows give “last 1h/24h” without re-scanning.
• Two feature classes. Streaming features (velocities, counts, rates) update in seconds via the aggregator. Batch features (this account’s 90-day chargeback rate, lifetime spend, tenure) are recomputed daily by a batch job and written to the same store. The scorer does not care which pipeline produced a feature - it reads one flat feature vector per entity. This is the classic dual-pipeline feature store.
• Freshness vs cost knob. The sliding-window granularity and update frequency trade freshness against aggregator cost. Velocity counters need to be seconds-fresh to catch a burst; a 90-day spend average can lag by a day. Match each feature’s refresh to how fast it actually moves.
• On-demand features. A few features depend on this request and cannot be precomputed (distance between this login’s geo and the last login’s geo, amount relative to the account’s mean). These are computed in the scorer from the fetched raw state - cheap arithmetic on already-fetched values, not a new query.

The move is the same as every real-time system: turn an expensive read-time aggregation into a cheap read of a precomputed value, by paying the cost continuously in a stream instead of all at once on the request.

2. Synchronous Scoring Under 100ms Without a Single Point of Failure

The score is a gate on login and checkout. It must be fast, and - more subtly - it must not become the thing that takes those flows down.

Approach A: A central scoring service the request calls and waits on, hard (naive)

Every login/checkout makes a blocking call to a scoring service and refuses to proceed until it gets a verdict. If the scorer is slow or down, the caller waits or errors.

Where it breaks: two failure modes. First, latency coupling - if the scorer’s p99 spikes (a slow feature-store shard, a GC pause, a model reload), every login inherits that latency, and a 300ms scorer stall becomes a 300ms checkout stall for everyone. Second, and worse, availability coupling - if the scoring service is down and the caller blocks on it, then a scorer outage blocks every login and every payment on the platform. You have taken a fraud tool and turned it into a global kill switch for revenue. Making a synchronous dependency mandatory means its failures are your failures.

Approach B: A fast stateless scorer with strict timeouts and a fail-open contract

Keep it synchronous but engineer the failure behavior explicitly:

• Stateless, horizontally scaled scorer fleet. Each instance holds the model in-process (tens of MB) and is otherwise stateless, so we scale to 300K QPS by adding instances behind a load balancer and lose nothing when one dies. Model in-process means inference is a local function call, not a network hop - shaving a round trip off the budget.
• In-process model, not a remote model server, for the hot path. A tree model runs in ~1-10ms locally. A separate model-serving tier would add a network hop and a second thing to keep up. Co-locating the model with the scorer keeps the budget tight. (A heavy deep model that needs GPU batching would justify a separate inference tier, which is a reason to prefer trees here.)
• Hard timeouts with graceful degradation. Every downstream call (feature fetch, model) has a strict timeout well inside the 100ms budget. If the feature store is slow, the scorer proceeds with whatever features it got plus defaults for the rest, or falls back to rules-only. A partial score beats a timeout.
• Fail open (for the common path), by policy. If the whole scorer is unreachable, the caller does not block - it applies a fallback policy: allow (optionally flagged for async review) for low-risk flows, or challenge (step-up auth) for medium flows. This is a deliberate risk trade: briefly letting some fraud through is a far smaller loss than blocking all legitimate users. High-assurance flows may fail closed to review instead - a large money movement would rather be held than let through blind. The fail-open-vs-closed choice is per flow, set by the caller.
• Async logging off the response path. The scored-event log write goes to Kafka after the response is returned, fire-and-forget with local buffering. Logging is essential but it is not allowed to add latency or to fail the score - if Kafka is slow, we buffer and keep serving.
• Shed load at the rules layer. During an attack wall, the cheap rules layer blocks known-bad traffic before the feature fetch and model, so the expensive path only runs for traffic that passed the cheap filter. This keeps the model tier from being overwhelmed exactly when volume spikes.

score(event):
   verdict = rules.evaluate(event)              # ~sub-ms, may short-circuit
   if verdict.terminal: return verdict          # blocklist hit, hard cap -> done
   try:
       features = feature_store.multiget(entities, timeout=15ms)
   except Timeout:
       features = partial_or_defaults            # degrade, do not fail
   p = model.score(features)                     # ~1-10ms, in-process
   decision = combine(verdict, p, policy)        # precedence rules
   respond(decision)                             # <-- user unblocked here
   kafka.emit_async(event, features, p, decision) # after responding

Put together: a stateless in-process-model scorer, strict timeouts with feature-degradation, a fail-open (or per-flow fail-closed) contract, and cheap rules that shed attack load - that is how you stay under 100ms and make sure the fraud gate can never become the outage that stops all logins.

3. Rules + Model Cascade: Cheap Common Case, Auditable Decision

We have two decision sources - deterministic rules and a probabilistic model. Combining them well is both a performance and a correctness problem.

Approach A: Run the model on everything and treat its score as the decision (naive)

Every event fetches features and runs the model; the decision is just score > threshold.

Where it breaks: three ways. Cost - you pay the full feature-fetch-plus-model cost even for obviously-fine traffic and obviously-bad traffic (a blocklisted card still gets a full model run), wasting the expensive path on cases a cheap check settles. Rigidity - when an attack pattern is spotted at 2am, you cannot wait for a daily retrain to respond; you need a rule you can push in minutes. Opacity - a bare model score is hard to explain to a customer, a regulator, or an analyst (“why was I blocked?” “the model said 0.83” is not an answer). A model-only design is slow, slow to adapt, and unauditable.

Approach B: A layered cascade - rules gate first, model for the grey area, rules can override

Order the decision by cost and certainty:

1. HARD RULES (deterministic, sub-ms, no features needed)
     - deny list (card/device/IP/account) -> BLOCK, short-circuit
     - allow list (trusted partner)        -> ALLOW, short-circuit
     - hard business caps (txn > $X w/o 2FA)-> CHALLENGE, short-circuit
   -> settles the certain cases cheaply, sheds attack load, gives fast response.

2. MODEL (probabilistic, for everything that survived the rules)
     - fetch features, run model -> p in [0,1]

3. DECISION BANDS + OVERRIDE RULES
     p < 0.2            -> ALLOW
     0.2 <= p < 0.8     -> CHALLENGE (step-up auth / captcha / OTP)
     p >= 0.8           -> BLOCK  (or REVIEW for high-value)
     override rules can still bump a band (e.g. new-attack signature -> BLOCK
       regardless of p; VIP account -> never auto-block, route to REVIEW)

• Rules are the fast, adaptive, explainable layer. They short-circuit certain cases (no feature fetch, no model), they are pushable in minutes as a config change (the emergency response to a live attack), and they give a human-readable reason for every hard decision. They are also where hard compliance constraints live (sanctions lists, regulatory caps).
• The model is the grey-area layer. It only runs on traffic the rules did not settle - which is most legitimate traffic but a smaller absolute cost since the attack wall was shed. It handles the nuanced “this looks slightly off” cases rules cannot express.
• Decision bands, not a single threshold. Fraud is not binary; the cost of a false positive (blocking a real customer) differs from a false negative (letting fraud through). Three bands let the medium-risk case get a challenge (step-up auth) instead of a hard block - recovering legitimate users who trip the model while still stopping the confident-bad case. Thresholds are tuned per event type and per segment against the business cost of each error.
• Auditable by construction. Every decision logs the triggering rule(s), the model score, the model version, and the feature snapshot. “Blocked because deny-list rule R17 matched device fingerprint X” or “challenged because model v2026-07-29 scored 0.61 driven by card_txn_1h=14.” That record is the answer to the customer, the regulator, and the analyst, and it is the training data.
• Precedence is explicit. Hard allow/deny rules beat the model; override rules can escalate a band; the model fills the middle. Writing that precedence down (rather than letting it emerge) is what makes the combined decision deterministic and reproducible.

The cascade is why the system is fast (cheap cases short-circuit), adaptive (rules push in minutes, model retrains daily), and auditable (every decision has a stated reason) - three things a bare model cannot deliver alone.

4. Daily Retraining Without Training-Serving Skew

The adversary adapts, so a static model decays. Retraining daily is easy to say and full of traps.

Approach A: Retrain on hand-built features, deploy when metrics look good (naive)

A data scientist writes a feature-extraction script for training, trains on historical data, checks AUC, and ships the model. Serving has its own feature code written by the platform team.

Where it breaks: training-serving skew. The training script and the serving code compute “transactions in the last hour” slightly differently - one counts inclusive of the current event, one uses a different window boundary, one has access to a label-leaking field the serving path does not. The model learns on features that do not match what production feeds it, so it scores worse in production than offline metrics promised, silently. Worse, label leakage: training on a feature that is only known after the outcome (e.g. “was this transaction refunded”) produces a model with a spectacular offline AUC that is useless live. And deploying straight to 100% on the strength of offline metrics means a bad model hits all traffic before anyone notices. Offline-good is not production-good.

Approach B: Shared feature definitions, point-in-time correct training, shadow + canary rollout

Make training and serving structurally share their features and prove the new model in production shadow before it decides anything:

• One feature definition, two runtimes. Features are defined once (in the feature store’s definition layer). The streaming/batch pipelines materialize them for serving; the training job reads the same definitions to build the training set. There is no second, hand-written feature script to drift. This is the single most important defense against skew.
• Point-in-time correctness (log the features as served). The cleanest way to guarantee the model trains on exactly what it saw: the scorer already logged the exact feature snapshot it used for every event (part 2). Training joins those logged snapshots to the labels - so the training features are, by construction, identical to serving features. No re-derivation, no window-boundary mismatch, no leakage of post-event fields (the snapshot was taken before the outcome existed).
• Late labels and the join. Labels arrive weeks late (chargebacks). The training job joins scored-events from ~90 days ago to labels that have since arrived, using the event ID. Events still within the label-maturation window are handled carefully (a transaction 3 days old may still be charged back - treat unlabeled-recent as provisional, not confirmed-good).
• Class imbalance handling. Fraud is under 1%, so the training job rebalances (undersample negatives / oversample or weight positives) and optimizes for PR-AUC / recall at a fixed low false-positive rate, not accuracy. The evaluation metric is the business one: caught-fraud-dollars at an acceptable customer-friction rate.
• Offline eval on a held-out future window. Evaluate on a time-split holdout (train on days 1-83, test on days 84-90), never a random split - a random split leaks future patterns into training and inflates metrics. The question is “does a model trained on the past catch next week’s fraud,” so the test set must be the future.
• Shadow mode before it decides. The new model runs in parallel with the live model on real traffic, scoring every event but not affecting decisions, for a day. We compare its scores/decisions to the incumbent and to labels as they arrive. This catches skew and regressions on live traffic with zero customer risk.
• Canary then full rollout. If shadow looks good, route a small slice (1-5%) of decisioning traffic to the new model, watch the block rate, challenge rate, and downstream fraud/chargeback signals, then ramp to 100%. Keep the previous version in the registry for instant rollback. Model versions are logged per decision, so a bad rollout is diagnosable and reversible.
• Monitoring for drift. Continuously watch feature drift (input distributions shifting) and score drift (the score distribution or block rate moving) - both are early warnings that either an attack changed or the model is going stale, and both can trigger an off-cycle retrain or an emergency rule.

daily:
  events = read_log(last 90d)                 # features EXACTLY as served
  labels = read_labels(matured)               # chargebacks, reports, reviews
  train_set = join(events, labels, on=event_id)   # point-in-time correct
  train_set = rebalance(train_set)            # handle <1% positive rate
  model = train(train_set)                    # tree model
  eval   = evaluate(model, time_split_holdout, metric=PR-AUC / recall@FPR)
  if eval.passes:
     registry.publish(model)                  # then SHADOW -> CANARY -> full

Put together: shared feature definitions, training on the exact logged serving-time snapshots, time-split evaluation, and shadow-then-canary rollout - that chain keeps the model both fresh (daily) and honest (no skew, no leakage, no blind 100% deploys). Freshness without that discipline just ships a broken model faster.

API Design & Data Schema

Scoring API (the synchronous gate)

POST /v1/score
  {
    "event_id":     "evt_9f13ac",        # unique id, minted by caller (idempotency)
    "event_type":   "transaction",       # login | transaction | signup | message
    "actor": {
       "user_id":   "usr_4471",
       "card_id":   "card_88ab",         # entity ids we key features on
       "device_id": "dev_ff21",
       "ip":        "203.0.113.9",
       "session_id":"sess_7c"
    },
    "context": {
       "amount":    4999.00,             # event-specific fields
       "currency":  "INR",
       "merchant":  "mrc_12",
       "geo":       "IN-KA",
       "ts":        1753833600123
    },
    "fail_mode":    "open"               # open | closed  (per-flow policy)
  }

  200 OK
  {
    "event_id":     "evt_9f13ac",
    "score":        0.61,               # fraud probability in [0,1]
    "decision":     "challenge",        # allow | challenge | block | review
    "reasons":      [                   # auditable, human-readable
       "model:v2026-07-29 p=0.61",
       "top_feature: card_txn_1h=14 (high)",
       "top_feature: geo_mismatch=true"
    ],
    "model_version":"2026-07-29",
    "latency_ms":   38
  }

Notes:
  - Synchronous; returns inside the p99 100ms budget.
  - event_id is the idempotency key - a retried score returns the same verdict.
  - On scorer unavailability the caller applies fail_mode (open -> allow/flag,
    closed -> review) rather than blocking the user's action.

Feedback / label API (asynchronous, drives training)

POST /v1/label
  { "event_id": "evt_9f13ac", "label": "fraud",     # fraud | legit | spam
    "source": "chargeback", "labeled_at": 1755300000, "detail": {...} }
  202 Accepted  -> appended to the labels topic, joined to the event for training.

Sources: chargeback (payment processor), user_report (spam button),
         review_outcome (analyst), confirmed_ato (security ops).

Analyst / admin API

GET  /v1/review/queue?assignee=me&limit=50     # review-flagged events + features + score
POST /v1/rules                                  # push/update a rule (config, minutes to live)
GET  /v1/models                                 # list versions, metrics, current + canary
POST /v1/models/{version}/promote               # shadow -> canary -> full, or rollback

Data model

Four stores for four jobs - a hot feature store (serving reads), an event backbone (transport), an immutable log (training + audit), and a rules/registry config store.

Online feature store - KV (Redis / Aerospike / Cassandra), the hot serving state:

key:   feature:{entity_type}:{entity_id}     # e.g. feature:card:card_88ab
value: {                                       # a flat feature vector, upserted
   txn_1h: 14, txn_24h: 51, amt_sum_1h: 61200,
   failed_logins_5m: 0, distinct_devices_7d: 2,
   chargeback_rate_90d: 0.003, tenure_days: 412,
   updated_at: 1753833590
}
access:  multiget by entity keys; single-key reads, ~1-5ms.
sharding: by entity_id (consistent hashing). replicated for HA.
ttl:     streaming features refresh continuously; stale keys expire.

Event backbone - Kafka:

topic: scored-events   partition by event_id (or actor for locality), RF=3, acks=all
topic: labels          partition by event_id
purpose: decouple serving from the streaming aggregator, the log archiver,
         and the training pipeline; absorb 300K/sec spikes.

Immutable scored-event log - Object store (S3/GCS), columnar (Parquet):

path:   s3://fraud-events/dt=2026-07-30/hr=10/part-*.parquet
row:    { event_id, event_type, actor{...}, context{...},
          feature_snapshot{ ...exact features used... },   # point-in-time
          score, decision, reasons[], model_version, rules_fired[], ts }
purpose: THE training set (features as served) + the audit trail.
retain:  ~90 days hot for training; tier older to cold for compliance retention.

Why log the full feature snapshot per event and not just the raw event: it is the only way to guarantee point-in-time-correct training and to reproduce any past decision exactly. The event alone would force re-derivation of features, which reintroduces skew.

Rules + model registry - config store (a small SQL/versioned store):

rules(   rule_id PK, type, condition_json, action, priority,
         enabled, created_by, created_at, expires_at )      # pushable in minutes
models(  version PK, artifact_uri, metrics_json, status,     # shadow|canary|live|retired
         trained_at, feature_schema_version )
Both are small, versioned, and audited - every decision references the exact
rule ids and model version that produced it.

Why these choices: a KV store (not SQL) for online features because serving is single-key point lookups under a hard latency budget, which KV does in ~1ms and a relational aggregate scan cannot; columnar object storage for the log because training is a big analytical scan over columns (features) that Parquet reads cheaply, and it is immutable/rebuild-friendly; Kafka because 300K/sec of un-droppable events must be buffered and fanned out to three independent consumers; and a small SQL/versioned store for rules and models because those are low-volume, high-consistency config that must be auditable and instantly changeable. Each store does the one thing it is best at, and the log is the only precious, non-rebuildable asset.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Feature-fetch latency on the p99 path. Fetching many features per event at 300K QPS inside ~15ms is the tightest budget; a slow shard blows the whole score’s latency. Fix: a sharded, replicated in-memory/SSD KV feature store, batched multi-gets (one round trip per entity, not per feature), aggressive timeouts with defaults, and co-locating the feature store near the scorer fleet. Read replicas absorb read QPS; the scorer degrades gracefully rather than waiting.

2. Scorer as a single point of failure for login/checkout. A synchronous mandatory dependency can turn its outage into a platform-wide revenue outage. Fix: fail-open (per-flow, optionally fail-closed) contract, strict timeouts, and rules-only fallback. The gate degrades to cheap rules or allow-with-flag rather than blocking every user; high-assurance flows fail to review.

3. Attack-wall load spikes (credential stuffing, card testing). Traffic can 10x in minutes, aimed exactly at the expensive path. Fix: shed at the cheap rules layer - blocklists and hard caps short-circuit known-bad traffic before feature fetch and model, so the expensive tier only sees filtered traffic. Kafka absorbs the write spike; autoscale the stateless scorer fleet on QPS.

4. Hot entity / hot key. A targeted account, a shared corporate IP, or a payment-processor IP concentrates reads and counter-updates on one feature-store shard and one aggregator key. Fix: replicate hot keys across read replicas, and for hot aggregation keys sub-shard the counter (key + hash(sub)%K) and sum at read time. Cache the hottest feature values in the scorer’s local process cache with a short TTL.

5. Training-serving skew and label leakage. The model scores worse in production than offline, silently. Fix: one shared feature definition, train on the exact logged serving-time feature snapshots (point-in-time correct), time-split evaluation, and shadow mode before any traffic. The logged snapshot makes leakage structurally impossible (it predates the label).

6. Bad model rollout. A regressed daily model could block legitimate users at scale the moment it deploys. Fix: shadow -> canary -> full rollout with the incumbent kept for instant rollback, decisions tagged with model_version, and live guardrail metrics (block rate, challenge rate, downstream chargebacks) that auto-halt a bad ramp.

7. Late-arriving labels. Chargebacks land weeks after the event, so “is this fraud” is unknown for a while. Fix: a label-maturation window - join events to labels over ~90 days, treat recent-unlabeled as provisional not confirmed-legit, and re-train continuously so late labels flow in. Complement with fast proxy labels (user reports, review outcomes) that arrive sooner.

8. Feature/model drift over time. The adversary changes; yesterday’s model goes stale. Fix: daily retrain plus drift monitoring on input features and score distribution; a drift alarm triggers an off-cycle retrain and lets analysts push an emergency rule (minutes) as a stopgap while the model catches up.

9. Streaming aggregator lag or failure. If the feature aggregator falls behind, velocity features go stale and miss a live burst. Fix: Flink checkpointing of window state with Kafka offsets, autoscaled parallelism keyed by entity, and watermarks so windows are well-defined; on lag, the scorer still has slightly-stale features (degraded, not blind), and the batch path backfills.

10. Event-log growth (~10 TB/day) and cost. The immutable log cannot grow hot forever. Fix: columnar compression, ~90 days hot for training, tiering older data to cold object storage for compliance retention, and dropping raw payloads once the feature snapshot (the only thing training needs) is captured.

11. Reproducibility / audit under dispute. A regulator or customer asks “why did you block me on this date.” Fix: every decision is immutably logged with its feature snapshot, model version, and rule ids, so any past decision is exactly reconstructable - the audit story is a lookup, not a forensic reconstruction.

Wrap-Up

The trade-offs that define this design:

• Precomputed features over query-on-read. We pay for behavioral features continuously in a streaming aggregator and read them as one fast KV lookup, trading eventual-consistency in the counters for the ability to fetch history-derived signal inside a 15ms budget instead of scanning the database on every request.
• Fail open over fail safe (mostly). The fraud gate is synchronous but degrades to rules-only or allow-with-flag when unhealthy, trading a small window of let-through fraud for never turning the scorer into a global login/checkout outage - with high-assurance flows failing closed as the deliberate exception.
• Rules-then-model cascade over a bare model. Cheap deterministic rules settle the certain cases and shed attack load, the model handles the grey area, and decision bands map risk to allow/challenge/block - trading a single elegant score for speed, minute-scale adaptability, and an auditable reason on every decision.
• Daily retraining with skew discipline over a static or naively-retrained model. Shared feature definitions, training on the exact logged serving-time snapshots, time-split evaluation, and shadow-then-canary rollout keep the model both fresh against an adapting adversary and honest against training-serving skew - freshness without that discipline just ships a broken model faster.
• Derived state over precious state. The feature store, aggregates, and even the model are all rebuildable from the immutable scored-event log; only that log (features as served, plus labels) is precious, which is what makes training reproducible and any past decision auditable.

One-line summary: a stateless fail-open scoring fleet that runs a cheap rules cascade then an in-process model over features read in milliseconds from a streaming-fed online feature store, logs every decision with its exact feature snapshot to an immutable event log, and retrains daily on those snapshots joined to late-arriving labels with shadow-then-canary rollout - trading eventual-consistency in features and a small fail-open risk for sub-100ms scoring of billions of events a day against an adapting adversary.