Everyone pictures a CAPTCHA as a picture of squiggly text. “Render some distorted letters, ask the user to type them, check the string, done.” Then the interviewer adds the constraints that turn it into a real system: it is not one page, it is a service in front of every signup, login, password reset, and checkout across thousands of sites, so it must answer in the tens of milliseconds before the form submits; it is not thousands of solves, it is millions of challenges per second; the thing it is fighting adapts - the moment you ship a challenge, someone builds a solver, a farm of humans, or an ML model to beat it, so a static challenge is dead on arrival; and the verdict it returns crosses a trust boundary - the browser is hostile, so a naive “you passed” flag is trivially forged and replayed. The whole problem is adversarial by construction.

A CAPTCHA service is deceptive because the demo (render letters, check the answer) is a five-minute afternoon and the actual system is dominated by four hard decisions: how you tell human from bot at all when the best signal is behavior not a puzzle (a risk-scoring pipeline over browser and interaction signals, not a squiggly image), how you return a verdict the origin server can trust across a hostile browser without it being forged or replayed (a signed, single-use, short-lived verification token verified server-to-server), how you serve millions of challenges a second globally under a tight latency budget without your own service becoming the outage that blocks every login on the internet (an edge-served, stateless, fail-open fleet), and how you keep winning against an adversary who studies every challenge you ship (a feedback loop that mines evasion patterns and rotates challenges and models continuously). All of it while a real grandmother logging in never gets blocked and a bot farm paying humans two cents a solve still loses money.

Let me do it properly.

Functional Requirements (FR)

In scope:

  • Issue a challenge / risk assessment for a protected action. A site embeds our widget on a form (signup, login, reset, checkout, comment). When the user reaches that action we produce a verdict: this actor is probably human (let them through invisibly), probably a bot (block), or uncertain (present an interactive challenge). The default should be invisible - most real users never see a puzzle.
  • Present an interactive challenge when risk is uncertain. When the invisible signal is not enough, serve a challenge the human can pass and the bot cannot (image selection, an interaction puzzle, a behavioral proof). Support multiple challenge types and rotate them.
  • Verify the outcome server-to-server. After the user “passes”, the site’s backend must be able to ask us “is this token real, unused, and for this action?” and get a trustworthy yes/no. The browser is never trusted to assert its own success.
  • Score behavioral and environmental signal. Use mouse/touch dynamics, timing, browser/device fingerprint, IP reputation, and history to compute a bot-likelihood, so the invisible path works and the challenge is a fallback, not the primary gate.
  • Learn from evasion. Ingest ground-truth signal (a “passed” session that later turned out to be a fraudulent signup, solver-farm patterns, sudden solve-rate spikes on one challenge type) and feed it back to rotate challenges and retrain the scoring model. The adversary adapts; so must we.
  • Accessibility fallback. A human who cannot solve a visual challenge (blind, motor-impaired) must have an audio or alternative path. A CAPTCHA that locks out real disabled users is a broken CAPTCHA.
  • Configurable per site. Each site sets its own sensitivity (how aggressive), allowed challenge types, and score threshold, and gets its own keys.

Explicitly out of scope (say it so you own the scope):

  • The origin’s business logic. We return human/bot risk and a verified token; whether the site then allows the signup, rate-limits, or asks for email verification is the site’s decision. We are a gate signal, not the door.
  • Full fraud/account-takeover scoring. Deep transaction-level fraud (is this stolen card, is this money laundering) is a separate fraud pipeline. We answer the narrower “is this a human or an automated agent,” which feeds that system as one signal. (See a fraud-pipeline design for that half.)
  • WAF / DDoS / volumetric protection. Absorbing a raw packet flood is the network/WAF layer’s job. We sit above it, distinguishing human from bot in application traffic that already passed the network filter.
  • Identity verification / KYC. Proving who someone is (document checks) is a different vertical. We prove only that they are human-operated, not their identity.
  • The exact ML model architecture. Whether the scorer is gradient-boosted trees or a neural net over interaction sequences is a data-science choice. We design the service that scores, challenges, verifies, and retrains; we assume a fast tabular model as the default.

The one decision that drives everything: a CAPTCHA is an adversarial trust gate that must render a verdict the origin can cryptographically trust, in front of every sensitive action, at global scale and low latency, against an adversary who beats any static challenge - so the design is dominated by behavioral risk scoring, a signed single-use verification token, an edge-served fail-open fleet, and a continuous evasion-learning loop, not by any single clever puzzle. Everything below follows from that.

Non-Functional Requirements (NFR)

  • Scale: on the order of 2M challenge/assessment requests per second at peak across all protected sites globally. That is the invisible-assessment volume (fires on nearly every protected pageload/action); the subset that escalates to an interactive challenge is far smaller (single-digit percent of real traffic, much higher during an attack). Bots are a large and spiky fraction - a credential-stuffing run arrives as a wall of hundreds of thousands of requests a second aimed at one site.
  • Latency: the assessment must fit inside the user’s form-submit budget. Target p99 under 100ms for the invisible score call and p99 under 50ms for the server-side token verify (a single signed lookup). The widget load must be tiny and cached at the edge so it does not slow the host page.
  • Availability: we sit in front of login and checkout for thousands of sites, so the availability contract is subtle: 99.99%+, and critically the service must fail open for the common path. If our scorer is down we do not block every login on every customer site - the site’s integration is told to allow (optionally flagged) rather than hard-fail, because blocking all legitimate users worldwide is a far worse outcome than briefly letting some bots through. (A site may opt a high-value flow into fail-closed.)
  • Consistency: risk signals and reputation are eventually consistent - an IP’s reputation being seconds stale is fine. But token verification must be strongly consistent and exactly-once: a token is single-use, and a double-spend (replay) must be impossible even across regions and races. That single-use guarantee is the one place we pay for strong consistency.
  • Durability: every issued token, every challenge served, every score and outcome is logged immutably - this is the audit trail (why did we challenge this user), the replay-detection state, and the training data (the signals exactly as the scorer saw them).
  • Security / adversary model: the browser is hostile. The widget, the challenge, and the client-side score are all attacker-visible and attacker-modifiable, so nothing client-asserted is trusted; the only trusted artifact is a token our backend minted and our backend re-verifies. Challenges must resist automated solvers, replay, and solve-farm outsourcing.
  • Adaptivity: model and challenge freshness are first-class - rotate challenge variants continuously, retrain the scorer on fresh evasion labels, and be able to push an emergency rule within minutes when a new solver is spotted.

Back-of-the-Envelope Estimation (BoE)

Let me ground the design in numbers.

Traffic:

Protected sites:                    ~1,000,000 sites embedding the widget
Assessment requests/day:            ~150,000,000,000  (150B invisible checks)
Average QPS = 150B / 86,400 sec      ≈ 1,700,000 QPS   (misleading average)
Traffic is spiky - business hours + regional peaks + attack walls:
Peak assessment QPS (say ~1.5x avg)  ≈ 2,000,000 QPS    (the number we size for)

Interactive challenges: real humans escalate rarely (invisible passes them).
   Say ~3% of legit traffic escalates -> tens of thousands of interactive
   challenges/sec normally, spiking hard during a bot attack on a site.
Bot fraction: highly variable - background ~20-40% of raw automated traffic,
   and during a credential-stuffing run a single site can see 90%+ bots as a
   wall of 100K+ req/sec. We must shed that wall cheaply, not model each one.

The 1.7M average QPS is a trap. Bot attacks do not arrive smoothly - a credential-stuffing or scalper run is a coordinated wall aimed at one endpoint. We size the assessment tier and the edge for ~2M req/sec steady with headroom to absorb a per-site attack spike, and we design the cheap reputation/rules layer to shed the wall before it reaches the scorer.

Read vs write on the hot paths:

ASSESSMENT (invisible score), per request:
  READ:  fetch reputation + history features for {ip, device, site, asn}
         from a hot KV store, batched -> ~1-5ms.
  RUN:   score the collected signals -> ~1-5ms (tabular model, in-process).
  WRITE: async-emit the assessment event (signals + score + verdict) to Kafka.
  2M/sec reads of a few keys each -> the reputation store serves on the order
  of tens of millions of key-values/sec. Read latency is the tight budget.

TOKEN VERIFY (server-to-server), per protected action:
  Far lower volume than assessment (only when a form actually submits) but
  must be EXACTLY-ONCE: check signature (stateless, ~microseconds) + check-and-
  burn the token id in a strongly-consistent store so it cannot be reused.

The assessment path is read-dominated and cheap (fetch reputation, score). The token-verify path is lower volume but strongly consistent (the single-use burn is the one synchronous cross-region cost). Sizing the assessment reads and keeping the token burn fast-but-exactly-once are the two infrastructure constraints.

Storage:

Immutable assessment log (signals + score + verdict + challenge outcome):
  ~1 KB/event packed.  150B/day * 1 KB ≈ 150 TB/day.
  Retain ~30-90 days hot for training + audit, tier the rest to cold storage.
  Big, sequential, cheap in columnar form.

Reputation / feature store (hot, low-latency state):
  Keyed by ip, device_fp, asn, site, account. Hundreds of millions of active
  keys * ~500 bytes ≈ low hundreds of GB of HOT KV state. Sharded in-memory /
  SSD KV (Redis / Aerospike / Cassandra).

Token store (single-use, short-lived): only LIVE tokens - a token lives
  ~2 minutes. At, say, 200K token-issues/sec * 120s TTL ≈ 24M live tokens *
  ~200 bytes ≈ a few GB. Tiny and self-expiring, but must be strongly
  consistent for the burn.

Challenge asset store: the pool of image/audio challenge variants. Pre-
  generated in bulk, served from the CDN. Tens of millions of assets, cheap
  object storage, aggressively CDN-cached and rotated.

The split is the usual shape: a fat immutable log of everything we assessed (training + audit), a thin hot reputation store the scorer reads, a tiny strongly-consistent token store for single-use, and a CDN-fronted asset pool for challenges. The log is big and cheap; the hot stores are small and expensive.

Bandwidth and compute:

Widget + challenge delivery: the JS widget is small (~tens of KB) and edge-
  cached, so the host page pays almost nothing. Interactive challenge assets
  (images/audio) are served from the CDN, not our origin.

Scoring: a tabular model scores in ~1-5ms on CPU. 2M/sec across the assessment
  fleet is a large but ordinary CPU fleet; the model is cheap, the FEATURE/
  REPUTATION FETCH is the cost. (A heavy sequence model over raw mouse traces
  would flip this and need batching; we keep the hot-path model cheap.)

Challenge generation: pre-generated OFFLINE in bulk and rotated, NOT rendered
  per request - generating a distorted image per call would never fit the
  latency or the QPS. Generation is a batch job feeding the asset pool.

Inference is cheap; the expensive things are fetching reputation within the budget at 2M QPS, the exactly-once token burn, and generating and rotating enough challenge variety that no single solver covers the pool. Size for those.

High-Level Design (HLD)

The system has three planes over one contract. The client/edge plane loads a tiny widget, collects behavioral and environmental signals in the browser, and calls the nearest edge. The serving plane is a stateless, edge-served, fail-open gate: it scores the signals, decides invisible-pass / challenge / block, issues challenges from a pre-generated pool, and - separately - verifies single-use tokens server-to-server. The learning plane consumes the immutable assessment log plus outcome labels, mines evasion patterns, rotates challenges, and retrains the scorer. The glue is a signed single-use verification token: it is the only artifact that crosses the hostile browser and is trusted, because the origin’s backend re-verifies it against us before acting.

                     ASSESSMENT + CHALLENGE PATH   (p99 < 100ms, fail-open, at edge)
   ┌────────────────────────────────────────────────────────────────────────────────┐
   │  Browser on a protected site                                                     │
   │   ┌────────────────────┐  loads tiny widget (edge-cached JS)                     │
   │   │  CAPTCHA Widget    │  collects signals: mouse/touch dynamics, timing,        │
   │   │  (untrusted JS)    │  device fingerprint, JS-challenge (proof-of-work),      │
   │   └─────────┬──────────┘  site key. NEVER trusted to assert its own result.      │
   │             │  POST /v1/assess  {site_key, signals}                              │
   │      ┌──────▼───────────────┐  (nearest region, anycast)                         │
   │      │  Edge / API Gateway  │  TLS, per-site rate-limit, cheap bot filter        │
   │      └──────┬───────────────┘                                                    │
   │      ┌──────▼───────────────────┐  (1) cheap REPUTATION/RULES first              │
   │      │  Assessment Service      │───► Rules + Reputation (deny lists, ASN/IP rep, │
   │      │  (stateless edge fleet)  │      known-bot fingerprints) -- may short-circuit│
   │      │                          │  (2) fetch features (batched multiget)          │
   │      │                          │◄──► Reputation/Feature Store (KV, hot)          │
   │      │                          │  (3) score signals -> bot-likelihood            │
   │      │                          │───► Scoring Model (in-proc tabular, versioned)  │
   │      │                          │  (4) decide: pass / challenge / block           │
   │      └──────┬───────────────────┘                                                │
   │   pass ─────┤ mint signed single-use TOKEN (short TTL) -> return to widget        │
   │  challenge ─┤ return challenge id + asset ref (from CDN pool) -> widget renders    │
   │             │           user solves -> POST /v1/verify-solve -> token on success   │
   │             │  (5) async: emit assessment-event to Kafka (never blocks response)   │
   └─────────────┼──────────────────────────────────────────────────────────────────┘
                 │  token travels through the browser into the site's form submit
   ┌─────────────▼─────────────────────────────────────────────────────────────┐
   │  SERVER-TO-SERVER VERIFY   (origin backend -> us; p99 < 50ms; exactly-once) │
   │   Origin backend  POST /v1/siteverify {secret, token, action}               │
   │        │  verify signature (stateless) + CHECK-AND-BURN token id            │
   │        ▼                                                                    │
   │   ┌─────────────────────┐  strongly-consistent, single-use                  │
   │   │  Token Store        │  (per-region primary, id burned once, TTL ~2m)    │
   │   └─────────────────────┘  -> {success, score, action, hostname}            │
   └────────────────────────────────────────────────────────────────────────────┘
                 │  assessment-events + solve-outcomes
        ┌────────▼───────────────────┐
        │   KAFKA (event backbone)   │◄──────── labels topic (fraud outcomes, solver
        │  assessments | solves | lbl│           farms, solve-rate anomalies, reports)
        └───┬──────────────────┬─────┘
   stream   │ (reputation)     │ archive (training + audit)
   ┌────────▼─────────┐   ┌────▼───────────────────┐
   │ Streaming Agg     │   │  Immutable Assess Log   │
   │ (Flink)           │   │  -> Object store (S3),  │
   │ - IP/ASN velocity │   │     columnar, dt/hr     │
   │ - fp reuse counts │   └────┬───────────────────┘
   │ - upsert reput.   │        │  daily batch
   └────────┬─────────┘   ┌─────▼───────────────────┐
            │ writes back │  Learning Pipeline       │
   ┌────────▼─────────┐   │  - mine evasion patterns │
   │ Reputation/Feat.  │◄──┤  - rotate challenge pool │
   │ Store (hot KV)   │   │  - retrain scorer        │
   └──────────────────┘   │  - shadow + canary       │
                          └─────┬───────────────────┘
             new model + rules  │
                     ┌──────────▼───────┐   pre-generated challenge variants
                     │  Registry +      │──► Challenge Asset Pool (CDN) 
                     │  Challenge Pool  │
                     └──────────────────┘

Request flow - assessing an actor (the invisible hot path):

  1. The host page loads our tiny widget (edge-cached JS). It collects signals in the browser - cursor/touch dynamics, event timing, a lightweight JS proof-of-work, device/browser fingerprint - and calls POST /v1/assess with the site key at the nearest edge region (anycast).
  2. Reputation and rules first. The edge Assessment Service runs the cheap deterministic layer: per-site rate limits, IP/ASN reputation, known-bot fingerprints, deny lists. A clear hit short-circuits - a known bot datacenter IP mid-attack is blocked without scoring, which is what lets the cheap layer absorb the attack wall.
  3. Fetch features. For the ambiguous majority, the scorer fetches reputation/history features for the entities involved (ip, device fingerprint, asn, site, account if known) from the hot Reputation/Feature Store, batched into a few multigets - the tightest latency budget on the path.
  4. Score. The scorer runs the versioned tabular model in-process on the collected signals plus fetched features, producing a bot-likelihood.
  5. Decide. Bands map likelihood to pass / challenge / block. On pass, mint a signed, single-use, short-TTL token and return it to the widget - the user never sees a puzzle. On challenge, return a challenge id and an asset reference (image/audio from the CDN pool); the widget renders it, the user solves, POST /v1/verify-solve checks the answer, and on success a token is minted. On block, no token.
  6. Log asynchronously. After responding, the scorer emits the assessment-event (signals, score, verdict, challenge outcome, model version, rules fired) to Kafka, fire-and-forget, so logging never adds latency.

Request flow - the site trusting the result (server-to-server verify):

  1. The token rides through the hostile browser into the site’s form submit. The site’s backend never trusts the token on its face - it calls POST /v1/siteverify with its secret key, the token, and the expected action.
  2. We verify the signature (stateless - the token is signed with our key) and check-and-burn the token id in the strongly-consistent Token Store: if the id is unseen and unexpired, mark it used and return success with the score and the bound action/hostname; if it was already burned or expired, return failure. This exactly-once burn is what stops replay.

Request flow - the asynchronous learning plane:

  1. Reputation update (streaming): a Flink job consumes assessment/solve events and maintains windowed aggregates - requests per IP/ASN in the last minute, fingerprint reuse counts, solve-rate per challenge variant - and upserts them into the Reputation Store, so scoring this actor sharpens the signal for the next one.
  2. Label ingestion: ground-truth arrives late and async - a “passed” session that produced a fraudulent signup, a challenge variant whose solve rate suddenly jumps (a solver was built), a customer’s abuse report, a confirmed solver-farm IP range - onto the labels topic, joined to the original assessment by id.
  3. Learning pipeline: a batch job reads the immutable log joined to labels, mines evasion patterns (which variants are being solved too well, which fingerprints farm across sites), rotates the challenge pool (retires beaten variants, generates fresh ones), and retrains the scorer, then ships new models and rules via shadow then canary.

The key architectural insight: the only thing that crosses the hostile browser and is trusted is a token our backend minted and re-verifies - everything the client says about itself (its score, its “I passed”) is treated as attacker-controlled. That single trust-boundary discipline is what most naive designs get wrong, and it is why the token, not the challenge, is the heart of the system.

Component Deep Dive

Four hard parts: (1) telling human from bot when the best signal is behavior, not a puzzle (the risk-scoring path), (2) returning a verdict the origin can trust across a hostile browser without forgery or replay (the signed single-use token), (3) serving millions of challenges a second globally without becoming the outage that blocks every login (edge-served, fail-open serving), and (4) staying ahead of an adversary who beats every static challenge (the evasion-learning loop). I will walk each from naive to scalable.

1. Telling Human from Bot: The Squiggly Image -> Behavioral Risk Scoring

The whole point is distinguishing a human from an automated agent. The instinct is a hard puzzle; the reality is that the puzzle is the weakest signal.

Approach A: A hard visual puzzle everyone must solve (the naive instinct)

Show every user distorted text or an image grid; if they solve it, they are human.

Where it breaks: three ways at once. Solvers beat it - modern OCR and vision models read distorted text and pick “all the buses” better than humans do, so the puzzle that stops a bot today is trivially automated tomorrow. Farms outsource it - a bot operator pays a human solver farm a fraction of a cent per solve, so even an unbeatable-by-machine puzzle is beaten by cheap human labor; the puzzle does not distinguish “human at the keyboard” from “human paid to solve for a bot.” And it punishes real users - every legitimate person now does annoying work, accessibility suffers, and conversion drops. A mandatory puzzle is beatable, farmable, and user-hostile. Wrong primary mechanism.

Approach B: Invisible behavioral + environmental risk scoring, puzzle only as fallback

Make the default path invisible: score the actor from signals that are hard to fake at scale, and only present a puzzle when the score is genuinely uncertain.

Signals collected (client, untrusted) + fetched (server, trusted):
  Interaction dynamics: cursor path curvature, velocity, pauses, key timing,
     touch pressure/area on mobile, scroll cadence -> humans are noisy and
     jittery, naive bots are too smooth or too uniform.
  Environmental: browser/device fingerprint consistency, headless-browser
     tells, automation-framework artifacts, timezone/locale coherence.
  Proof-of-work: a small JS puzzle the client must compute -> makes each
     request cost the bot CPU, taxing volume without touching the human.
  Reputation (server-side, trusted): IP/ASN reputation, datacenter vs
     residential, fingerprint reuse across sites, velocity in last minute,
     account age/history if known.
  -> model outputs bot-likelihood in [0,1]
  • Behavior and reputation are the primary signal; the puzzle is the fallback. Most real users are passed invisibly - the model is confident they are human from how they moved and where they came from, and they never see a challenge. The interactive puzzle only fires for the uncertain middle, which is a small slice of legitimate traffic (and a large slice during an attack).
  • Client signals are untrusted; reputation is trusted. Everything collected in the browser can be forged, so it is evidence, not proof - the model weighs it alongside server-side reputation the attacker cannot set. A bot can fake a smooth mouse trace, but it cannot easily fake a residential IP with a long clean history and a fingerprint not seen farming across 400 sites.
  • Proof-of-work taxes volume. A small client-side computation is nothing for one human but multiplies cost across a million-request attack, shifting the economics against the bot without friction for the user.
  • Farm-resistance comes from cost and reputation, not puzzle hardness. Since a human farm can solve any puzzle, we do not try to make an unsolvable puzzle - we make solving expensive and traceable: the farm’s IPs, fingerprints, and solve-rate patterns concentrate in reputation, so the farm gets throttled and flagged as a population even when each individual solve looks human. We beat the economics, not the individual solve.
  • The challenge is a fallback with variety, not the gate. When we do challenge, we pull from a large rotated pool (part 4) so no single solver covers it.

The move is the same as fraud scoring: stop trying to build one unbeatable test and instead score many weak, hard-to-fake-together signals - most of which the honest user emits for free - so the honest user pays nothing and the bot has to fake all of them at once, at scale, cheaply, which it cannot.

2. A Verdict the Origin Can Trust: The Signed, Single-Use Token

The verdict has to travel from us, through a hostile browser, to the site’s backend, and be trusted there. This is the real heart of the system.

Approach A: The widget sets a “passed = true” flag the site reads (naive)

On success, the widget writes a success flag (a cookie, a JS variable, a hidden form field) and the site’s frontend/backend reads it and proceeds.

Where it breaks: the browser is the attacker. Anything the client asserts, the client can forge - a bot skips our widget entirely and submits the form with captcha_passed=true, or replays a real success flag a thousand times. There is no cryptography binding the flag to a real assessment, to this action, or to a single use. A client-asserted verdict is not a verdict; it is a suggestion the attacker writes for themselves. Trusting the browser is the original sin.

Approach B: A signed, single-use, short-lived token verified server-to-server

Move the trust to a token only we can mint and only we can validate, and make the site’s backend re-verify it out of band:

On PASS / successful solve, mint token:
  token = {
    id:        uuid,                 # unique, for single-use burn
    site_id:   "site_88ab",
    action:    "login",             # bound to the action requested
    hostname:  "acme.com",          # bound to the origin
    score:     0.04,                # bot-likelihood, for the site to threshold
    issued_at: ts, exp: ts+120s     # short TTL
  }
  signed = sign(token, our_private_key)   # HMAC or asymmetric; client cannot forge

Site backend verifies (server-to-server, not in the browser):
  POST /v1/siteverify {secret, token, expected_action:"login"}
   1. verify signature with our key           -> forged/tampered? reject
   2. check exp                                -> expired? reject
   3. check action + hostname match request    -> mismatched? reject
   4. CHECK-AND-BURN id in Token Store         -> already used? reject (replay)
   5. success -> {success:true, score, action, hostname}
  • Signed, so it cannot be forged. The token is signed with our key; the client can read it but cannot mint or alter one. The site verifies the signature via us (siteverify), so a hand-crafted passed=true is rejected at step 1.
  • Single-use, so it cannot be replayed. The token id is burned on first verify in a strongly-consistent store. A replay finds the id already used and fails. This is the one place we pay for strong consistency, because a replay is a real breach - solve once, submit a thousand times.
  • Short-lived, so a stolen token rots fast. A ~2-minute TTL means the live-token set is tiny (a few GB) and a leaked token is worthless in minutes, bounding replay-window risk and store size together.
  • Bound to action and origin, so it cannot be diverted. The token names the action (login) and hostname it was issued for; the site checks the token’s action matches the action it is guarding, so a token minted for a low-value comment form cannot be replayed against a checkout, and a token issued for acme.com cannot be used on evil.com.
  • Verified server-to-server, so the browser is out of the trust path. The critical check happens backend-to-backend with the site’s secret key; the browser only carries the token. The site never trusts the token’s face value - it asks us.
  • The score rides along. The token carries the bot-likelihood so the site can apply its own threshold (a bank blocks at 0.3, a comment form at 0.8) without a second call.

Exactly-once at scale. The burn must be atomic even across regions and races (two concurrent siteverify on the same token). We shard the Token Store by token id so each id has one owning primary; the burn is a conditional write (set id=used if absent) - a compare-and-set that exactly one caller wins. Because tokens are short-lived and id-sharded, a given token’s primary is one node, so the burn is a fast local strongly-consistent op, not a global consensus round. Cross-region tokens verify against their home region (encoded in the id) to keep the burn single-owner.

Put together: signed (no forgery), single-use via an atomic burn (no replay), short-lived (bounded risk and storage), bound to action and origin (no diversion), and re-verified server-to-server (browser out of the trust path). That token, not the puzzle, is what makes a CAPTCHA’s verdict actually mean something.

3. Serving Millions a Second Without Becoming the Outage

We are in front of login and checkout for a million sites. We must be fast, global, and - more subtly - we must never be the thing that takes those sites down.

Approach A: A central region the widget calls and the site blocks on (naive)

Every assessment and every verify hits one central cluster, and the site refuses to let the user proceed until it gets our answer.

Where it breaks: two failure modes. Latency by geography and coupling - a user in Sydney round-trips to us-east for every login; a GC pause or a hot shard in our one cluster adds latency to every login on every customer site. Availability coupling - if our cluster is unreachable and the site blocks on us, then our outage becomes a login/checkout outage for a million sites at once. We have turned a bot tool into a global internet kill switch. A mandatory, central, blocking dependency makes our every hiccup everyone’s outage.

Approach B: Edge-served, stateless, fail-open, with the burn as the only strong-consistency cost

Push serving to the edge, keep it stateless, and engineer the failure behavior explicitly:

  • Edge / anycast serving. The widget and assessment endpoint are served from many regions via anycast, so a user hits the nearest edge - low latency globally, and no single region is the world’s bottleneck. The tiny widget JS and the CDN challenge assets are edge-cached, so the host page barely pays.
  • Stateless assessment fleet with the model in-process. Each assessment instance holds the model (tens of MB) and reputation-cache locally and is otherwise stateless, so we scale to 2M QPS by adding instances and lose nothing when one dies. In-process scoring is a local call, not a hop - shaving the budget.
  • Cheap layer sheds the attack wall. Reputation and rate limits at the edge block known-bot datacenter traffic before feature fetch and model, so the expensive path only runs for traffic that passed the cheap filter. During a 100K/sec credential-stuffing run on one site, most of it dies at the edge as datacenter-ASN denies, never touching the scorer.
  • Hard timeouts with graceful degradation. Every downstream call (reputation fetch, model) has a strict timeout well inside the budget. If reputation is slow, the scorer proceeds on client signals plus defaults, or falls back to rules-only. A partial score beats a timeout.
  • Fail open (for the common path), by policy. If assessment is unreachable, the site integration does not block - it applies a fallback: allow (optionally flagged) for normal flows, or the site may present its own step-up. Briefly letting some bots through beats blocking every legitimate user on a million sites. A high-value flow can opt fail-closed (require a manual step) - a per-site, per-action choice.
  • The token burn is the only strong-consistency cost, and it is off the assessment path. Assessment (2M QPS) is fully eventually-consistent and fail-open. The strongly-consistent burn happens only on siteverify, which is far lower volume and id-sharded to a single owner - so the expensive consistency is isolated to where it is truly needed and never gates the high-QPS invisible path.
  • Async logging off the response path. The assessment-event goes to Kafka after the response, buffered locally. Logging is essential but never adds latency or fails the assessment.
assess(signals):
   v = reputation_rules.evaluate(signals)        # edge, sub-ms, may short-circuit
   if v.terminal: return v                        # datacenter-ASN deny, hard cap -> done
   try:
       feats = reputation_store.multiget(entities, timeout=10ms)
   except Timeout:
       feats = partial_or_defaults                # degrade, do not fail
   p = model.score(signals, feats)                # ~1-5ms, in-process
   verdict = decide(v, p, site_policy)            # pass / challenge / block
   if verdict == pass: verdict.token = mint_token(...)
   respond(verdict)                               # <-- user unblocked here
   kafka.emit_async(signals, feats, p, verdict)   # after responding

Put together: edge-served stateless assessment with the model in-process, a cheap reputation layer that sheds attack walls, strict timeouts with degradation, a fail-open (per-site fail-closed) contract, and the only strong-consistency cost isolated to the low-volume token burn - that is how you stay under 100ms globally at 2M QPS and guarantee the CAPTCHA can never be the outage that blocks every login on the internet.

4. Staying Ahead of the Adversary: The Evasion-Learning Loop

Any challenge or model you ship is studied and beaten. Winning is not a one-time challenge; it is a loop that rotates faster than the adversary adapts.

Approach A: One good challenge and a model you retrain when someone complains (naive)

Ship a strong challenge type and a scoring model; update them reactively when abuse gets bad.

Where it breaks: the adversary is continuous and you are not. The day you ship a challenge, someone starts building a solver; a fixed challenge pool has a finite shelf life measured in weeks. A model trained once decays as bots learn which signals you weight and fake exactly those. And “update when someone complains” means the loss has already happened at scale before you react - by the time chargebacks or fraud reports arrive, the solver has run for weeks. A static defense against an adapting attacker loses on a delay.

Approach B: Continuous evasion mining, challenge rotation, and skew-safe retraining

Make defense a continuous loop fed by the same log the whole system already writes:

  • Mine evasion signals continuously. The learning pipeline watches for the tells of a beaten defense: a solve-rate spike on one challenge variant (it got automated), a fingerprint or IP range farming across many sites, a cluster of “passed” sessions that produced downstream fraud. These are the early warnings that a specific challenge or model weakness is being exploited.
  • Rotate the challenge pool, do not render per request. Challenges are pre-generated offline in bulk into a large pool (rendering a fresh distorted image per request would never fit 2M QPS or the latency). We serve from the pool via CDN and continuously retire beaten variants and generate fresh ones, so no solver ever covers the live pool - by the time a variant is automated, it is aging out. Variety and rotation, not per-variant hardness, is the defense.
  • Retrain the scorer on fresh evasion labels, without skew. The scorer retrains daily-plus on the immutable log joined to outcome labels. Critically, we train on the exact signal snapshot the scorer logged at serving time (point-in-time correct) joined to labels, so the model learns on precisely what production feeds it - no second hand-written feature pipeline to drift. This is the same training-serving-skew discipline any scoring system needs.
  • Late and proxy labels. Ground truth is late (a fraudulent signup is confirmed days later). We use fast proxy labels - a variant’s solve-rate anomaly, a farm’s fingerprint concentration, honeypot fields only a bot fills - that arrive in minutes, alongside slow confirmed labels, so the loop reacts before the full outcome is known.
  • Emergency rules in minutes. When a new solver is spotted, an analyst pushes a rule (deny this ASN, throttle this fingerprint pattern, force-challenge this signature) as a config change live in minutes - the stopgap while the model retrains and the pool rotates. Rules adapt in minutes; the model in a day; the pool continuously.
  • Shadow then canary for models and challenges. A new model runs in shadow on live traffic (scoring, not deciding) to catch regressions and skew before it blocks anyone; a new challenge variant is canaried to a slice and watched for solve-rate and human-pass-rate before it joins the pool. Nothing new decides at 100% blind.
  • Honeypots and diversity as free labels. Invisible honeypot fields and interaction traps that only automation trips give a continuous, cheap, high-confidence bot label stream that feeds both reputation and training.
continuous:
  watch(log) -> solve_rate_by_variant, fp_reuse_by_range, downstream_fraud
  if variant.solve_rate spikes:   retire(variant); pool.generate_fresh()
  if range farms across sites:     analyst.push_rule(minutes)
  daily+:
    train_set = join(logged_signal_snapshots, labels)   # point-in-time correct
    model = train(rebalanced(train_set))                 # bots are spiky, weight them
    if shadow_ok(model): canary -> full  (keep prev for instant rollback)

Put together: continuous evasion mining, a large pre-generated challenge pool that rotates faster than solvers cover it, skew-safe retraining on the exact logged signals, proxy labels and honeypots for fast feedback, and minute-scale emergency rules - that loop keeps us rotating faster than the adversary adapts, which is the only way to win a game where every static defense is eventually beaten.

API Design & Data Schema

Widget / assessment API (browser to edge)

POST /v1/assess
  {
    "site_key":   "pk_live_88ab",         # public site key, identifies the site
    "action":     "login",                # login | signup | reset | checkout | comment
    "signals": {                          # ALL client-collected -> untrusted evidence
       "interaction": { "cursor": "...", "timings": [...], "touch": {...} },
       "fingerprint": "fp_hash_ff21",
       "pow":         "0000a3f1...",      # proof-of-work solution
       "ts":          1754092800123
    }
  }

  200 OK  (invisible pass)
  { "verdict": "pass",
    "token":   "eyJ...signed...",         # signed, single-use, ~120s TTL
    "score":   0.04 }

  200 OK  (challenge required)
  { "verdict": "challenge",
    "challenge_id": "chl_7c",
    "type":         "image_select",       # image_select | audio | interaction
    "asset_url":    "https://cdn.cap.example/chl/7c/..." }

  200 OK  (block)  -> { "verdict": "block", "reason": "datacenter_asn+high_score" }

POST /v1/verify-solve
  { "challenge_id": "chl_7c", "answer": [...], "site_key": "pk_live_88ab" }
  200 OK -> { "verdict": "pass", "token": "eyJ..." }   # or {"verdict":"fail"} -> retry

Server-to-server verify API (origin backend to us)

POST /v1/siteverify
  { "secret":          "sk_live_...",     # site SECRET key, backend-only, never in browser
    "token":           "eyJ...",
    "expected_action": "login",           # the action the site is guarding
    "remoteip":        "203.0.113.9" }    # optional, cross-check

  200 OK
  { "success":        true,
    "score":          0.04,               # bot-likelihood; site applies its own threshold
    "action":         "login",            # bound action, MUST match expected_action
    "hostname":       "acme.com",         # bound origin
    "challenge_ts":   1754092800,
    "error_codes":    [] }                # e.g. ["timeout-or-duplicate"] on replay/expiry

Notes:
  - The token is BURNED here (single-use). A second call for the same token
    returns success=false, ["timeout-or-duplicate"].
  - Signature + action + hostname + burn all checked server-side. The browser
    is never trusted to assert success; it only carries the token.

Admin / config API

POST /v1/sites                  # provision a site: keys, sensitivity, allowed challenge types
GET  /v1/sites/{id}/stats       # pass/challenge/block rates, solve rates, bot fraction
POST /v1/rules                  # push an emergency rule (config, minutes to live)
GET  /v1/challenges             # challenge pool variants, solve rates, status (live|canary|retired)
POST /v1/models/{version}/promote  # shadow -> canary -> full, or rollback

Data model

Four stores for four jobs - a hot reputation store (assessment reads), a strongly-consistent token store (single-use), an event backbone (transport), and an immutable log (training + audit), plus a CDN-fronted challenge pool.

Reputation / feature store - KV (Redis / Aerospike / Cassandra), hot serving state:

key:   rep:{entity_type}:{entity_id}         # e.g. rep:ip:203.0.113.9, rep:fp:fp_hash_ff21
value: {                                       # flat feature vector, upserted by stream
   req_1m: 812, req_1h: 4103, is_datacenter: true, asn_rep: 0.12,
   fp_reuse_sites_7d: 411, solve_fail_rate_1h: 0.7,
   account_age_days: 0, updated_at: 1754092790
}
access:  multiget by entity keys; single-key reads, ~1-5ms.
sharding: by entity_id (consistent hashing); read replicas for hot keys.
consistency: eventually consistent (seconds-stale reputation is fine).

Token store - strongly-consistent KV, single-use, short-TTL:

key:   tok:{token_id}
value: { site_id, action, hostname, score, issued_at, used: false }
op:    CHECK-AND-BURN = conditional set used=true IF used=false (compare-and-set)
sharding: by token_id; each id has ONE owning primary (single-owner burn).
ttl:   ~120s auto-expiry -> live set stays tiny (a few GB); id from home region.
consistency: STRONG for the burn - exactly-once, no cross-region double-spend.

Event backbone - Kafka:

topic: assessments   partition by entity (ip/fp) for locality, RF=3, acks=all
topic: solves        partition by challenge_id
topic: labels        partition by assessment_id  (fraud outcomes, farm flags, reports)
purpose: decouple edge serving from the streaming aggregator, log archiver,
         and learning pipeline; absorb 2M/sec + attack spikes.

Immutable assessment log - Object store (S3/GCS), columnar (Parquet):

path: s3://cap-events/dt=2026-08-02/hr=10/part-*.parquet
row:  { assessment_id, site_id, action, entities{ip,fp,asn,account},
        signal_snapshot{ ...exact signals scored... },   # point-in-time
        score, verdict, challenge_id, solve_outcome,
        model_version, rules_fired[], ts }
purpose: THE training set (signals as scored) + the audit trail.
retain: ~30-90 days hot; tier older to cold for compliance.

Challenge asset pool - Object store + CDN:

path:  s3://cap-challenges/{variant}/{asset_id}.{png|wav}   # pre-generated in bulk
meta:  challenges( variant_id PK, type, generated_at, status,   # canary|live|retired
        solve_rate, human_pass_rate )
serve: via CDN, aggressively cached; variants continuously rotated in/out.

Why these choices: a KV store (not SQL) for reputation because assessment is single-key point lookups under a hard latency budget at 2M QPS, which KV does in ~1ms and a relational scan cannot; a separate strongly-consistent, id-sharded token store because single-use is the one exactly-once requirement and isolating it keeps its cost off the huge assessment path; columnar object storage for the log because training is a big analytical scan over signal columns that Parquet reads cheaply and it is immutable/rebuild-friendly; Kafka because 2M/sec of un-droppable events must buffer and fan out to three consumers; and a CDN-fronted pre-generated pool because challenges must be served at edge scale, not rendered per request. Each store does the one thing it is best at, and the log is the only precious, non-rebuildable asset.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Reputation-fetch latency on the p99 path. Fetching features per assessment at 2M QPS inside ~10ms is the tightest budget; a slow shard blows the score’s latency. Fix: a sharded, replicated in-memory/SSD KV store, batched multigets (one round trip per entity), aggressive timeouts with defaults, and a per-instance local cache of the hottest reputation keys. The scorer degrades on partial features rather than waiting.

2. The service as a single point of failure for every customer’s login. A synchronous mandatory dependency can turn our outage into a million sites’ outage. Fix: edge/anycast serving, a fail-open (per-site fail-closed) contract, strict timeouts, and rules-only fallback. The gate degrades to cheap rules or allow-with-flag rather than blocking every user.

3. Attack-wall load spikes (credential stuffing, scalpers). One site’s traffic can 100x in minutes, aimed at the expensive path. Fix: shed at the cheap edge reputation layer - datacenter-ASN denies and per-site rate limits short-circuit the wall before feature fetch and model, and proof-of-work taxes each attacker request. Kafka absorbs the write spike; autoscale the stateless fleet on QPS.

4. Token store contention and cross-region double-spend. The single-use burn is a strong-consistency hotspot and a replay is a real breach. Fix: shard tokens by id to a single owning primary, make the burn a compare-and-set (exactly one winner), and home each token to its issuing region (encoded in the id) so verify is a local strong op, not global consensus. Short TTL keeps the live set and the risk window tiny.

5. Hot entity / hot key. A shared corporate NAT IP, a popular datacenter range, or a targeted site concentrates reads and counter-updates on one shard. Fix: replicate hot reputation keys across read replicas, sub-shard hot aggregation counters (key + hash(sub)%K, sum at read), and local-cache the hottest values with a short TTL. Legitimate shared-IP false positives are softened by weighting fingerprint/history over raw IP.

6. A beaten challenge variant (solver built). A single challenge type gets automated and its solve-rate spikes. Fix: a large pre-generated pool that continuously rotates - retire spiking variants automatically, generate fresh ones, canary new ones on solve-rate and human-pass-rate. No solver covers the live pool because it is always aging out under them.

7. Solve farms (human-solved bots). Cheap human labor solves any puzzle, so per-solve hardness fails. Fix: attack the economics and population, not the solve - proof-of-work raises per-request cost, and farm fingerprints/IP ranges concentrate in reputation and get throttled as a group even when each solve looks human. Honeypots give free bot labels.

8. Training-serving skew and model decay. The scorer scores worse in production than offline, silently, while bots adapt. Fix: train on the exact logged serving-time signal snapshots (point-in-time correct), one shared feature definition, time-split evaluation, and shadow mode before any traffic. Retrain daily-plus on fresh evasion labels.

9. Bad model or challenge rollout. A regressed model could block real humans, or a bad variant could pass bots, at scale on deploy. Fix: shadow -> canary -> full with the incumbent kept for instant rollback, decisions tagged with model_version, and live guardrails (human-pass-rate, block-rate, solve-rate) that auto-halt a bad ramp.

10. Accessibility lockout. A too-hard or vision-only challenge blocks real disabled users - a correctness failure, not an edge case. Fix: always offer an audio and alternative path, tune bands so accessibility flows do not over-challenge, and monitor human-pass-rate by challenge type as a first-class metric - a variant that humans fail is retired regardless of its bot-stopping power.

11. Log growth (~150 TB/day) and cost. The immutable log cannot grow hot forever. Fix: columnar compression, ~30-90 days hot for training, tiering older data to cold storage for retention, and dropping raw client payloads once the signal snapshot (all training needs) is captured.

12. Reproducibility / audit under dispute. A customer asks “why did you challenge/block my user.” Fix: every decision is immutably logged with its signal snapshot, model version, and rule ids, so any past decision is exactly reconstructable - the audit story is a lookup, not a forensic reconstruction.

Wrap-Up

The trade-offs that define this design:

  • Behavioral risk scoring over a hard puzzle. We score many weak, hard-to-fake-together signals and pass most real users invisibly, trading the false comfort of one “unbeatable” challenge for a gate that costs honest users nothing and forces bots to fake everything at once, cheaply, at scale - which they cannot.
  • A signed single-use token over a client-asserted flag. The only artifact that crosses the hostile browser is a signed, short-lived, single-use token the origin re-verifies server-to-server, trading a second backend round trip for a verdict that cannot be forged, replayed, or diverted - the browser is never in the trust path.
  • Edge-served fail-open over a central blocking dependency. Assessment runs stateless at the edge and degrades to rules-only or allow-with-flag when unhealthy, trading a small window of let-through bots for never turning the CAPTCHA into the outage that blocks every login on a million sites - with high-value flows failing closed as the deliberate exception.
  • Strong consistency isolated to the token burn. The 2M-QPS assessment path is fully eventually-consistent and fail-open; only the low-volume, id-sharded single-use burn pays for exactly-once, trading a clean separation of concerns for both speed on the hot path and a hard no-replay guarantee where it matters.
  • A continuous rotation loop over a static defense. A pre-generated challenge pool that rotates faster than solvers cover it, skew-safe daily retraining on logged signals, proxy labels and honeypots, and minute-scale emergency rules keep us adapting faster than the adversary - the only way to win a game where every fixed challenge is eventually beaten.
  • Derived state over precious state. Reputation, aggregates, the challenge pool, and the model are all rebuildable from the immutable assessment log; only that log (signals as scored, plus labels) is precious, which is what makes training reproducible and any past decision auditable.

One-line summary: an edge-served, stateless, fail-open assessment fleet that scores behavioral and reputation signals to pass most humans invisibly and challenge only the uncertain from a continuously rotated pool, mints a signed single-use short-TTL token that the origin re-verifies server-to-server with an exactly-once burn, logs every decision with its exact signal snapshot to an immutable log, and retrains on those snapshots joined to late evasion labels with shadow-then-canary rollout - trading a second verify round trip and eventual-consistency in reputation for tamper-proof, replay-proof, human-versus-bot verdicts at millions of challenges a second against an adapting adversary.