An A/B testing platform is deceptively easy to describe and brutal to get right. “Show half the users the blue button and half the green button, then see which converts better.” Every part of that sentence hides a landmine. Which half - and does a returning user stay in the same half tomorrow, or does the button flicker between visits? What happens when there are not two experiments but three thousand running at once, some of them overlapping on the same page? How do you count 500M users worth of impressions and conversions per experiment without a nightly batch job that reports yesterday’s winner today? And the one that sinks most naive designs: how do you decide the green button actually won, versus won by random noise, without a statistician manually eyeballing a spreadsheet? The real problem is this: assign 500M users to variants across thousands of concurrent experiments deterministically and with sub-millisecond latency on the request path, capture a firehose of exposure and conversion events without losing any, and compute per-experiment statistical significance in near-real-time - all while guaranteeing that a user never flips variants mid-experiment and that overlapping experiments do not silently poison each other’s results.

Everything hard here - deterministic bucketing, the event pipeline, streaming significance, experiment isolation - serves two non-negotiable properties: assignment integrity (the right user sees the right variant, consistently, forever, computed fast enough to sit on the critical rendering path) and statistical validity (the numbers you report are actually the numbers, computed correctly, not corrupted by double-counting, sample-ratio mismatch, or peeking). You do not “flip a coin”; you hash the user into a deterministic bucket so no assignment state has to be stored or looked up, you treat exposures and conversions as an append-only event stream, and you pre-aggregate metrics continuously so significance is a cheap read, not a batch scan. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Create and configure experiments. A product manager defines an experiment: a name, the variants (control + one or more treatments), the traffic split (50/50, 90/10, etc.), a targeting rule (which users are eligible - country, platform, logged-in, % of traffic), a primary metric, and start/stop dates.
  • Assign users to variants. Given a user_id and an experiment_id, return which variant that user is in - deterministically (same user always gets the same variant for the life of the experiment) and fast (this call sits on the page-render path).
  • Bucket and gate traffic. Only a configured fraction of eligible users enters the experiment at all; the rest see the default. Ramp-up (“start at 5% of traffic, increase to 50%”) must be possible without reshuffling already-assigned users.
  • Log exposure and conversion events. When a user is actually shown a variant, log an exposure (a.k.a. impression). When they perform the metric action (click, purchase, sign-up), log a conversion. These are the raw data for every metric.
  • Compute metrics and significance. Per experiment, per variant: exposures, conversions, conversion rate, and a statistical significance test (is the difference between control and treatment real or noise) with a confidence interval and p-value, updated in near-real-time.
  • Feature flags and kill switch. The same assignment machinery powers feature flags; an experiment must be stoppable instantly (kill switch) so a bad variant can be turned off in seconds.
  • Mutual exclusion / layers. Two experiments that would interfere (both changing checkout) must be runnable as mutually exclusive so no user is in both.

Explicitly out of scope (say it out loud so you own the scope):

  • The UI for building experiments. The dashboard is a thin client over the config API; I design the API and the config store, not the front-end.
  • Choosing the statistical test as a research problem. I design the system that computes significance continuously; I treat “z-test vs sequential test vs Bayesian” as a pluggable method and pick sensible defaults, not derive the math.
  • General product analytics / clickstream warehousing. The experiment platform consumes exposure and conversion events; a full analytics warehouse (funnels, retention, cohorts) is a separate system I feed, not own.
  • Auth, billing, org/permissions plumbing. Assume a user_id identifies the unit of assignment and an authenticated PM creates experiments; move on.
  • Client SDK internals per platform. I define the SDK contract (what it fetches, caches, and sends) but not the per-language implementation.

The one decision that drives everything: assignment must be a pure deterministic function of (user_id, experiment_id, salt) with no stored per-user state, because storing an assignment row for 500M users x thousands of experiments is both a storage and a latency disaster - and events must be an append-only stream feeding continuous pre-aggregation, because scanning raw events per significance query does not scale. Unlike a database read (you look up stored state) here you compute the assignment from a hash so there is nothing to store or look up; unlike a nightly report (batch scans raw rows) here metrics are rolled up continuously so significance is a counter read. That asymmetry - compute assignment instead of storing it, pre-aggregate metrics instead of scanning them - is the whole problem.

Non-Functional Requirements (NFR)

  • Scale: 500M users. Thousands of concurrent experiments (say up to ~10,000 active). A user hitting the app is evaluated against every experiment they are eligible for on each page load - potentially dozens of assignment evaluations per request. Exposure + conversion events run into the tens of billions per day.
  • Latency: variant assignment under ~1ms, effectively free, because it sits on the request/render path and blocking a page on a network call to an experiment service is unacceptable. Achieved by evaluating assignment locally in the SDK from cached config (a hash, no network call). Config fetch (SDK pulls the experiment ruleset) tolerates ~100ms and is cached. Significance/metrics reads for the dashboard tolerate seconds; the pipeline that updates them targets seconds-to-minutes from event to updated metric.
  • Availability: 99.9%+ on assignment - if the experiment service is down, the SDK must fail safe to the control/default variant using its last-cached config, never block or error the page. Event ingestion must be highly available (dropping events corrupts results); the analytics read path can tolerate brief unavailability.
  • Consistency: assignment is strongly deterministic (the same input always yields the same variant, everywhere, forever) but config propagation is eventually consistent (a new experiment or a ramp change reaches all SDKs within seconds, not instantly). Metrics are eventually consistent (a conversion from ten seconds ago may not be counted yet). The hard invariant: a user must never flip variants once assigned within an experiment’s life - determinism guarantees this without any stored state.
  • Durability: raw exposure and conversion events are the crown jewels (they are the experiment results and are legally/financially load-bearing) - an accepted event must not be lost; it lands on a durable log. Experiment configs are small and must be durable (a lost config invalidates running experiments). Pre-aggregated metrics are regenerable by replaying the event log, so they need availability, not durability.
  • The real budget is assignment latency and result integrity. It is acceptable for a config change to take a few seconds to propagate and for metrics to lag by seconds; it is not acceptable to add network latency to page render, to lose events, or to report a “winner” that is a counting bug or a statistical artifact. The whole design computes assignment locally and treats event integrity as sacred.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Users and assignment QPS (the hot path):

500M users. Say ~200M are daily-active (DAU).
   Each DAU triggers ~10 page/screen loads per day.
   200M * 10 = 2B page loads/day.
On each load the SDK evaluates the user against every eligible active experiment.
   Say ~50 experiments are eligible for a given user on a given page (of ~10k total).
   2B loads * 50 evaluations = 10^11 = 100B assignment evaluations/day.
   100B / 86,400 ≈ ~1.16M evaluations/sec steady. Peak (~4x) ≈ ~4.6M/sec.
If each evaluation were a network call to a service:
   4.6M assignment calls/sec, EACH adding latency to page render. Absurd.
=> assignment MUST be a local, in-SDK hash computation. Zero network calls per eval.
   The service's job shrinks to shipping config, not answering per-eval requests.

Event volume (exposures + conversions - the write path):

Exposures: a user actually shown a variant logs one exposure per (experiment, session).
   Not every eligible eval is an exposure (many experiments target a page the user
   does not reach). Say ~15 exposures per page load actually fire.
   2B loads * 15 = 30B exposure events/day.
Conversions: far rarer - the metric action. Say ~2 conversion events per load.
   2B * 2 = 4B conversion events/day.
Total ≈ 34B events/day.
   34B / 86,400 ≈ ~400,000 events/sec steady. Peak ~3x ≈ ~1.2M events/sec.
No transactional DB absorbs 1.2M writes/sec. This is a partitioned event LOG (Kafka).

Storage:

Raw events: 34B/day * ~200 bytes/event ≈ ~6.8 TB/day of raw event log.
   Retain hot (~90 days for re-analysis) ≈ ~600 TB. Older -> cold object storage.
Experiment configs: ~10k active experiments * a few KB each ≈ tens of MB. Tiny.
   The FULL config bundle SDKs fetch is small enough to ship whole and cache in RAM.
Pre-aggregated metrics: per experiment, per variant, per metric, per time-bucket.
   10k experiments * ~3 variants * ~5 metrics * (counters) ≈ millions of counter rows.
   Even with per-minute time buckets and 90-day retention, this is GBs, not TBs -
   because it is aggregates, not raw events. This is what the dashboard reads.

Bandwidth (config distribution):

Config bundle (~10k experiments, targeting rules, splits, salts) compressed ≈ a few MB.
   500M SDK instances polling every ~60s for the delta:
   most polls return "no change" (a cheap 304); real deltas are pushed via a CDN
   or a streaming channel. The full bundle is a CDN object; SDKs pull deltas.
   This is a read-heavy, cacheable, CDN-friendly distribution problem, not a per-user one.

Compute for significance:

Significance is computed from AGGREGATES (n, conversions per variant), not raw events.
   A z-test / sequential test over two counters is microseconds of math.
   The cost is not the test; it is keeping the aggregates fresh - a streaming job
   incrementing counters at ~1.2M events/sec. That is a Flink/stream-processing load,
   bounded and horizontally scalable, NOT a per-query scan of 6.8 TB/day.

The takeaways: assignment is a ~1-5M evaluations/sec firehose where any network call per evaluation is fatal - forcing local in-SDK hash-based assignment with the service reduced to shipping a small cacheable config bundle; events are a ~1.2M-events/sec write firehose that must be a durable partitioned log; storage is dominated by a ~600 TB hot raw-event log (the results, kept for re-analysis) while the dashboard-facing metrics are tiny aggregates; and significance is cheap math over those aggregates, so the real work is a streaming job keeping counters fresh, not scanning raw events per query. Those four - deterministic assignment, event ingestion, streaming aggregation/significance, and experiment isolation - are the interview.

High-Level Design (HLD)

The architecture splits along the fundamental seam of experimentation: the control plane (define experiments, distribute config - low volume, must be consistent and durable) is separate from the data plane (assign users locally, capture events, aggregate metrics - astronomically high volume, must be fast and lossless). Config flows outward from a small durable store to every SDK; events flow inward from every SDK to a durable log that a streaming job rolls up into metrics. Assignment itself happens nowhere central - it happens inside each SDK, as a pure function of cached config.

   CONTROL PLANE (low volume, consistent, durable)
   ┌──────────┐   define/edit   ┌──────────────┐   store    ┌──────────────────┐
   │ PM / Dash│──experiment────>│ Experiment   │───────────>│ Config Store      │
   │ (web UI) │   (variants,    │ Config API   │            │ (SQL: experiments,│
   └──────────┘   split, rules) │ (validate)   │            │ variants, rules)  │
                                └──────────────┘            └────────┬─────────┘
                                                                     │ publish bundle
                                                                     ▼
                                                            ┌──────────────────┐
                                                            │ Config Distributor│
                                                            │ (CDN / streaming  │
                                                            │  channel; deltas) │
                                                            └────────┬─────────┘
   ──────────────────────────────────────────────────────────────── │ pull/push
   DATA PLANE (high volume, fast, lossless)                          │ config
                                                                     ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │  CLIENT + SDK  (500M instances)                                            │
   │   * caches the config bundle in RAM                                        │
   │   * ASSIGN LOCALLY:  variant = bucket( hash(user_id + exp_salt) )  (<1ms,  │
   │     no network call) - evaluate targeting + split from cached config       │
   │   * emit EXPOSURE when a variant is actually shown; EXPOSURE+CONVERSION    │
   │     events batched and sent async                                          │
   └───────────────────────────────┬──────────────────────────────────────────┘
                                    │ batched events (async, fire-and-forget)
                                    ▼
                           ┌──────────────┐    durable   ┌──────────────────┐
                           │ Event Ingest │────log──────>│ Event Log         │
                           │ (validate,   │              │ (Kafka, by        │
                           │  dedup gate) │              │  experiment_id)   │
                           └──────────────┘              └────────┬─────────┘
                                                                  │ consume (stream)
                         ┌────────────────────────────────────────┤
                         ▼ (stream)                                ▼ (bulk)
                ┌──────────────────┐                     ┌────────────────────┐
                │ Aggregation Job   │  increment          │ Raw Event Archive  │
                │ (Flink): per      │  counters           │ (object storage;   │
                │ (exp,variant,     │  dedup by event_id  │  re-analysis, audit)│
                │ metric,bucket)    │                     └────────────────────┘
                └───────┬───────────┘
                        │ writes aggregates
                        ▼
                ┌──────────────────┐        ┌──────────────────────────────┐
                │ Metrics Store     │<──────│ Significance / Stats Service  │
                │ (counters:        │ reads │ (z-test / sequential test on  │
                │  n, conversions)  │       │  counters -> p-value, CI)     │
                └──────────────────┘        └───────────────┬──────────────┘
                                                            ▼
                                                   ┌──────────────┐
                                                   │  Dashboard    │
                                                   │ (results, live)│
                                                   └──────────────┘

Config flow (define an experiment, get it to every SDK):

  1. A PM defines an experiment via the Config API, which validates it (splits sum to 100%, salt assigned, no illegal overlap) and writes it to the Config Store (a small SQL DB - experiments, variants, targeting rules, mutual-exclusion layers).
  2. The Config Distributor compiles all active experiments into a compact config bundle and publishes it (to a CDN as a versioned object, and/or via a streaming channel). SDKs poll for deltas or receive pushes; the bundle reaches all SDKs within seconds. This is eventually consistent and CDN-cacheable - a read-heavy fan-out of a tiny payload.

Assignment flow (a user loads a page, under 1ms, no network call):

  1. The SDK already holds the config bundle in RAM. For each active experiment the user is eligible for, it evaluates targeting (country/platform/logged-in rules) locally, then computes assignment: hash(user_id + experiment_salt) -> a bucket in [0,10000), and maps the bucket to a variant per the configured split. Deterministic, pure, sub-millisecond, no I/O.
  2. When the assigned variant is actually rendered, the SDK emits an exposure event. Later, if the user performs the metric action, it emits a conversion event. Both are buffered and sent asynchronously in batches - never on the render path.

Event flow (results into the system):

  1. Batched events hit the Event Ingest gate, which validates and lands them on a durable partitioned log (Kafka), partitioned by experiment_id. Each event carries a client-generated event_id for idempotent dedup.
  2. The log fans out: the Aggregation Job (Flink) consumes the stream and increments per-(experiment, variant, metric, time-bucket) counters in the Metrics Store, deduping by event_id; a separate consumer archives raw events to object storage for re-analysis and audit.

Significance flow (is the treatment winning?):

  1. The Significance/Stats Service reads the small aggregate counters (n and conversions per variant) from the Metrics Store and runs the configured statistical test (z-test for proportions, or a sequential test that is safe under continuous monitoring), producing a p-value, confidence interval, and lift per variant vs control.
  2. The Dashboard reads these results live. Because they are computed from continuously-updated aggregates, the PM sees near-real-time significance, not a nightly report.

The key insight: assignment is a pure function computed at the edge (no central state, no network call), and results are a one-way event stream rolled up into small aggregates. The control plane keeps experiments consistent and pushes a tiny config out; the data plane assigns locally and streams events back. Nothing on the hot path talks to a central assignment service, and nothing on the read path scans raw events. That separation is what lets assignment hit millions/sec at sub-millisecond latency while metrics update within seconds.

Component Deep Dive

The hard parts, naive-first then evolved: (1) deterministic variant assignment - how to bucket 500M users across thousands of experiments with no stored state and no network call; (2) the event pipeline - how to capture tens of billions of events/day without loss or double-counting; (3) streaming significance - how to compute valid statistics continuously without peeking bias; (4) experiment isolation - how overlapping experiments avoid poisoning each other. Sample-ratio mismatch threads through all four as the integrity check.

1. Variant assignment: why a stored-assignment table fails, then deterministic hash bucketing

Naive approach: when a user first enters an experiment, flip a weighted coin, and store the result in an assignments(user_id, experiment_id, variant) table. On every subsequent request, look up the row so the user stays put.

Where it breaks:

  • Storage explodes. 500M users x up to thousands of experiments each = potentially hundreds of billions of assignment rows. Every new experiment writes up to 500M rows; every ramp-up writes more. The table is enormous, mostly to encode information a hash could regenerate for free.
  • Latency is fatal. Assignment is on the render path. A DB/cache lookup per (user, experiment) - dozens per page load, ~4.6M evaluations/sec at peak - means a network round trip before the page can render. From the BoE this is physically absurd; even a cache at that QPS with sub-ms budget is a monster you should not need.
  • First-write races. Two concurrent requests for the same new (user, experiment) both flip a coin and both try to write - now you need a transaction or a race-safe upsert per assignment, at millions/sec. Miserable.
  • Cold-start and offline are impossible. An SDK on a flaky mobile network cannot block page render on a lookup, and cannot assign at all if the service is unreachable.

Evolution: assignment as a pure deterministic hash - variant = bucket(hash(user_id + experiment_salt)) - computed locally in the SDK, storing nothing. The core trick: a good hash of the same input always yields the same output, so you never need to remember an assignment - you recompute it, identically, every time, everywhere, for free.

Given user_id "u_8123", experiment "exp_42" with salt "s_42abc" and split
   control:50%, treatment:50%:

  h = hash( "u_8123" + ":" + "s_42abc" )          # e.g. MurmurHash / xxHash, 64-bit
  bucket = h mod 10000                            # a bucket in [0, 10000)
  # map buckets to variants by the split:
  #   buckets   0 .. 4999  -> control
  #   buckets 5000 .. 9999 -> treatment
  variant = (bucket < 5000) ? "control" : "treatment"

  => u_8123 lands in the SAME bucket every time, on every device, forever,
     with ZERO stored state and ZERO network calls.

Why the salt (per-experiment) matters: if you hashed user_id alone, a user in bucket 200 would be in the low bucket of every experiment - their assignments across experiments would be perfectly correlated, so experiments could not be treated as independent. A per-experiment salt re-randomizes each user’s bucket independently per experiment, so being control in exp_42 says nothing about their variant in exp_43. This is what makes concurrent experiments statistically independent.

Traffic gating and ramp-up fall out of the same buckets. To run an experiment at only 10% of traffic: reserve buckets [0, 1000) as “in experiment” and [1000, 10000) as “not in experiment / default”, then split the in-experiment buckets by the variant ratio. Ramping to 50% means extending the in-experiment range to [0, 5000) - and critically, users already in [0, 1000) keep their exact variant; you only add new buckets. The buckets are stable; you move the boundary, never reshuffle. This is why hash bucketing beats random assignment: ramp-up is monotonic and never flips an existing user.

Ramp-up without reshuffling (10% -> 50%), same salt, same buckets:
  at 10%:  buckets [0,500)->control  [500,1000)->treatment  [1000,10000)->default
  at 50%:  buckets [0,2500)->control  [2500,5000)->treatment [5000,10000)->default
  A user in bucket 300 (control) STAYS control at both 10% and 50%.
  Only newly-included buckets (1000..5000) get their first assignment. No flips.

Why this is correct at scale: assignment is O(1) CPU, zero I/O, zero storage - a hash and a comparison, done in the SDK from RAM-cached config. 4.6M evaluations/sec is trivial because each is a few nanoseconds of local math with no shared state to contend on. Determinism gives the hard invariant (a user never flips) for free, no locks, no transactions, no table. The only thing the SDK needs from the network is the small config bundle (salts, splits, targeting), which is cached and refreshed lazily. You replaced a hundred-billion-row table and millions of lookups/sec with a stateless hash - the single most important move in the whole design.

Sticky assignment for logged-out users. The unit of assignment is user_id. For anonymous users, use a stable client-generated device_id/cookie as the hash input so they stay consistent across a session; when they log in, you can optionally re-key to user_id (accepting one controlled transition at login, logged as such so it does not corrupt metrics). The rule: hash on the most stable identifier available, and never change it mid-experiment for the same logical user.

2. The event pipeline: why direct DB writes fail, then a durable log with idempotent aggregation

Naive approach: when an exposure or conversion happens, the SDK calls a service that writes a row to a database and increments the experiment’s counters transactionally.

Where it breaks:

  • Write volume is impossible. ~1.2M events/sec at peak. No transactional DB, and no synchronous per-event counter update, absorbs that. Incrementing a shared per-experiment counter transactionally at that rate is a lock-contention meltdown on hot experiments.
  • On the render/interaction path. A synchronous event write couples the user’s action to your ingestion latency; a slow write path slows the app. Events must be fire-and-forget.
  • Loss and double-count both corrupt results. A dropped conversion under-reports a variant; a retried request double-counts it. In a system whose entire output is counts, both are catastrophic - they change which variant “wins.”
  • No replay. If a metric definition changes or a bug is found, a system that only kept incremented counters cannot recompute; the raw events are gone.

Evolution: events are append-only to a durable partitioned log, aggregated asynchronously by a streaming job that dedups by event_id. Decouple capture from computation: capture is a cheap durable append; computation is a downstream stream job.

SDK buffers events, sends in async batches (with retry) ─┐
                                                         ▼
   Event Ingest gate: validate schema, attach receive_ts, pass through
                                                         ▼
   Kafka topic "events", partition = hash(experiment_id)   # durable, replicated
                                                         │
                  ┌──────────────────────────────────────┤
                  ▼ (stream, exactly-once via dedup)      ▼ (bulk)
   Aggregation Job (Flink):                       Raw Archive (object storage):
     keyed by (experiment_id, variant, metric)      every event, partitioned by
     maintains a dedup set of seen event_id         day/experiment, for re-analysis
     increments: exposures_n, conversions_n,        and audit. Source of truth for
       sum, sum_sq (for variance)                   REPLAY if aggregates are wrong.
     writes counters to Metrics Store per
       time-bucket (e.g. per-minute)

Idempotency is the linchpin. Every event carries a client-generated event_id (a UUID). The SDK may retry a batch (flaky network), so the same event can arrive twice. The aggregation job keeps a dedup set of recently-seen event_ids (a bounded window - say last N minutes, backed by a fast store / RocksDB state in Flink) and drops duplicates before incrementing. Exactly-once counting is achieved not by exactly-once delivery (impossible over a lossy network) but by at-least-once delivery + idempotent processing. This is the property that keeps counts correct under retries.

Why partition by experiment_id: each experiment’s events land on the same partition set, so the aggregation job can maintain per-experiment counters with local state and no cross-partition coordination. It also means one experiment’s firehose does not need cross-partition joins to aggregate. (The hot-experiment case - one viral experiment overloading its partition - is handled in Bottlenecks by sub-partitioning on hash(experiment_id, variant) or a random suffix.)

Exposure discipline - only count what was shown. A subtle but critical rule: log an exposure only when the variant is actually rendered to the user, not merely when the SDK evaluated the assignment. Counting evaluations instead of exposures inflates n with users who were assigned but never reached the changed surface, diluting the effect and biasing the rate. The SDK fires the exposure at the render call site. This “trigger on exposure” discipline is what keeps the denominator honest.

Why this is correct at scale: the write path is a durable append (Kafka handles 1.2M/sec across partitions, replicated so accepted events are not lost), fully async (never on the user’s path), and replayable (raw archive lets you recompute any metric). Counting is idempotent (dedup by event_id), so retries do not corrupt results. The expensive part - aggregation - is a horizontally-scaled stream job, not a synchronous per-event DB transaction. You traded “write and count in one synchronous transaction” for “append durably now, count idempotently downstream” - the only shape that survives the volume while keeping counts exact.

3. Streaming significance: why batch-and-eyeball fails, then continuous aggregates with peeking-safe tests

Naive approach: nightly, scan the raw events, compute each variant’s conversion rate, and let a PM eyeball whether the treatment looks better; call it when it “looks significant.”

Where it breaks:

  • Scanning raw events per report does not scale. 6.8 TB/day; computing rates by scanning raw rows is a heavy batch job that reports yesterday’s data and cannot answer “how is it doing right now?”
  • Eyeballing is not statistics. “Treatment is at 5.2% vs control 5.0%” says nothing without sample size and variance. With small n, a 0.2-point gap is pure noise; with huge n, it is a real win. A human eyeballing rates ships false winners constantly.
  • Peeking inflates false positives - the biggest trap. If you run a fixed-horizon z-test but check it repeatedly and stop as soon as it crosses p < 0.05, your real false-positive rate is far above 5% - you are guaranteed to eventually cross the line by chance. Continuous monitoring with a naive test is statistically invalid, and this is exactly how real teams ship phantom wins.
  • Batch is too slow for a kill switch. A treatment tanking revenue must be caught in minutes, not tomorrow morning.

Evolution: continuously-updated sufficient statistics feeding a test that is valid under continuous monitoring. Two moves: pre-aggregate the sufficient statistics so the test is a cheap read, and use a test designed to be peeked at.

The aggregation job maintains, per (experiment, variant), running SUFFICIENT STATISTICS:
   n            = exposures (the denominator)
   x            = conversions (for a binary/rate metric)
   sum, sum_sq  = for continuous metrics (revenue), to get mean and variance online
These are all INCREMENTAL - each event bumps a counter. No raw scan ever needed.

Significance = a function of just these counters:
   control:   n_c, x_c   -> p_c = x_c/n_c
   treatment: n_t, x_t   -> p_t = x_t/n_t
   pooled SE, z-score, p-value, and a confidence interval on the LIFT (p_t - p_c)/p_c.
   For revenue: Welch's t-test from (mean, variance, n) per variant.
This math over a handful of counters is MICROSECONDS - computed on demand per dashboard read.

Peeking-safe by construction. Because the whole point is to watch results live, the fixed-horizon z-test is the wrong default - it is only valid if you look once at a pre-committed sample size. Use one of:

ApproachHow it handles continuous monitoringTrade-off
Fixed-horizon z-testOnly valid at the pre-set n; peeking inflates false positivesSimple, but unsafe to watch live
Sequential testing (e.g. mSPRT, always-valid p-values)p-value/CI valid at every moment; stop anytimeSlightly larger n to reach significance
Group sequential (alpha spending)Pre-planned interim looks with adjusted thresholdsMust plan the looks in advance
Bayesian (posterior on lift, P(treatment better))Naturally supports continuous readoutNeeds priors; different interpretation

The pragmatic default is sequential testing / always-valid confidence intervals: they are designed so you can look as often as you like and stop the moment the interval excludes zero, without inflating error. The dashboard then honestly shows “significant” only when the always-valid interval crosses, so a PM watching it live cannot manufacture a false positive by peeking. The system enforces the statistics so humans cannot cheat them.

Why this is correct at scale: the dashboard read is a counter fetch + microsecond math, not a data scan, so significance is genuinely real-time and cheap regardless of how many billions of raw events underlie it. Freshness comes from the streaming aggregation (seconds behind), not from re-scanning. And validity is guaranteed by the choice of test, not by trusting the user not to peek. You traded “batch-scan then eyeball” for “continuously maintained sufficient statistics + a monitoring-safe test read on demand” - fast, cheap, and statistically honest.

Sample-ratio mismatch (SRM) - the integrity alarm. A 50/50 experiment should send ~50% of exposures to each variant. If the observed split is 50.5/49.5 at large n, something is broken - a logging bug, a redirect dropping one variant, a bot skewing traffic - and the experiment’s results are untrustworthy regardless of significance. The stats service runs a chi-square SRM check on the exposure counts every time it computes significance and flags the experiment if the ratio deviates beyond chance. SRM is the smoke detector: a “significant” result on an SRM-failing experiment is a bug, not a finding, and the platform must surface it loudly.

4. Experiment isolation: why global randomization fails, then salts, layers, and mutual exclusion

Naive approach: run every experiment independently with the same randomization, and let users be in as many experiments as they qualify for at once.

Where it breaks:

  • Correlated assignment without per-experiment salts. As covered in section 1, hashing user_id alone correlates a user’s variant across experiments - they cannot be treated as independent, and interaction effects contaminate every experiment. (Per-experiment salt fixes this.)
  • Conflicting experiments collide. Two teams both experiment on the checkout button - one tests color, one tests copy. A user in “green” from exp A and “new copy” from exp B sees a combination neither team intended, and each experiment’s metric is polluted by the other’s treatment. The measured lift is not attributable to either change.
  • Unbounded overlap dilutes power. With thousands of experiments, a single user is in dozens; the noise from all the other concurrent changes swells the variance of any one experiment, making real effects harder to detect.

Evolution: per-experiment salts for independence, plus a layer/domain model for mutual exclusion where experiments must not overlap. Two mechanisms for two problems.

INDEPENDENCE (default): each experiment has its own salt, so bucketing is
   statistically independent across experiments. Most experiments can safely
   overlap - a color test and an unrelated recommendation test do not interact,
   and independent randomization means their effects average out (are orthogonal).

MUTUAL EXCLUSION (when they DO interact): organize experiments into LAYERS.
   A user gets ONE bucket per layer; experiments in the same layer partition that
   layer's buckets, so a user is in AT MOST ONE experiment per layer.

   Layer "checkout"  (buckets 0..9999, salt = layer_salt):
     exp_A (button color)  -> buckets [0, 3000)
     exp_B (button copy)   -> buckets [3000, 6000)
     holdback/unused       -> buckets [6000, 10000)
   A user hashes into the layer once; their layer-bucket puts them in A, or B,
   or neither - NEVER both. The two checkout experiments are mutually exclusive.

   Experiments in DIFFERENT layers overlap freely and independently (own salts).

This is the layered / overlapping experiment model (Google’s “domains and layers”): non-interacting experiments live in different layers and overlap fully (maximizing how many experiments you can run at once), while interacting experiments share a layer and are mutually exclusive (guaranteeing clean attribution). The config store encodes layer membership; the SDK, when evaluating, first hashes the user into each layer, then resolves which experiment (if any) in that layer owns their bucket - all still local, still deterministic, still no network call.

Why this is correct at scale: it lets you run thousands of experiments concurrently (the whole requirement) without them silently poisoning each other. Independence-by-salt is the cheap default for the common case (most experiments do not interact); layers are the explicit tool for the dangerous case (experiments on the same surface). Both are pure config the SDK evaluates locally. You traded “everything overlaps and hope for the best” for “independent by default via salts, mutually exclusive by design via layers” - the only way to scale experiment count without sacrificing result validity.

API Design & Data Schema

Traffic splits into a low-volume control plane (define experiments, fetch config) and a high-volume data plane (ingest events, read metrics). The assignment “API” is notably absent as a hot endpoint - it is an SDK-local computation, by design.

REST / RPC endpoints

# --- Control plane: experiment management (low volume, PM-facing) ---
POST /api/v1/experiments
  Body: { "name":"checkout_button_color", "layer":"checkout",
          "variants":[ {"key":"control","split":50}, {"key":"green","split":50} ],
          "targeting":{ "country":["US","CA"], "platform":["ios","android"],
                        "traffic_pct":10 },
          "primary_metric":"purchase_conversion", "test":"sequential",
          "start":"2026-07-18", "end":"2026-08-01" }
  -> 201 { "experiment_id":"exp_42", "salt":"s_42abc", "status":"draft" }
  # server validates splits sum to 100, assigns a salt, checks layer bucket availability.

PATCH /api/v1/experiments/exp_42   { "targeting":{"traffic_pct":50} }   # ramp up
  -> 200; only EXTENDS the in-experiment bucket range - never reshuffles existing users.

POST /api/v1/experiments/exp_42/stop   { "reason":"kill_switch" }
  -> 200; config bundle updated within seconds; SDKs stop assigning, serve default.

GET  /api/v1/experiments/exp_42/results
  -> 200 {
       "experiment_id":"exp_42",
       "srm_ok": true,                          # sample-ratio-mismatch check
       "variants":[
         {"key":"control","n":812004,"conversions":40120,"rate":0.0494},
         {"key":"green","n":810233,"conversions":42998,"rate":0.0531,
          "lift":0.0749,"p_value":0.003,"ci":[0.031,0.118],"significant":true} ],
       "as_of":"2026-07-17T09:41:00Z" }        # near-real-time, from aggregates

# --- Config distribution: SDK pulls the ruleset (cacheable, CDN-fronted) ---
GET  /api/v1/config?since_version=1013
  -> 200 { "version":1041, "experiments":[ ...compact rules, salts, splits... ] }
     or 304 Not Modified   # SDK polls ~every 60s; most polls are 304.

# --- Data plane: event ingestion (high volume, fire-and-forget) ---
POST /api/v1/events   # batched by the SDK
  Body: { "events":[
     { "event_id":"e_9f3...", "user_id":"u_8123", "experiment_id":"exp_42",
       "variant":"green", "type":"exposure", "ts":1752745260 },
     { "event_id":"e_a01...", "user_id":"u_8123", "experiment_id":"exp_42",
       "variant":"green", "type":"conversion", "metric":"purchase",
       "value":49.90, "ts":1752745320 } ] }
  -> 202 Accepted; lands on the event log. event_id makes retries idempotent.

Assignment has no runtime endpoint - the SDK computes variant = bucket(hash(user_id + salt)) locally from cached config. This is the point of the design: the highest-QPS operation makes zero network calls.

Data store schemas

1. Config Store - relational (SQL, e.g. Postgres). Small, low-write, must be consistent and durable; the source of truth for experiment definitions. Relational because experiments/variants/layers are structured, relational, and edited transactionally - and the volume is tiny (thousands of rows).

experiments(
  experiment_id PK, name, layer_id FK, salt, status,           -- status: draft/running/stopped
  targeting_json, primary_metric, test_type, start_ts, end_ts,
  created_by, created_ts )
variants(
  variant_id PK, experiment_id FK, key, split_pct, bucket_lo, bucket_hi )  -- bucket range
layers(
  layer_id PK, name, salt )                                    -- for mutual exclusion
  INDEX on experiments(status) -- to compile the active bundle fast

2. Config Bundle - a compiled, versioned, cacheable blob (CDN object + in-SDK RAM). The Distributor compiles active experiments into a compact bundle SDKs cache. Not a DB - a versioned artifact, served from a CDN, refreshed by delta.

{ "version":1041,
  "layers":[ {"id":"checkout","salt":"L_ck9"} ],
  "experiments":[
    {"id":"exp_42","salt":"s_42abc","layer":"checkout","bucket_lo":0,"bucket_hi":1000,
     "variants":[{"key":"control","lo":0,"hi":500},{"key":"green","lo":500,"hi":1000}],
     "targeting":{"country":["US","CA"],"platform":["ios","android"]}} ] }
   (~few MB for 10k experiments; gzipped; SDK holds it in RAM, evaluates locally)

3. Event Log - durable partitioned stream (Kafka), partitioned by experiment_id. The source of truth for results and the raw fuel for re-analysis. Append-only, replicated, replayable; retained ~90 days hot, then archived to object storage. Not a queryable DB.

Topic: events   (partition = hash(experiment_id); sub-partition hot experiments)
  { event_id, user_id, experiment_id, variant, type, metric, value, ts }
  ~1.2M events/sec peak; replicated; replayable to recompute any aggregate.

4. Metrics Store - a counter/time-series store (e.g. a wide-column store like Cassandra, or a purpose-built OLAP counter store), keyed by (experiment_id, variant, metric). Holds the pre-aggregated sufficient statistics the dashboard and stats service read. Small (aggregates, not raw events); write-heavy from the stream, point-read by the dashboard. NoSQL/columnar because the access pattern is high-throughput counter increments and keyed point reads, not relational joins.

Key: (experiment_id, variant, metric, time_bucket)  ->
   { n, conversions, sum, sum_sq }         -- sufficient statistics
   time_bucket = per-minute; roll up to hour/day for the dashboard.
   ~millions of small rows total; GBs, not TBs. Regenerable by replaying the log.

5. Raw Event Archive - object storage (S3/GCS), partitioned by day/experiment. Cold, cheap, immutable copy of every event for re-analysis (new metric definitions), audit, and disaster recovery of aggregates. This is what makes the Metrics Store safely regenerable.

Why these choices: the config is small, structured, transactional - SQL is exactly right and its tiny volume makes relational overhead free. The config bundle is read-by-500M-SDKs, tiny, cacheable - a versioned CDN blob, not a per-request DB read. Events are an append-only firehose, replayable - a partitioned log (Kafka), where SQL would be catastrophic. Metrics are aggregate counters, write-heavy, keyed point-reads - a columnar/counter store, not a raw-event scan. The archive is big, cold, immutable - object storage. Each layer gets the store its access pattern demands - SQL for the small consistent config, a log for the event firehose, a counter store for aggregates, object storage for the cold raw archive - and, crucially, there is no store at all on the assignment path because assignment is computed, not stored.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Assignment on the request path (breaks first if done wrong, catastrophically). ~4.6M evaluations/sec, each on the render path - any network call is fatal. Fix: assignment is a local in-SDK hash (bucket(hash(user_id + salt))), zero I/O, zero stored state, deterministic. The only network dependency is fetching the small config bundle, which is CDN-cached, delta-polled (~60s), and held in RAM. The service never answers per-evaluation calls; it only ships config. This removes the single biggest scaling wall by not building the wall.

2. Config distribution fan-out. 500M SDKs need the current ruleset within seconds of a change (new experiment, ramp, kill switch). Fix: the config bundle is a versioned CDN object; SDKs poll GET /config?since_version=N and get a cheap 304 when unchanged, a small delta when changed. Kill switches propagate in seconds because the bundle is tiny and CDN-distributed. For faster propagation, a streaming push channel (server-sent events / websockets to a fan-out tier) notifies SDKs to refetch. Either way it is a cacheable read-heavy fan-out of a few MB, not a per-user computation.

3. Event ingest write volume. ~1.2M events/sec at peak; no DB absorbs that synchronously. Fix: events are async, batched, fire-and-forget from the SDK to an ingest gate that appends to a partitioned Kafka log. All aggregation is downstream and async. The write path is a durable append; nothing on the user’s interaction path waits for it.

4. Hot experiment / viral partition. One experiment (a homepage test everyone sees) concentrates a huge share of events on its experiment_id partition, overloading that partition and its aggregation task. Fix: sub-partition hot experiments - key by hash(experiment_id, variant, random_suffix) so a single experiment’s firehose spreads across many partitions; the aggregation job sums the per-sub-partition counters into the experiment total. Detect hot experiments by throughput and auto-scale their partition count. Because aggregation is commutative counter addition, splitting and re-summing is trivially correct.

5. Aggregation-job dedup state. Idempotent counting needs a seen-event_id set; at 1.2M events/sec the naive “remember every event_id forever” is unbounded. Fix: dedup over a bounded window (e.g. last 10-15 minutes of event_ids in Flink’s RocksDB-backed keyed state), sized to cover realistic SDK retry delays; older duplicates are astronomically rare and caught by the periodic re-aggregation from the archive. Keying dedup state by experiment_id (same as the partition) keeps it local - no cross-partition lookup per event.

6. Metrics read hotspots on the dashboard. A popular experiment’s results page is refreshed constantly; the stats service recomputes significance per read. Fix: significance is microsecond math over a handful of counters, so recompute is cheap; still, cache the computed results per experiment for a short TTL (seconds) so a hundred dashboard refreshes hit one computation. The Metrics Store is keyed by experiment_id so reads spread across experiments; no single hot key beyond the caching layer.

7. Statistical-integrity failures (the subtle bottleneck that corrupts output, not throughput). Peeking inflates false positives; SRM silently invalidates results; counting evaluations instead of exposures biases the denominator. Fix: peeking-safe sequential tests (always-valid intervals) so live monitoring cannot manufacture significance; an automatic SRM chi-square check on every results computation that flags and quarantines mismatched experiments; and exposure-triggered logging (count only rendered variants, deduped by event_id). The platform enforces the statistics so users cannot accidentally cheat them - the integrity fixes matter as much as the throughput fixes.

8. Shard keys, stated plainly. Event log -> experiment_id (keeps each experiment’s events together for local, coordination-free aggregation; sub-partitioned by (experiment_id, variant, suffix) for hot experiments to spread the firehose). Metrics Store -> (experiment_id, variant, metric, time_bucket) (per-experiment point reads spread across thousands of experiments; no natural hot key beyond a caching layer). Config Store -> tiny, unsharded relational. Assignment -> no shard key, because there is no stored assignment; the hash is the sharding of users into buckets, computed at the edge. Sharding events by user_id would scatter one experiment’s data across all partitions and force a cross-partition aggregate; experiment_id keeps aggregation local. The experiment is the unit of aggregation and the write key; the user is bucketed by a stateless hash and never stored.

9. Single points of failure and graceful degradation. The Config API and Distributor are stateless and horizontally scaled behind a CDN; if they are fully down, SDKs keep serving from last-cached config - assignments stay stable and correct, only new experiments/ramps are delayed. Kafka is replicated; the aggregation job checkpoints and resumes (at-least-once + idempotent dedup means resume never double-counts). The Metrics Store is regenerable by replaying the archive, so it needs availability, not durability. The critical resilience property: assignment fails safe to control/default from cached config and never blocks or errors the page - experimentation is best-effort personalization on top of a guaranteed working product. No single component’s failure blanks or breaks the app.

10. Multi-region. The control plane can run centrally (experiments are defined rarely); the config bundle replicates to every region’s CDN edge, so SDKs everywhere fetch locally. Event ingest is region-local (Kafka per region) to avoid cross-region latency on the write path; regional event streams are aggregated per region and the per-region counters are summed into a global metric (again, commutative counter addition makes this exact). A user assigned in one region gets the identical variant in another because assignment is a pure hash of the same user_id + salt - determinism makes assignment automatically consistent across regions with zero replication. That is the quiet payoff of computing rather than storing assignment: the hardest multi-region problem (consistent per-user state everywhere) simply does not exist here.

Wrap-Up

The trade-offs that define this design:

  • Compute assignment, do not store it. Assignment is a pure deterministic hash - bucket(hash(user_id + salt)) - evaluated locally in the SDK, so there is no assignment table (which would be hundreds of billions of rows), no per-evaluation network call (which would add latency to every page render), and no cross-region replication of per-user state (determinism makes it consistent everywhere for free). A stored-assignment table was rejected on storage, latency, and consistency all at once.
  • Independence by salt, isolation by layer. Per-experiment salts make thousands of concurrent experiments statistically independent by default (the common, non-interacting case), while a layer/domain model makes interacting experiments mutually exclusive by design (clean attribution on shared surfaces). This is what lets experiment count scale to thousands without them silently poisoning each other.
  • Append durably, count idempotently. Events are an async, batched, fire-and-forget append to a durable partitioned log, aggregated downstream by a stream job that dedups by event_id - achieving exact counts under retries via at-least-once delivery plus idempotent processing, never via impossible exactly-once delivery. Synchronous per-event DB writes and counter transactions were rejected as impossible at 1.2M events/sec and unsafe under retries.
  • Pre-aggregate, then read; and make the test peeking-safe. Significance is microsecond math over continuously-maintained sufficient statistics (n, conversions, sum, sum_sq), never a scan of the 6.8 TB/day raw log - so results are near-real-time and cheap. Validity is guaranteed by choosing a monitoring-safe sequential test and by auto-checking sample-ratio mismatch, so the platform enforces correct statistics rather than trusting a human not to peek. Batch-scan-and-eyeball was rejected as slow, unscalable, and statistically dishonest.
  • Fail safe over fail hard. If the control plane is down, SDKs serve last-cached config and keep assigning correctly; if a stage of ingestion lags, events buffer and replay; if the metrics store is lost, it regenerates from the archive. The engineering effort goes into a guaranteed working default (control variant, always) with experimentation layered on top - never into strong consistency the problem does not need.

One-line summary: an experimentation platform that assigns 500M users to variants across thousands of concurrent experiments by a stateless per-experiment-salted hash computed locally in the SDK (no stored state, no network call, deterministic and consistent everywhere), captures exposures and conversions as an idempotent append-only event stream, and rolls them into continuously-updated sufficient statistics so a peeking-safe sequential significance test with an automatic sample-ratio-mismatch guard reads out real-time, statistically valid results - so assignment is free and correct on the hot path, events are lossless and exactly counted, and the numbers the dashboard reports are actually the numbers.