Everyone models an ad server as a lookup: “the user opens the feed, we fetch an ad, we show it, done.” The interviewer lets that sit for a moment and then piles on the constraints that make it a real system. There are not ten ads to choose from, there are millions of live campaigns; the best ad for this user has to be chosen from that pool in under 50 milliseconds while the feed is already rendering; advertisers set budgets and if you overspend theirs you eat the cost, if you underspend it you lose their trust and their business; the spend has to be paced evenly across the day, not blown in the first hour; a large fraction of clicks are bots and you must not charge advertisers for them; and all of this runs at 100 billion impressions a day. An ad server is not a lookup. It is a real-time per-impression auction over a targeted candidate set, constrained by budgets and paced over time, with a fraud filter in the loop - and the ad shown is just its output.

The whole design comes down to four decisions: how you narrow millions of campaigns down to the few hundred eligible for one impression fast enough to matter (targeting and candidate retrieval), how you score and rank those candidates and run the auction inside a 50ms budget (ranking and the auction), how you enforce and pace budgets across thousands of ad servers without double-spending or overspending (distributed budget control), and how you keep bots from draining budgets and corrupting the feedback loop (fraud filtering). Let me build it properly.

Functional Requirements (FR)

In scope:

  • Campaign management (advertiser side). Advertisers create campaigns: an objective (clicks, impressions, conversions), a total and/or daily budget, a bid, targeting rules (geo, age, gender, interests, device, lookalike audiences), creatives, and a schedule (start/end dates, dayparting). They can pause, edit, and top up budgets, and see delivery/spend reporting.
  • Ad selection per impression (user side). When a user’s feed is assembled and an ad slot appears, the system selects the single best eligible ad for that user in that slot, in real time, under a hard latency budget.
  • The auction. Among eligible candidates, rank by expected value to the platform (a function of bid and predicted engagement), pick the winner, and compute what the winner pays (second-price or a variant).
  • Budget enforcement and pacing. Never spend more than a campaign’s budget. Spread the daily budget smoothly across the day (pacing) rather than exhausting it in the first hour.
  • Fraud / invalid-traffic filtering. Detect and discard bot impressions and clicks so advertisers are not billed for invalid traffic, and so the ranking models are not poisoned by fake engagement.
  • Event logging and billing. Record every impression, click, and conversion durably, deduplicate them, and produce the spend that bills the advertiser and updates the budget.
  • Feedback loop. Feed engagement events back into the engagement-prediction models so ranking improves.

Explicitly out of scope (say this out loud so you own the scope):

  • The organic feed ranking. How posts from friends/pages are ranked is a separate system. I consume “a feed is being built for user U and here are the ad slots” and I fill the slots; I do not rank the organic content.
  • The billing/payments backend. Charging the advertiser’s card, invoicing, credit, and tax are a separate financial system. I produce the durable, deduplicated, billable spend events and hand them off; I do not design the ledger or the card processor.
  • Creative storage and moderation. Storing image/video creatives on a CDN and the content-policy review that approves an ad are adjacent systems. I reference an approved creative by ID.
  • Training the ML models. I design the serving of a CTR/CVR model (low-latency scoring) and the pipeline that feeds it labeled events, but not the model architecture or the offline training job itself.

The one decision that drives everything: ad selection is a funnel, not a scan. You cannot score millions of campaigns per impression. You retrieve a few hundred candidates by targeting, rank those, and auction among the top few. Every hard problem here is about making one stage of that funnel fast, correct, or economically honest.

Non-Functional Requirements (NFR)

  • Scale: 100B impressions/day. That is 100e9 / 86400 ≈ 1.16M impressions/sec average, and peak is 2-3x that, so design for ~3M ad requests/sec. Millions of active campaigns. Clicks are ~1-2% of impressions; conversions are a small fraction of clicks.
  • Latency: the entire ad-selection call has a hard budget of 50ms p99. It runs inside the feed-assembly request; the feed cannot wait. If ad selection is slow, the slot ships empty (or with a house ad), never blocks the feed.
  • Availability: 99.99%+ on the serving path. An ad server failing must degrade gracefully - drop the slot or serve a cached/house ad - never fail the feed request.
  • Consistency: ad ranking is eventually consistent (a CTR estimate 30s stale is fine). Budget enforcement is where consistency bites: it must be strong enough to not materially overspend, but we accept small, bounded overspend at the edges in exchange for not doing a global transaction per impression. Billing/spend records must be exactly-once for money.
  • Durability: every billable event (impression that costs money, click, conversion) must be durably logged and deduplicated. Losing a click means losing revenue; double-counting means overbilling an advertiser.
  • Freshness: budget-spent counters must be fresh to within a second or two across the fleet, or fast-spending campaigns overshoot. Targeting/campaign config changes (pause, budget edit) must propagate to all ad servers within seconds.
  • Fairness/economics: the auction must be truthful enough that advertisers bid honestly, and pacing must be smooth so a campaign is not starved late in the day.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Request rate:

100B impressions/day = 100e9 / 86400 ≈ 1.16M/sec average.
Peak (prime time, ~2.5x) ≈ 3M ad requests/sec.
Each ad request may fill 1 slot but evaluates ~hundreds of candidates.

Candidate funnel per impression:

Millions of live campaigns -> targeting index narrows to ~1,000 eligible
  -> budget/pacing filter -> ~500 -> light ranking (cheap CTR) -> top ~50
  -> full ranking (heavy CTR/CVR model) -> auction over ~10 -> 1 winner.
Scoring 500 candidates * 3M req/sec = 1.5B candidate-scorings/sec.
That is why the funnel must cut hard and early: full ML scoring runs on ~50,
not 500, and never on millions.

Storage - event logs (the firehose):

Impressions: 100B/day. Each logged event ~400 bytes (ids, user ctx, ts, price).
  100e9 * 400B = 40 TB/day of impression logs.
Clicks ~1.5% = 1.5B/day * 400B = 0.6 TB/day.
Conversions ~ smaller.
Raw event storage ≈ ~40 TB/day ≈ ~15 PB/year (compressed, tiered to cold).
Aggregated/rolled-up analytics are far smaller and kept hot.

Storage - campaign metadata:

Say 10M active campaigns * ~5 KB (targeting, budget, creatives refs, bid)
  = 50 GB. Small. Fits in memory, fully replicated to every ad server region.
Targeting index (inverted: attribute -> campaign set) similar order, indexed.

Budget counters (the hot mutable state):

10M campaigns, each needs a live "spent today" counter updated on every
billable event. Peak billable events (impressions priced + clicks) ~ a few
hundred k/sec of counter increments, but these are spread across campaigns.
Counter store: 10M keys * ~100 bytes ≈ 1 GB. In-memory, sharded.

Bandwidth:

Ad response payload is small (creative refs + tracking urls), say ~2 KB.
3M req/sec * 2 KB = 6 GB/sec egress of ad responses. Creatives themselves
served from CDN, not from the ad server.

The takeaways: a tiny campaign metadata set (50 GB) that must be replicated everywhere and read hot; a brutal candidate funnel that must cut millions to tens before any expensive scoring; a firehose of event logs (~40 TB/day) that is write-heavy and must be durably captured then rolled up; and a hot budget-counter plane (~1 GB) that must be kept fresh across a global fleet without a per-impression global transaction. The funnel, the budget plane, and the event pipeline are the design drivers - not raw storage.

High-Level Design (HLD)

Three planes. A serving plane that answers ad requests in <50ms (stateless ad servers reading replicated metadata + indexes + a live budget cache). A budget/pacing control plane that aggregates spend and pushes back throttle signals. An event/billing pipeline that captures the firehose, filters fraud, deduplicates, bills, and feeds the models.

                         ┌────────────────────────────────────────┐
  Advertiser ── UI ──▶   │  Campaign Service (CRUD, budget, target)│
                         │   -> Campaign DB (Postgres, source of   │
                         │      truth) -> publishes config changes │
                         └──────────────┬─────────────────────────┘
                                        │ config stream (Kafka)
                       ┌────────────────▼───────────────────┐
                       │  Config distribution -> replicated  │
                       │  Campaign Cache + Targeting Index   │
                       │  on EVERY ad server (in-memory)     │
                       └────────────────┬───────────────────┘
                                        │  (read)
  Feed Service ──ad request──▶ ┌────────▼─────────┐   GET spent:{cid}
  (user U, slot,               │   AD SERVER       │◀────────────────┐
   context)                    │  (stateless):     │                 │
                               │  1 retrieve cand. │        ┌────────▼────────┐
        ◀── winning ad ────────│  2 budget filter  │        │  Budget Cache    │
             (creative ref,    │  3 rank (CTR/CVR) │        │ (Redis, sharded, │
              tracking urls)   │  4 auction        │        │  spent counters) │
                               └────────┬──────────┘        └────────▲────────┘
                                        │ impression/click log       │ push spend
                                        ▼                            │ + pace signal
                       ┌────────────────────────────────────┐        │
                       │  Ingestion (Kafka): impressions,    │  ┌─────┴──────────┐
                       │  clicks, conversions (firehose)     │  │ Budget/Pacing   │
                       └────────────────┬───────────────────┘  │ Aggregator      │
                                        │                       │ (Flink)         │
              ┌─────────────────────────┼───────────────────────┘                │
              ▼                         ▼                        ▲                │
     ┌────────────────┐       ┌──────────────────┐               │                │
     │ Fraud Filter    │       │  Dedup + Join     │──billable────┘                │
     │ (real-time +    │──────▶│  (impression↔     │  spend                        │
     │  async scoring) │ valid │   click match)    │                               │
     └────────────────┘        └────────┬─────────┘                                │
                                        │ clean events                             │
                    ┌───────────────────┼──────────────────┐                       │
                    ▼                   ▼                   ▼                       │
          ┌──────────────┐    ┌──────────────┐    ┌──────────────┐                 │
          │ Billing spend │    │ Model training│    │ Analytics /   │◀────reporting─┘
          │ (ledger)      │    │ feature store │    │ warehouse     │
          └──────────────┘    └──────────────┘    └──────────────┘

Write path (advertiser side), infrequent:

  1. Advertiser creates/edits a campaign in the Campaign Service. It validates and writes to the Campaign DB (Postgres - relational, transactional, the source of truth for budgets and targeting).
  2. Every config change (new campaign, pause, budget edit, targeting change) is published to a Kafka config stream. A distribution layer applies changes to the in-memory Campaign Cache and Targeting Index replicated on every ad server, within seconds. This is what makes selection a pure in-memory operation - no ad server ever queries Postgres on the hot path.

Read path (ad request), 3M/sec, must finish in 50ms:

  1. The Feed Service assembling a user’s feed hits an ad slot and calls an Ad Server with the user context (user id, resolved audience/interest attributes, geo, device, slot).
  2. The ad server runs the funnel: (a) retrieve candidates from the Targeting Index by the user’s attributes; (b) drop candidates whose budget is exhausted or paced-out using the Budget Cache; (c) score survivors with a cheap model, keep the top ~50, score those with the full CTR/CVR model; (d) run the auction - rank by eCPM = bid * predicted_engagement, pick the winner, compute the second-price payment.
  3. The ad server returns the winning creative reference plus signed tracking URLs, and logs the impression to the ingestion firehose.

Event path (billing + feedback), continuous and asynchronous:

  1. Impressions/clicks/conversions land in Kafka. The Fraud Filter scores them; invalid traffic is dropped (and never billed).
  2. Clean events are deduplicated and joined (a click is matched to its impression). Billable spend is pushed to the Budget/Pacing Aggregator (Flink), which updates authoritative per-campaign spend and pushes fresh counters + pacing throttle signals back into the Budget Cache.
  3. Clean events also flow to the billing ledger, the model feature store/training pipeline, and the analytics warehouse for advertiser reporting.

The key insight: the hot path is entirely in-memory and read-only. Millions of campaigns become a few hundred candidates via a pre-built in-memory index, budgets are checked against a cached counter (not a database), and the expensive ML runs on tens of candidates. Everything durable, mutable, or slow - config writes, spend aggregation, fraud, billing - happens off the request path.

Component Deep Dive

The hard parts, naive-first then evolved: (1) candidate retrieval from millions of campaigns, (2) ranking and the auction inside 50ms, (3) budget enforcement and pacing across a distributed fleet, (4) real-time fraud filtering.

1. Candidate retrieval - millions of campaigns to hundreds

Naive approach: on each impression, loop over all campaigns and check targeting. For the user, iterate every live campaign, evaluate its targeting predicate (geo matches? age matches? interest matches?), collect the ones that pass.

Where it breaks:

  • O(campaigns) per impression. 10M campaigns * 3M impressions/sec = 3e13 predicate evaluations/sec. Impossible. Even 10M evaluations for a single impression blows the 50ms budget by orders of magnitude.
  • The work is embarrassingly redundant. Most campaigns fail on the first attribute (wrong country). You are paying to reject millions of obvious non-matches per impression.
  • Predicate complexity. Targeting is a conjunction of many attributes with OR-sets inside (interests IN {…}, age IN range, device IN {…}). Evaluating that per campaign per impression is expensive even ignoring the count.

First evolution: an inverted index from attribute value to campaign set. Build, offline and incrementally, an index: for each attribute value (country=IN, interest=cricket, age_bucket=25-34, device=android), store the set of campaign IDs that target it. On an impression, take the user’s attribute values, look up the matching campaign sets, and intersect/union them. This flips the cost from “scan all campaigns” to “gather a few posting lists and combine them” - the same trick a search engine uses. Retrieval becomes proportional to the number of matching campaigns, not the total.

But targeting is not a pure AND of single values. A campaign targets country=IN AND interest IN {cricket, movies} AND age IN {18-24, 25-34}. That is a conjunction of disjunctions. Naive intersection of posting lists does not express “user matches if their interest is in the campaign’s allowed set.”

The answer: a Boolean-expression inverted index with conjunction/disjunction handling and a matcher engine.

  • Index each campaign’s targeting as a Boolean expression over attributes. Decompose it into (attribute, allowed-value-set) clauses. For a disjunctive clause like interest IN {cricket, movies}, the campaign appears in the posting list of both interest=cricket and interest=movies.
  • At query time, gather the posting lists for all of the user’s attribute values (the user’s country, each of their interests, their age bucket, their device). A campaign is a candidate only if, for every one of its required attribute dimensions, at least one of the user’s values hit a posting list. Use the classic approach (like the “Indexing Boolean Expressions” / DNF matching technique): index the number of conjuncts each campaign requires, and count how many the user satisfied; a campaign matches when its satisfied-conjunct count equals its required count.
  • Handle NOT/exclusion targeting (e.g. “exclude users who already converted”) with a separate exclusion posting list checked after the positive match.
  • Layer coarse partitioning on top. Shard the index by the highest-cardinality mandatory attribute (usually country/geo), so an impression only ever touches its region’s slice of the index. This keeps posting lists short.
Targeting index (in-memory, per ad server, rebuilt from config stream):
  country=IN    -> {c12, c88, c91, ...}
  interest=cricket -> {c88, c204, ...}
  age=25-34     -> {c12, c88, ...}
  device=android-> {c88, c91, ...}
  campaign_conjunct_count: c88 -> 4 (needs country+interest+age+device)

Retrieval for user{IN, cricket, 25-34, android}:
  gather posting lists for each user attribute value
  for each candidate campaign, count satisfied conjuncts
  keep campaigns whose satisfied == required   -> ~1,000 candidates

This cuts millions to ~1,000 in a few milliseconds, entirely in memory. The index lives on every ad server and is updated incrementally from the config Kafka stream, so a paused campaign or a targeting edit disappears from candidacy within seconds without any hot-path database call. Retrieval is the first and biggest cut of the funnel; get it wrong and nothing downstream has a latency budget left.

2. Ranking and the auction inside 50ms

Naive approach: run the full CTR/CVR model on every candidate and sort. For each of the ~1,000 candidates, run the deep engagement model to predict click probability, compute bid * pCTR, sort, take the top one.

Where it breaks:

  • The heavy model is too slow at that fan-out. A deep CTR model might cost ~100 microseconds per candidate on GPU/optimized inference. 1,000 candidates * 100us = 100ms - already double the entire request budget, and that is before retrieval, budget checks, and the auction.
  • Most candidates are hopeless. A campaign with a tiny bid and mediocre relevance will never win; spending full model inference to rank it 400th is wasted compute.
  • Feature fetching dominates. The heavy model needs user features and ad features; fetching and assembling features for 1,000 candidates is itself expensive.

The fix: a multi-stage ranking cascade - cheap models cut the field before expensive ones run.

  • Stage 0 - budget/pacing filter (from section 3). Drop candidates whose budget is exhausted or whose pacing throttle says “skip this impression.” This can remove a large fraction cheaply, before any scoring. ~1,000 -> ~500.

  • Stage 1 - lightweight scorer. A cheap model (logistic regression / a small tree ensemble, or a cached historical CTR per (campaign, coarse-context)) gives a rough pCTR. Compute approx eCPM = bid * approx_pCTR for all ~500 survivors. This is nanoseconds-to-microseconds each. Keep the top ~50 by approx eCPM.

  • Stage 2 - full model. Run the heavy CTR/CVR model with full user+ad cross features only on the top ~50. 50 * 100us = 5ms - fits the budget. Produce accurate pCTR (and pCVR for conversion objectives).

  • Compute eCPM per candidate. The platform ranks by expected revenue per impression:

    • For CPC (pay per click): eCPM = bid_cpc * pCTR * 1000.
    • For CPA (pay per action): eCPM = bid_cpa * pCTR * pCVR * 1000.
    • For CPM (pay per impression): eCPM = bid_cpm. This normalizes every campaign to “expected revenue per thousand impressions” so heterogeneous bid types compete on one axis.

  • The auction: rank by eCPM, apply a second-price (generalized second-price / VCG-style) payment. The highest eCPM wins the slot. The winner does not pay their own bid; they pay the minimum they would have had to bid to still beat the runner-up. Concretely, for CPC:

    winner pays per click:  price_cpc = (eCPM_2nd / (pCTR_winner * 1000)) + epsilon

    where eCPM_2nd is the second-highest candidate’s eCPM. This is the truthful-auction mechanism: it makes bidding your true value the dominant strategy, so advertisers do not have to game their bids. Add reserve prices (a floor below which no ad shows, protecting user experience and preventing near-zero-value ads).

  • Blend in a relevance/quality term. Pure eCPM ranking floods users with high-bid, low-quality ads and degrades the feed. The real ranking score is bid * pCTR * quality_factor (an “ad rank” a la search ads), where quality penalizes low-relevance or high-complaint creatives. This aligns advertiser value with user experience.

funnel latency budget (target p99 < 50ms):
  retrieve candidates (index)        ~3-5ms   1M campaigns -> ~1,000
  budget/pacing filter               ~1ms     ~1,000 -> ~500
  stage-1 cheap scorer               ~2ms     ~500 -> top 50
  feature fetch + stage-2 heavy model ~10ms   50 scored accurately
  auction + pricing + policy checks  ~2ms     1 winner + price
  serialize response + log           ~2ms
  --------------------------------------------------------------
  headroom for network + tail        remaining budget

The mental model: ranking is a funnel of increasing cost and decreasing count. Each stage spends more compute per candidate but on fewer candidates, so total cost stays bounded and the 50ms budget holds. The auction on top makes the economics truthful; the quality term keeps the feed livable.

3. Budget enforcement and pacing across a distributed fleet

This is the “hard part” the problem foregrounds, and it is where naive designs quietly lose money.

Naive approach: keep spent in the database; on each impression, read spent, compare to budget, and if under budget serve and increment. SELECT spent FROM campaigns WHERE id=?; if spent < budget: serve; UPDATE spent = spent + price.

Where it breaks:

  • A database write per impression. 3M impressions/sec, each doing a read-modify-write on a campaign row. No relational database survives that, and a hot campaign is a single-row hotspot that serializes.
  • The read-modify-write race across thousands of ad servers. Thousands of servers concurrently read spent=990, all see budget 1000, all serve, all increment. The campaign overspends massively. A per-impression database transaction to prevent it would add tens of ms and a global lock - fatal to the latency budget.
  • No pacing. Even if enforcement were correct, a campaign with a $10,000 daily budget and a high bid would win every auction at 6am and exhaust the budget by 7am, starving the advertiser of the rest of the day’s audience. Advertisers want smooth delivery, not a morning spike.

Fix part A - distributed budget counters with local allocation (leaky enforcement, corrected async).

  • Authoritative spend lives in the Budget/Pacing Aggregator, which consumes the billable event stream and knows true spend per campaign within a second or two. It writes the live spent:{campaign} into a sharded in-memory Budget Cache (Redis), which ad servers read on the hot path (a single O(1) GET, cached locally for ~1s).
  • Ad servers do not increment the global counter per impression. Instead the control plane hands each ad server a local budget lease - a slice of the remaining budget it is allowed to spend before checking back (e.g. “you may spend $50 of campaign C”). The ad server decrements its local lease as it serves, and requests a new lease when it runs low or it expires. This is the standard way to avoid a global write per impression: distribute the budget as leases, reconcile centrally.
  • Accept bounded overspend at the boundary. When a campaign nears its cap, the last outstanding leases across servers can slightly overspend. You bound this by shrinking lease sizes as the remaining budget shrinks (give out pennies, not dollars, near the cap) and by hard-stopping via a fast “campaign exhausted” broadcast on the config/pacing channel. The trade is explicit: a tiny, bounded overspend (fractions of a percent) in exchange for never doing a global transaction per impression. For most ad systems that is the right trade; the alternative (strict no-overspend) costs latency the feed cannot pay.

Fix part B - pacing (spreading the budget over time).

  • Compute a target spend curve per campaign. The simplest is uniform: target_spend_by(t) = daily_budget * (t / day_length). Better curves weight by predicted traffic/opportunity so the campaign spends more when its audience is actually online (dayparting-aware).
  • Throttle probabilistically to track the curve. The pacing aggregator compares actual spend to the target curve and emits a pacing rate per campaign in [0,1]. The ad server, when a paced campaign is a candidate, admits it into the auction only with probability = pacing rate (a coin flip per impression). If a campaign is spending ahead of its curve, its rate drops toward 0 and it wins fewer auctions; if behind, the rate rises toward 1 so it catches up. This is a feedback controller on spend velocity: measure spend vs. target, adjust admission probability, repeat every few seconds.
  • Why probabilistic and not a hard gate. A hard on/off gate makes delivery bursty (fully on, then fully off). Probabilistic throttling produces smooth, continuous delivery and naturally spreads a campaign across many servers and time buckets without coordination - each server independently flips a coin at the current rate.
  • Combine with the auction correctly. Pacing modifies participation, not the bid math. A throttled-out candidate simply does not enter this impression’s auction; the ones that do compete at their true eCPM. (Some systems instead implement pacing as a bid multiplier - lowering effective bid to spend slower - which is smoother economically but more complex; probabilistic admission is the simpler, common answer.)
Budget/Pacing Aggregator (Flink), every ~2s per campaign:
  actual = spend_so_far_today(campaign)
  target = pacing_curve(campaign, now)          # planned spend by now
  if actual >= daily_budget:  rate = 0; broadcast EXHAUSTED
  elif actual >  target:      rate = clamp(rate * 0.9, 0, 1)   # spending fast -> slow down
  elif actual <  target:      rate = clamp(rate * 1.1, 0, 1)   # behind -> speed up
  write pace:{campaign} = rate, spent:{campaign} = actual  -> Budget Cache
  refresh leases based on remaining budget (smaller near the cap)

Ad server, per candidate campaign on the hot path:
  if spent:{campaign} >= budget: drop
  if local_lease(campaign) <= 0: drop (and async request more)
  if random() > pace:{campaign}: drop            # pacing coin flip
  else: keep in auction; on win, decrement local_lease

The principle: budget is enforced by distributing spendable leases and reconciling centrally, and paced by a feedback controller that turns “spend velocity vs. plan” into an admission probability. You never put a database write or a global lock on the impression path; you accept a tiny bounded overspend as the price of that, and you shrink it near the cap.

4. Real-time fraud filtering

Naive approach: count every impression and click, bill for all of them, clean up later in analytics. Serve, log, bill, and if fraud is found next week, issue a credit.

Where it breaks:

  • You bill advertisers for bots. A large fraction of raw web traffic is automated. Charging advertisers for bot clicks and refunding later erodes trust and invites disputes and churn.
  • Budgets drain to fraud. Bot clicks burn real budget in real time, so legitimate audiences get starved by fake engagement - the pacing controller sees “spend” that bought nothing.
  • The models get poisoned. Fake clicks feed the CTR training pipeline, teaching the ranker that a fraudulent pattern is “engaging,” which then wins more auctions. Fraud in the feedback loop compounds.

The fix: a two-tier filter - fast inline heuristics on the serving/ingest path, and heavier asynchronous scoring before billing and before the counter update.

  • Tier 1 - inline, cheap, on the request/ingest path. Rule- and signal-based checks that run in microseconds: rate limits per user/IP/device, known-bad IP/data-center ranges, missing or malformed device signatures, impossible click timing (click before the ad could have rendered), and simple velocity checks (this device clicked 40 ads in 2 seconds). These block the obvious junk immediately, so it never counts and never bills. Cheap enough to run in the ~50ms budget or at the ingestion edge.
  • Tier 2 - asynchronous, heavier, before spend is finalized. In the event pipeline, before an event is marked billable and before it increments the authoritative spend counter, run a heavier fraud model: behavioral features (session patterns, click-through anomalies, conversion mismatch), device-graph clustering (many “users” sharing fingerprints), and ML classifiers trained on labeled invalid traffic. Events flagged here are diverted to a discarded stream - not billed, not fed to model training, not counted toward budget.
  • Delayed/held billing window. Do not bill instantly on the click. Hold billable events in a short buffer (seconds to minutes) so tier-2 scoring and impression-click join can complete; only then finalize spend. This is why the budget counter is fed by the aggregator after filtering, not by the ad server at serve time - the money-affecting number is always the post-fraud one.
  • Feedback and blocklists. Confirmed fraud updates blocklists (IP, device, publisher-placement) that tier-1 consults, so patterns caught by the slow path get blocked by the fast path next time. This is the same fast/slow reconciliation shape as the rest of the system.
  • Impression-click consistency as a fraud signal. A click with no matching served impression, or a click on an impression from a different user/session, is almost certainly invalid. The dedup-and-join stage that pairs clicks to impressions doubles as a fraud gate: unjoined clicks are suspect.
serve time (tier 1, inline):  block known-bad IP / DC ranges, rate limits,
                              impossible timing, malformed signatures
                              -> obvious bots never counted
ingest (Kafka) -> Fraud Filter (tier 2, async, held ~seconds):
  behavioral + device-graph + ML classifier
  join click <-> impression (unjoined click = suspect)
  valid  -> Dedup+Join -> billable spend -> counter update -> billing + training
  invalid-> discarded stream (audited, never billed, never trained on)
confirmed fraud -> update blocklists -> consulted by tier 1 next time

The principle: filter in two tiers and bill late. Catch the obvious inline so it never costs, catch the subtle asynchronously before money or model training is touched, and make the post-filter number the only one that ever affects budget, billing, or the ranker. The money-and-model path is always downstream of fraud, never upstream.

API Design & Data Schema

The advertiser side is REST/CRUD; the serving side is a single fast internal call; the event side is streams.

Advertiser-facing (campaign management)

POST /api/v1/campaigns
  body: { advertiser_id, name, objective:"clicks",
          daily_budget:10000, total_budget:300000, bid:{type:"cpc", amount:0.80},
          targeting:{ geo:["IN"], age:["25-34","35-44"], interests:["cricket"],
                      device:["android"], exclude_audiences:["converted_30d"] },
          creatives:["cr_991"], schedule:{start:"2026-07-24", end:"2026-08-24"} }
  -> 201 { campaign_id:"c_88", status:"pending_review" }

PATCH /api/v1/campaigns/c_88   body:{ daily_budget:15000 }   -> 200
POST  /api/v1/campaigns/c_88/pause                            -> 200

GET /api/v1/campaigns/c_88/report?from=..&to=..
  -> 200 { impressions:2.1e6, clicks:41000, ctr:0.019, spend:8200,
           avg_cpc:0.20, conversions:900, cpa:9.1 }

Serving (internal, feed service -> ad server)

POST /internal/v1/ad-request
  body: { user:{id, attrs:{geo,age,gender,interests[],device}},
          slot:{id, size, position}, context:{feed_session, ts} }
  -> 200 { ad:{ campaign_id:"c_88", creative_id:"cr_991",
                tracking:{ imp_url, click_url },   # signed, one-time
                price_cpc:0.62, auction_id:"a_7f3" } }
     -> 204 no-fill (no eligible ad above reserve; feed ships slot empty/house ad)

Event ingestion (ad server / client -> pipeline)

// impression (logged when the ad is shown)
{ "type":"IMPRESSION", "auction_id":"a_7f3", "campaign_id":"c_88",
  "user_id":"u_1", "creative_id":"cr_991", "price":0.0, "ts":... , "sig":... }
// click (client fires the signed click_url)
{ "type":"CLICK", "auction_id":"a_7f3", "campaign_id":"c_88",
  "user_id":"u_1", "ts":..., "sig":... }
// conversion (advertiser pixel / SDK)
{ "type":"CONVERSION", "click_id":..., "campaign_id":"c_88", "value":..., "ts":... }

Data stores - the right tool per plane

1. Campaign DB - PostgreSQL (relational, source of truth). Campaigns, budgets, targeting rules, creatives, schedules. Relational because these are structured, transactional (budget top-ups, edits must be ACID), moderate in volume (~10M rows), and read by humans for reporting. This is written rarely (advertisers edit campaigns) relative to the serving firehose, so a relational DB is perfect. It is never on the hot serving path - config is pushed to in-memory caches.

campaigns(
  campaign_id PK, advertiser_id FK, status, objective,
  daily_budget numeric, total_budget numeric, spent_total numeric,
  bid_type, bid_amount, start_date, end_date, created_at, updated_at )
  INDEX (advertiser_id, status), INDEX (status, end_date)
targeting(campaign_id FK, attr_key, attr_value)   -- normalized targeting clauses
  INDEX (attr_key, attr_value)   -- feeds the inverted index build
creatives(creative_id PK, campaign_id FK, cdn_url, format, review_status)

2. Campaign Cache + Targeting Index - in-memory, on every ad server. The ~50GB of campaign metadata and the inverted Boolean-expression index, replicated to every server and updated incrementally from the config Kafka stream. NoSQL/in-memory because access is “gather posting lists and campaign records by id” at microsecond latency - no joins, no transactions, pure key/set lookups.

in-memory:
  campaign:{cid} -> {bid, objective, creatives, quality, status}
  index[attr=value] -> sorted set of campaign_ids  (posting list)
  conjunct_count[cid] -> int

3. Budget Cache - Redis (sharded, in-memory), strongly-fresh counters. spent:{cid} and pace:{cid} per campaign, written by the aggregator, read O(1) by ad servers (with ~1s local cache). Sharded by campaign_id so a hot campaign’s counter lives on one shard and is not a global hotspot. In-memory and non-durable because it is derived from the durable event stream and can be rebuilt.

Redis (sharded by campaign_id):
  GET  spent:{cid}          # remaining budget check
  GET  pace:{cid}           # pacing admission probability
  # leases handled via a lease service / atomic DECR on a per-server sub-budget key

4. Event log / firehose - Kafka -> object storage + columnar warehouse. Impressions/clicks/conversions. Kafka for durable, replayable, high-throughput ingestion; raw events land in object storage (S3/Iceberg), rolled-up aggregates in a columnar warehouse (BigQuery/ClickHouse) for reporting. Append-only, immutable, partitioned by day/hour - the classic firehose shape. This is the one plane that must never lose a billable event.

Kafka topics: impressions, clicks, conversions, discarded(fraud)
  -> Flink: fraud filter, dedup (by auction_id/event id), impression<->click join
  -> billable spend  -> Budget Aggregator + billing ledger
  -> clean events    -> feature store (training) + warehouse (reporting)

Why Postgres for config, in-memory index for serving, Redis for budgets, Kafka+warehouse for events: the planes have opposite needs. Config is transactional and low-volume - relational. Serving needs microsecond set-lookups over a small replicated dataset - in-memory index. Budgets are hot mutable counters that must be fresh and sharded but are derived - in-memory Redis. Events are an unbounded, durable, replayable firehose feeding billing and ML - a log plus a warehouse. There is no single “ads database”; each plane gets the store that matches its lifetime and access pattern.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Scanning all campaigns per impression (breaks first). O(millions) targeting evaluations per impression is impossible at 3M req/sec. Fix: the inverted Boolean-expression targeting index in memory on every ad server, gathering posting lists for the user’s attributes and matching by conjunct count. Millions of campaigns become ~1,000 candidates in a few milliseconds. This is the single most important cut in the funnel.

2. Full ML scoring on every candidate. Running the heavy CTR model on ~1,000 candidates blows the 50ms budget. Fix: the multi-stage ranking cascade - budget/pacing filter, then a cheap scorer to keep the top ~50, then the heavy model on those 50 only. Cost stays bounded because each stage is more expensive but on fewer candidates.

3. Budget writes on the hot path. A DB read-modify-write per impression, plus the cross-server race, overspends and cannot scale. Fix: distributed budget leases (each server spends a local slice, reconciled centrally) plus an in-memory sharded counter read O(1) on the hot path. Accept a small bounded overspend near the cap; shrink lease size and broadcast “exhausted” to bound it.

4. Bursty, front-loaded delivery. A high-bid campaign exhausts its daily budget by morning. Fix: probabilistic pacing - a feedback controller compares spend to a target curve and emits an admission probability per campaign; ad servers flip a coin to admit the campaign into each auction. Smooth, coordination-free delivery.

5. Hot campaign / hot key. One viral campaign’s budget counter becomes a single-shard hotspot; one broadly-targeted campaign appears in huge posting lists. Fix: shard budget counters by campaign_id so each lives on one shard; for extreme hotness, replicate the counter read and reconcile writes async. For huge posting lists, cap candidacy per impression (sample the list) and rely on the ranker to find the winner among a bounded sample; log the truncation so it is visible.

6. Billing bots. Charging for invalid traffic and draining budgets to fraud. Fix: two-tier fraud filtering - inline heuristics that block the obvious at serve time, heavier async scoring that runs before spend and billing are finalized, with a short hold window so the money number is always post-fraud. Confirmed fraud updates blocklists the inline tier consults.

7. Double-counting / lost events. Retries and at-least-once delivery double-count clicks (overbilling) or drop them (lost revenue). Fix: dedup by event id / auction_id in the pipeline (idempotent counting), and impression-click join so a click without a matching impression is discarded. Kafka gives durable, replayable ingestion so nothing is lost.

8. Config propagation lag. A paused or budget-edited campaign keeps serving on stale ad-server caches, overspending or violating advertiser intent. Fix: push every config change through the config Kafka stream so all ad servers apply it within seconds; broadcast hard “exhausted/paused” signals with priority so they take effect fastest. The Budget Cache TTL and lease expiry bound how long a stale decision can persist.

9. Model/feature-store staleness and the feedback loop. CTR estimates drift; fraud in the loop poisons training. Fix: refresh online features frequently, feed only post-fraud clean events into training, and monitor CTR calibration. The fraud filter sits upstream of both billing and training so neither is corrupted.

10. Single points of failure and graceful degradation. Ad server, budget cache, index build. Fix: ad servers are stateless behind a load balancer (any server can serve any impression from its replicated index); the Budget Cache is replicated Redis with failover; if budget data is briefly unavailable, fail toward not serving a possibly-overspent campaign (or serve conservatively) rather than overspending. If ad selection fails entirely, the feed ships the slot empty or with a house ad - never blocks the feed. Ad serving is revenue optimization layered on the feed; it is never a gate on the feed rendering.

11. Multi-region / global. Fix: deploy the serving plane per region, each with its own replicated index and Budget Cache, so the latency-critical selection never crosses a region boundary. Budgets are global, so the pacing aggregator reconciles spend across regions and splits each campaign’s budget into per-region sub-budgets (allocated by expected traffic), rebalancing as regions spend at different rates. The event firehose replicates asynchronously to a global warehouse for billing and reporting.

Wrap-Up

The trade-offs that define this design:

  • A funnel over a scan. Millions of campaigns are cut to ~1,000 by an in-memory targeting index, to ~50 by a cheap scorer, to ~10 by the heavy model, to 1 by the auction. Every stage is more expensive per item but on fewer items, which is the only way to hold a 50ms budget over 3M requests/sec. Blur the funnel and you are scoring millions per impression and the latency budget is gone.
  • In-memory read-only hot path, everything durable off it. Config writes go to Postgres and are pushed to replicated in-memory indexes; budgets are read from a cached counter, never written on the hot path; ML runs on tens of candidates. The impression path never touches a database.
  • Bounded overspend over global transactions. Budgets are enforced by distributing spendable leases and reconciling centrally, not by a per-impression lock. We accept a tiny, bounded overspend near the cap as the explicit price of never paying transaction latency on the feed path, and we shrink it with smaller leases and a fast exhausted-broadcast.
  • Probabilistic pacing over hard gates. Delivery is smoothed by a feedback controller that turns spend-vs-plan into an admission probability, giving continuous coordination-free delivery instead of a bursty on/off gate.
  • Filter fraud in two tiers and bill late. Obvious bots are blocked inline so they never cost; subtle fraud is caught asynchronously before spend, billing, or model training is touched. The money-and-model path is always downstream of fraud.
  • Fail open on the feed. Every failure in the ad plane degrades to an empty slot or a house ad, never a blocked feed. Ads are a revenue layer on top of the feed, never a gate on it.

One-line summary: an ad server where an in-memory Boolean-targeting index cuts millions of campaigns to a few hundred candidates per impression, a multi-stage ranking cascade scores the survivors and a second-price auction picks the winner inside 50ms, budgets are enforced with distributed leases and paced by a probabilistic feedback controller, a two-tier fraud filter keeps bots out of billing and the training loop, and the whole plane fails open so ad selection never blocks the feed.