Most people picture a bidding platform as “a server that gets an ad request and says yes or no.” The interviewer lets that sit, then stacks the constraints that make it a genuinely hard system. You are not the exchange, you are the bidder - the demand side - connected to a dozen ad exchanges that each fire you a bid request for every impression they auction. The exchange gives you a hard deadline: reply within roughly 100 milliseconds end to end including the network, or your bid is simply discarded and someone else wins. For every one of those requests you have to look up who the user is (their segments, their recent behavior, their frequency caps) in a fraction of a millisecond, evaluate every one of an advertiser’s live targeting and bid rules against that profile, decide whether to bid and how much, and do it while making sure you never spend more than each campaign’s budget - a budget that is being drained simultaneously by bidders in three other data centers responding to the same campaign. A real-time bidding platform is not a yes/no server. It is a globally budget-constrained, sub-100ms decision engine that joins a live user profile to a set of advertiser rules on every single exchange request, at a scale of millions of requests per second - and the bid it returns is just the output.
The whole thing comes down to four decisions: how you answer an exchange’s bid request inside 100ms when the profile lookup alone cannot cost more than a millisecond (the hot path and the latency budget), how you keep a billion-user profile store in memory so that lookup is actually sub-millisecond (the profile plane), how you evaluate thousands of advertiser bid rules against live segments fast enough to fit the budget (rule evaluation), and how you pace and deduplicate budget spend across data centers so a global budget is never overspent even though no single machine sees all the spend (distributed budget pacing). Let me build it properly.
Functional Requirements (FR)
In scope:
- Bid request handling (OpenRTB). Connect to multiple ad exchanges. For each inbound bid request (a user, a device, a placement, a floor price, a deadline), decide whether to bid, pick the winning campaign internally, and return a bid response (price + creative) or a no-bid, inside the exchange’s timeout.
- User-profile lookup. For the user/device in the request, fetch their profile - audience segments, interests, frequency-cap counters, recent impressions/clicks, opt-out status - in sub-millisecond time.
- Targeting and bid-rule evaluation. For each eligible campaign, evaluate its targeting predicate against the profile and request context (geo, device, placement, site/app category, time), and compute a bid price from its bidding strategy (flat CPM, value-based, or model-driven).
- Internal auction. Among the campaigns that pass targeting and have budget, pick the one to bid on behalf of (highest internal value), and submit a single bid to the exchange.
- Budget pacing and spend control. Never exceed a campaign’s total or daily budget. Spread spend smoothly across the flight (pacing), and do it correctly when spend is happening in parallel across data centers.
- Win/loss and spend reconciliation. Process win notifications (and impression/click events) from exchanges, attribute spend to the right campaign, deduplicate it, and feed it back into budget state.
- Frequency capping. Enforce “show this user this campaign at most N times per day” even though impressions happen across exchanges and data centers.
Explicitly out of scope (say it so you own the scope):
- Being the exchange / running the seller-side auction. The exchange that collects bids from many DSPs and runs the first/second-price auction is a different system (that is the SSP/ad-exchange design). I am one bidder talking to it over OpenRTB.
- Creative storage, hosting, and moderation. Creatives live on a CDN and go through policy review elsewhere; I reference an approved creative by ID and return its markup/URL.
- ML model training. I serve a bid/CTR/CVR model at low latency and feed it labeled events, but the training job and model architecture are a separate pipeline.
- The advertiser billing backend. I produce durable, deduplicated spend events that bill the advertiser; the ledger, invoicing, and payments are a separate financial system.
- Attribution/measurement for conversions. I ingest conversion signals for optimization and budget, but the full multi-touch attribution product is adjacent.
The one decision that drives everything: on the hot path you join two things - a live user profile and a set of advertiser rules - under a hard network deadline, and you never block that join on anything durable or remote. Profiles are pre-loaded in memory, rules are compiled and indexed in memory, budgets are read from a local cached counter, and everything slow (spend aggregation, profile updates, model training, cross-DC reconciliation) happens off the request path.
Non-Functional Requirements (NFR)
- Scale: connected exchanges fan out a huge volume. Assume ~3M bid requests/sec at peak across all exchanges hitting our bidders. We bid on only a fraction (say we respond-to-bid on ~10-20%), but we must evaluate all 3M. Hundreds of millions to ~1B user profiles. Tens of thousands of active campaigns.
- Latency: the exchange deadline is ~100ms including round-trip network. Realistically our server-side budget is ~50-80ms p99, and within that the profile lookup must be sub-millisecond and rule evaluation a few milliseconds. Blow the deadline and the bid is thrown away - a slow bid is identical to no bid, but you still burned the compute.
- Availability: 99.99%+ on the bidding path, but with a crucial property: failing is cheap. A dropped bid request just means we do not bid on that impression - we lose a little revenue opportunity, we do not break anything. There is no “the page won’t load” failure mode on our side.
- Consistency: targeting/segments are eventually consistent (a segment 60s stale is fine). Budget is where consistency bites: it must be strong enough to not materially overspend a global budget, but we accept small bounded overspend rather than a global transaction per bid. Spend records that bill the advertiser must be exactly-once.
- Durability: every win/spend event must be durably logged and deduplicated. Losing a win means under-billing (we served but did not charge the advertiser or decrement budget - we overspend later); double-counting means overbilling.
- Freshness: budget-spent counters must be fresh across the global fleet within a second or two, or fast campaigns overshoot. Frequency-cap counters must be fresh enough to not blow the cap badly. Campaign config changes (pause, budget edit) must propagate within seconds.
- Throughput economics: because we evaluate 3M requests/sec and only bid on a fraction, per-request cost must be tiny. Wasting CPU on requests we will no-bid is the silent scaling killer.
Back-of-the-Envelope Estimation (BoE)
Real numbers with the arithmetic, because they dictate the architecture.
Request rate:
~3M bid requests/sec at peak across all connected exchanges.
Average maybe ~1.5M/sec. Design the serving fleet for 3M/sec.
Each request: 1 profile lookup + evaluate up to ~thousands of campaign rules
(cut hard by indexing to tens) + at most 1 outbound bid.
We respond-to-bid on ~15% -> ~450K bids/sec submitted.
Win rate on our bids ~ maybe 30% -> ~135K wins/sec -> ~135K spend events/sec.
User-profile store (the in-memory driver):
1B user/device profiles. Each profile held hot: segments (say 50 segment ids
@ ~4 bytes = 200B), frequency-cap counters (~100B), recent-event summary
(~200B), misc flags (~100B) -> ~600B-1KB per profile.
1e9 * 1KB = ~1 TB of hot profile data.
That does NOT fit on one box. Sharded across memory of many nodes:
1 TB / (say 128 GB usable RAM/node) ≈ ~8-16 nodes per replica, replicated.
Lookup must be sub-ms: a single sharded in-memory GET by user key.
Campaign metadata and rules:
~50K active campaigns * ~10 KB (targeting expr, bid strategy, creatives,
pacing config) = ~500 MB. Tiny. Fully replicated in-memory to every bidder,
compiled into a targeting index. Never read from a DB on the hot path.
Budget / frequency counters (hot mutable state):
50K campaigns, each a live "spent today" + "spent total" counter -> trivial in
size (~50K * 100B ≈ 5 MB) but hot: ~135K spend updates/sec globally,
and read on every candidate evaluation (~tens of millions of reads/sec).
Frequency caps: per (user, campaign) counters -> far larger, but sparse and
co-located with the user profile (it is a field on the profile).
Event firehose (wins / impressions / clicks):
Wins ~135K/sec. Plus impression + click beacons. Each event ~500B.
135K/sec * 500B ≈ 67 MB/sec ≈ ~5.8 TB/day of win/spend events (before
impression/click beacons, which add more). Durable, replayable, feeds
billing + budget + model training. Tiered to cold storage, rolled up hot.
Bandwidth:
Inbound bid requests: 3M/sec * ~2 KB (OpenRTB JSON) = ~6 GB/sec ingress.
Outbound bid responses: ~450K/sec * ~1 KB = ~450 MB/sec egress.
This is why bidders sit close to exchanges (peering / same region) - the
network portion of the 100ms is real and you cannot afford a cross-region hop.
The takeaways: a ~1 TB hot profile store that must be sharded in memory and read sub-millisecond; a tiny (~500 MB) campaign/rule set that must be compiled into an index and replicated to every bidder; a hot but small budget-counter plane that must be kept globally fresh without a per-bid global transaction; and a multi-TB/day event firehose that must be durably captured, deduplicated, and reconciled. The profile lookup, the rule index, and the cross-DC budget plane are the design drivers - not raw storage.
High-Level Design (HLD)
Three planes. A bidding plane that answers exchange requests in <100ms (stateless bidders reading a sharded in-memory profile store + a replicated rule index + a local budget cache). A budget/pacing control plane that aggregates spend globally across data centers and pushes back per-DC budget allocations and pacing signals. An event/reconciliation pipeline that captures wins/impressions/clicks, deduplicates, bills, updates profiles, and feeds the models.
Advertiser ─UI─▶ ┌──────────────────────────────────────────┐
│ Campaign Service (CRUD, budget, targeting)│
│ -> Campaign DB (Postgres, source of truth)│
│ -> publishes config changes (Kafka) │
└───────────────┬──────────────────────────┘
│ config stream
┌───────────────────▼────────────────────┐
│ Config distribution -> compiled Rule │
│ Index + Campaign Cache on EVERY bidder │
└───────────────────┬────────────────────┘
│ (read, in-memory)
Exchange ──OpenRTB bid req──▶ ┌───▼─────────────┐ GET profile:{uid}
(user, device, slot, │ BIDDER │◀──────────────────┐
floor, deadline ~100ms) │ (stateless): │ │
│ 1 profile lookup│ ┌─────────▼─────────┐
◀── bid response ─────────│ 2 rule match │ │ Profile Store │
(price, creative) │ 3 price + pace │ │ (sharded in-memory,│
or no-bid (204) │ 4 internal auc. │ │ ~1TB, replicated) │
└───┬──────────────┘ └─────────▲─────────┘
│ GET spent:{cid}, │ async
│ GET pace:{cid} │ profile
┌─────────▼─────────┐ │ updates
│ Budget Cache │ │
│ (Redis, per-DC, │ │
│ spent + pace) │ │
└─────────▲─────────┘ │
│ per-DC allocation + pace │
┌─────────────────────────────────┴──────────────┐ │
│ Global Budget/Pacing Controller │ │
│ (aggregates spend across ALL data centers, │ │
│ splits budget into per-DC sub-budgets) │ │
└─────────────────────────────────▲──────────────┘ │
│ true spend │
Exchange ─win notice─▶ ┌──────────┴───────────┐ │
(nurl / impression) │ Event Ingestion (Kafka)│ │
│ wins, imps, clicks │ │
└──────────┬───────────┘ │
▼ │
┌────────────────────────────────┐ │
│ Dedup + Attribute (Flink) │──profile update──┘
│ win->campaign, idempotent │
└───────┬──────────┬──────────┬───┘
▼ ▼ ▼
┌───────────┐ ┌──────────┐ ┌───────────┐
│ Billing │ │ Budget │ │ Analytics/ │
│ ledger │ │ Controller│ │ warehouse │
└───────────┘ └──────────┘ └───────────┘
Write path (advertiser side), infrequent:
- Advertiser creates/edits a campaign in the Campaign Service: targeting predicate, bid strategy, total/daily budget, frequency cap, creatives, flight dates. It validates and writes to the Campaign DB (Postgres - relational, transactional, the source of truth).
- Every config change is published to a Kafka config stream. A distribution layer compiles it into the in-memory Rule Index and Campaign Cache replicated on every bidder, within seconds. This is what makes bidding a pure in-memory operation - no bidder ever queries Postgres on the hot path.
Read path (bid request), 3M/sec, must finish well inside 100ms:
- An exchange sends an OpenRTB bid request to the nearest bidder (bidders are deployed in the same regions/peering points as the exchanges to keep the network portion of the deadline small).
- The bidder: (a) extracts the user/device key and does a sub-millisecond profile lookup against the sharded in-memory Profile Store; (b) runs the rule index to find campaigns whose targeting matches this profile+context; (c) for survivors, checks budget (
spent:{cid}from the Budget Cache) and pacing (pace:{cid}), drops exhausted/throttled ones, and computes each one’s bid price; (d) runs the internal auction - picks the highest-value campaign to bid on, applies frequency-cap and floor-price checks, and returns a single bid response (price + creative markup) or a204no-bid. - The bidder records the bid intent (for later win-matching) and moves on. It does not write any durable state on the hot path.
Event path (spend + reconciliation), continuous and asynchronous:
- When we win, the exchange fires a win notice (OpenRTB
nurl/ billing notice) and later impression/click beacons. These land in Kafka. - Dedup + Attribute (Flink) matches each win to the bid/campaign, deduplicates by bid id, and emits authoritative spend. Spend flows to the Global Budget/Pacing Controller, which aggregates true spend across all data centers, splits each campaign budget into per-DC sub-budgets, and pushes fresh
spent/paceinto each DC’s Budget Cache. - Clean events also update the user profile (frequency-cap counters, recent impressions), feed the billing ledger, the model feature store/training pipeline, and the analytics warehouse.
The key insight: the hot path is entirely in-memory and read-only, and it joins two things - a profile and a rule set - under a network deadline. Profiles are pre-sharded in memory, rules are a compiled index, budgets are a cached counter, and everything durable, mutable, slow, or global happens off the request path.
Component Deep Dive
The hard parts, naive-first then evolved: (1) the sub-100ms hot path and why the profile lookup must be sub-millisecond, (2) the in-memory profile store that makes that lookup possible, (3) rule evaluation against live segments, (4) budget pacing deduplicated globally across data centers.
1. The sub-100ms hot path and the latency budget
Naive approach: on each bid request, call a user-profile service over the network, then query a campaigns database for eligible campaigns, score them, and respond. Straightforward service-oriented design: profile service + campaign DB + bidder.
Where it breaks:
- The 100ms is mostly gone before you start. Of the ~100ms exchange deadline, the network round-trip from exchange to bidder and back can eat 20-40ms. That leaves you ~50-70ms of server time. One remote call to a profile service (even 5-10ms) plus one DB query (10-50ms) plus scoring blows it. A bid that arrives after the deadline is discarded - you did all the work and got nothing.
- A DB query per bid is impossible at 3M/sec. No relational database answers 3M eligibility queries per second. And most of those requests you will no-bid on, so you would be spending your most expensive resource on impressions you do not even want.
- Tail latency dominates revenue. At 3M/sec, p99 is what matters: 1% late at 3M/sec is 30K thrown-away bids/sec. A design that is “usually fast” but has a fat tail loses the auctions that matter.
The fix: collapse the hot path to in-memory, local, read-only operations, and treat the latency budget as a hard allocation.
- No remote service calls on the hot path. The profile lookup goes to a sharded in-memory store that the bidder reads directly (section 2), not a request/response microservice with its own queue. The rule set is a compiled in-memory index on the bidder itself (section 3). Budget is a locally-cached counter (section 4). The only remote thing the bidder touches is the profile store, and that is a single sharded GET, sub-millisecond.
- Budget the request explicitly and enforce it with a deadline. Every stage gets a slice; the request carries a deadline and any stage that would overrun returns a no-bid rather than a late bid. A no-bid costs nothing; a late bid costs compute and still loses.
hot-path latency budget (server-side, target p99 well under the ~100ms incl. network):
parse OpenRTB request ~1-2ms
profile lookup (sharded in-mem GET) <1ms user -> segments + freq caps
rule index match ~2-3ms ~50K campaigns -> tens eligible
budget + pacing filter ~1ms drop exhausted/throttled
price + internal auction ~2-3ms 1 campaign + bid price
build creative + serialize response ~2ms
------------------------------------------------------------
server total ~10-15ms p99; the rest of the ~100ms is network + exchange
- Shed load under pressure, do not queue. If a bidder is overloaded, it is better to immediately no-bid some requests than to queue them and answer everything late (and win nothing). A deadline-aware admission check at the front shaves CPU off requests that cannot be answered in time.
- Co-locate with exchanges. Deploy bidder clusters in the same regions / peering locations as each exchange so the network portion is a few milliseconds, not a cross-continent hop. This is why the design is multi-DC from day one, which in turn is what makes budget pacing hard (section 4).
The mental model: the 100ms is a budget you mostly spend on the network, so the server side must be a handful of in-memory operations with a hard deadline, and a request you cannot answer in time becomes a free no-bid, not a late bid.
2. The in-memory user-profile store (sub-millisecond lookup)
Naive approach: store profiles in a database (or a disk-backed KV store) and fetch on each bid. Keep ~1B profiles in Cassandra/DynamoDB/Postgres and GET profile[uid] per request.
Where it breaks:
- Disk latency is fatal. A disk-backed KV store answers in single-digit-to-tens of milliseconds p99. At a sub-millisecond requirement inside a ~50ms server budget, that is an order of magnitude too slow, and the tail (cache miss, compaction, GC) is worse.
- 1B profiles will not fit in one machine’s memory. At ~1KB each that is ~1 TB. You cannot hold it on a single box, so “just put it in a local cache” does not work - a local cache on each bidder would have a terrible hit rate against 1B keys and random user traffic.
- Profiles change constantly. Segments get recomputed, frequency-cap counters increment on every impression, opt-outs arrive. A static snapshot goes stale; a write-through to disk on every impression reintroduces disk on the path.
First evolution: a dedicated sharded in-memory store, keyed by user/device id, sitting next to the bidders. Partition the ~1 TB across many nodes (e.g. consistent-hashing the user key across ~16 nodes per replica, replicated 2-3x for availability). Each bid request does a single sharded GET to the one node that owns that user key - one network hop inside the data center, sub-millisecond, no disk. This is the shape of systems like Aerospike (hybrid memory) or a Redis-cluster / custom in-memory grid used for exactly this.
But two real problems remain:
- Cold/unknown users. A huge fraction of bid requests are for users we have never seen (new device, cleared cookies, privacy-restricted). A lookup miss must be fast and must not block - we bid (or no-bid) on context alone (geo, placement, device type) rather than waiting or doing expensive work. The store returns “miss” in sub-millisecond and the bidder proceeds with a context-only profile.
- Profile updates on the firehose. Frequency counters and recent-event summaries change on every win/impression. We cannot write them synchronously on the bid path. So:
The design: a sharded in-memory primary store, fed asynchronously, read-only on the hot path, with profiles assembled offline.
- Read path: single sharded in-memory GET by user key, sub-millisecond, miss returns instantly. Co-located in the same DC as the bidders.
- Write path (async): segment memberships are recomputed by an offline/near-real-time pipeline (the warehouse + a streaming segment engine) and bulk-loaded into the store. Frequency-cap and recent-event fields are updated by the event pipeline (section 4) consuming the win/impression stream - a few seconds behind, which is fine for caps.
- Replication and sharding key: shard by
hash(user_id or device_id)so each user lives on one shard (and its replicas). This co-locates everything about a user - segments and frequency caps - on one node, so the hot-path read is a single GET, and the frequency-cap increment is a single targeted write on the async path. Replicate 2-3x; on a node loss, replicas serve and the lost node reloads from the durable profile snapshot in object storage. - Memory footprint discipline: profiles are compact binary, not JSON - segment ids as packed integers, counters as fixed-width, recent events as a small ring buffer. The difference between 1KB and 4KB per profile is the difference between ~1 TB and ~4 TB of RAM, i.e. real money.
- Privacy/opt-out as a first-class field. The profile carries consent/opt-out flags; if a user is opted out, the hot path treats them as context-only and never uses behavioral segments. This has to be a sub-millisecond check on the same record, not a separate service call.
Profile Store (sharded in-memory, shard = hash(user_key)):
profile:{user_key} -> {
segments: packed int set # audience/interest memberships
freq_caps: {campaign_id -> count_today} # sparse
recent: ring buffer of recent (campaign, ts) # recency/decay
consent: opt_out flags
updated_at
}
Hot path: GET profile:{user_key} # <1ms, miss -> context-only profile
Async in: segment bulk-load (offline) + freq/recent updates (event stream)
Durable: periodic snapshot to object storage for rebuild on node loss
The principle: the profile store is a sharded in-memory grid read-only on the hot path and fed entirely asynchronously, keyed so one user lives on one shard, with a fast miss for cold users. You never touch disk and never write synchronously on the bid path; the sub-millisecond lookup is what the whole 100ms budget is built around.
3. Rule evaluation against live segments
Naive approach: for each bid request, loop over all campaigns and evaluate each one’s targeting predicate against the profile. For ~50K campaigns, check geo, segments, device, placement, frequency cap, budget.
Where it breaks:
- O(campaigns) per request, at 3M/sec. 50K campaigns * 3M requests/sec = 1.5e11 predicate evaluations/sec. Even at a few nanoseconds each that is impossible, and it blows the few-ms rule-eval slice of the latency budget by orders of magnitude for a single request (50K evaluations with set-membership checks is not free).
- The work is redundant. Most campaigns fail on the first clause (wrong country, wrong placement type). You pay to reject obvious non-matches on every request.
- Predicates are complex. Targeting is a conjunction of disjunctions:
geo IN {IN, LK} AND segment IN {auto_intenders, sports_fans} AND device IN {mobile} AND NOT segment IN {existing_customers}. Evaluating that per campaign per request is expensive even ignoring the count.
First evolution: invert the targeting into posting lists, like the ad-server / search-engine trick. Build, incrementally from the config stream, an index: for each attribute value (geo=IN, segment=auto_intenders, device=mobile, placement=instream), the set of campaign ids that target it. On a request, take the profile+context attribute values, gather the matching posting lists, and combine them. Cost becomes proportional to matching campaigns, not total campaigns.
But targeting is a conjunction of disjunctions with negations, so plain intersection is wrong. A campaign matches if, for every targeting dimension it requires, at least one of the user’s values hits, AND no exclusion value hits.
The design: a compiled Boolean-expression index with conjunct counting and an exclusion pass.
- Index each campaign as (dimension, allowed-value-set) clauses. For a disjunctive clause like
segment IN {auto_intenders, sports_fans}, put the campaign in the posting list of both values. Recordrequired_conjuncts[cid]= number of dimensions the campaign requires. - At request time, gather posting lists for all of the profile’s attribute values (its geo, each segment, device, placement, time bucket). Count, per candidate campaign, how many distinct required dimensions were satisfied. A campaign is a positive match when
satisfied == required(the classic DNF / “indexing Boolean expressions” technique). - Second pass for exclusions. Check each positive candidate against exclusion posting lists (NOT segments, NOT domains); drop any hit. This is cheap because the positive set is already tiny (tens of campaigns).
- Partition the index by a high-cardinality mandatory dimension (usually geo or exchange). A request only touches its geo/exchange slice, keeping posting lists short and the index cache-friendly.
- Segments come from the profile, so the whole thing stays in memory. The profile lookup already returned the user’s segment set; rule matching is pure in-memory set work against the compiled index. No service call, no DB.
Rule index (compiled in-memory per bidder, rebuilt from config stream):
geo=IN -> {c12, c88, c91, ...}
segment=auto_int -> {c88, c204, ...}
device=mobile -> {c88, c91, ...}
placement=instream -> {c88, ...}
required[c88] = 4 # geo + segment + device + placement
exclude[segment=existing_cust] -> {c88, ...}
match(profile{IN, [auto_int, sports], mobile}, ctx{instream}):
gather posting lists for each attribute value
count satisfied required-dimensions per candidate
keep where satisfied == required -> ~tens of campaigns
drop any hitting an exclusion list -> final eligible set
Then compute the bid price per eligible campaign and run the internal auction. For each survivor that still has budget and pacing headroom, compute its bid: flat CPM for simple campaigns, or value-based bid = target_value * pConversion / pCTR from the served model for optimized campaigns. Pick the campaign with the highest internal value to bid on behalf of (we submit one bid per request to the exchange), apply the exchange’s floor price, and respond. Frequency cap is checked here too - if the profile’s freq_caps[cid] for the chosen campaign is at its limit, it is dropped before bidding.
The principle: compile targeting into an inverted Boolean-expression index and evaluate by gathering the user’s posting lists and counting satisfied conjuncts, turning an O(50K) scan into an O(matching) in-memory lookup - the only way rule evaluation fits the few-ms slice at 3M/sec.
4. Budget pacing deduplicated globally across data centers
This is the hard part the problem foregrounds. A campaign’s budget is global, but bidding happens in multiple data centers simultaneously, and no single machine sees all the spend.
Naive approach: keep one global spent counter in a central database; each bidder reads it, and if under budget, bids, and increments it on a win. SELECT spent WHERE cid=?; if spent < budget: bid; on win UPDATE spent += price.
Where it breaks:
- A global synchronous counter across data centers is impossible at this latency. A bidder in Singapore reading/incrementing a counter in Virginia adds a cross-region round-trip (100ms+) to a path that has a ~50ms total budget. Dead on arrival.
- Read-modify-write races everywhere. Thousands of bidders across DCs read
spent=990, all see budget1000, all bid and win, and the campaign overspends by a wide margin. A global lock to prevent it reintroduces the cross-region latency. - Wins are delayed and out of band. You do not know you won at bid time - the exchange tells you later via the win notice. So “increment on win” is inherently asynchronous and arrives seconds after the bid; you cannot gate the bid on a number you do not have yet.
- No pacing. Even if enforcement were correct, a campaign with a big daily budget and an aggressive bid would win everything in the first hour and starve the rest of the day.
Fix part A - split the global budget into per-DC sub-budgets, enforced locally.
- A Global Budget/Pacing Controller consumes the deduplicated win/spend stream from every data center (the only place that sees all spend) and knows true global spend per campaign within a second or two.
- It allocates each campaign’s remaining budget into per-DC sub-budgets, proportional to each DC’s recent win rate / traffic for that campaign. Each DC gets, say, “you may spend $4,000 of campaign C in the next interval.” The controller pushes this allocation into each DC’s local Budget Cache (Redis).
- Within a DC, bidders enforce against the local sub-budget using leases. The DC hands each bidder a small local lease (e.g. “$50 of campaign C”) that it decrements as it wins; the bidder requests a new lease when low. This avoids even an intra-DC global write per bid - the standard lease pattern.
- Reconcile and rebalance continuously. Because wins are delayed, each DC tracks bids outstanding (bids submitted but not yet resolved) and conservatively reserves against them so it does not promise budget it is about to spend. The controller rebalances every interval: a DC spending its sub-budget fast gets a bigger next slice, a slow DC gets a smaller one, and as the global budget nears zero the slices shrink toward pennies.
- Accept bounded overspend, bound it deliberately. Because enforcement is local and wins are delayed, the campaign can slightly overspend at the boundary (the last outstanding leases and in-flight wins across DCs). Bound it by: shrinking sub-budgets and leases as the remaining budget shrinks, reserving against outstanding bids, and broadcasting a hard EXHAUSTED signal globally when the controller sees the budget hit. The trade is explicit: a small, bounded overspend (a fraction of a percent) in exchange for never doing a cross-DC transaction per bid. For RTB that is the right trade; strict no-overspend would cost the latency the exchange deadline forbids.
Fix part B - pacing (spreading spend over the flight).
- Target spend curve per campaign. Uniform is
target_by(t) = daily_budget * t/day; better curves weight by predicted bid-opportunity volume so the campaign spends when its audience is actually being auctioned (traffic is not uniform across the day). - Probabilistic throttle to track the curve. The controller compares actual global spend to the target curve and emits a pacing rate per campaign in [0,1], pushed to every DC’s Budget Cache as
pace:{cid}. On the hot path, when a paced campaign is eligible, the bidder admits it into the internal auction only with probability = pace rate (a coin flip per request). Ahead of curve -> rate drops toward 0, the campaign bids on fewer impressions; behind -> rate rises toward 1 to catch up. It is a feedback controller on spend velocity, recomputed every few seconds. - Why probabilistic, not a hard gate. A hard on/off gate makes delivery bursty and synchronizes all bidders. A coin flip at the current rate gives smooth, continuous, coordination-free delivery - each bidder independently flips at the current rate, and the behavior averages out across the fleet and across DCs.
- Pacing changes participation, not the bid math. A throttled-out campaign simply does not enter this request’s internal auction; the ones that do bid at their true computed price. (Some systems instead pace by lowering the bid - “bid shading for pacing” - which is economically smoother but more complex; probabilistic admission is the simpler, common answer.)
Fix part C - spend deduplication (why the global number is correct).
- Wins arrive as notices that can be duplicated or lost. Exchanges retry win notices; network drops them. So the reconciliation pipeline deduplicates by bid id (each outbound bid carries a unique id; a win notice references it). Counting is idempotent - a duplicate win notice for the same bid id updates spend once.
- Match win to bid to campaign. The bidder logged every bid it submitted (bid id -> campaign, price) to the event stream; the Dedup+Attribute stage joins the incoming win notice to that bid record to attribute spend to the right campaign. A win notice with no matching bid is suspect (dropped/flagged).
- This deduplicated, attributed spend is the only number that feeds the Global Controller, billing, and budget. Everything money- or budget-affecting is downstream of dedup, never the raw win firehose.
Global Budget/Pacing Controller (consumes ALL DCs' deduped spend, every ~2s):
global_spent = sum of deduped spend across all data centers
if global_spent >= daily_budget: broadcast EXHAUSTED to all DCs; rate=0
else:
target = pacing_curve(cid, now)
rate = clamp(rate * (target>global_spent ? 1.1 : 0.9), 0, 1) # catch up / slow down
for dc in data_centers:
sub_budget[dc] = (daily_budget - global_spent - reserved_inflight)
* dc_traffic_share(cid, dc)
push {sub_budget[dc], pace=rate} -> dc Budget Cache
Bidder hot path, per eligible campaign:
if spent:{cid} >= budget: drop
if local_lease(cid) <= 0: drop (async request more within the DC sub-budget)
if random() > pace:{cid}: drop # pacing coin flip
else keep; on WIN (async), decrement local_lease, log bid-id for dedup
The principle: a global budget is split into per-DC sub-budgets reconciled by a controller that is the only thing seeing all spend, enforced locally with leases and a probabilistic pacing coin flip, with spend made correct by idempotent dedup on bid id. No cross-DC transaction ever touches the bid path; the price is a small, deliberately bounded overspend.
API Design & Data Schema
The advertiser side is REST/CRUD; the exchange side is OpenRTB; the event side is streams.
Advertiser-facing (campaign management)
POST /api/v1/campaigns
body: { advertiser_id, name, objective:"conversions",
daily_budget:20000, total_budget:600000,
bid_strategy:{ type:"target_cpa", target:8.00, max_cpm:12.0 },
targeting:{ geo:["IN"], segments:["auto_intenders","sports_fans"],
device:["mobile"], placement:["instream"],
exclude_segments:["existing_customers"] },
frequency_cap:{ per_day:3 },
creatives:["cr_551"], flight:{start:"2026-11-18", end:"2026-12-18"} }
-> 201 { campaign_id:"c_88", status:"pending_review" }
PATCH /api/v1/campaigns/c_88 body:{ daily_budget:30000 } -> 200
POST /api/v1/campaigns/c_88/pause -> 200
GET /api/v1/campaigns/c_88/report?from=..&to=..
-> 200 { bids:4.1e8, wins:3.2e7, win_rate:0.078, spend:18400,
impressions:3.1e7, clicks:52000, conversions:2300, cpa:8.0 }
Exchange-facing (OpenRTB, exchange -> bidder)
POST /rtb/bid # OpenRTB 2.x bid request
body: { id:"req_7f3", tmax:100, # tmax = deadline in ms
imp:[{ id:"1", bidfloor:2.5, instl:0,
video:{...} | banner:{...} }],
user:{ id:"u_1", buyeruid:"...", geo:{...} },
device:{ ua, ip, devicetype, os }, site|app:{...} }
-> 200 { id:"req_7f3",
seatbid:[{ bid:[{ id:"bid_9a", impid:"1", price:4.20,
adid:"cr_551", nurl:"<win-notice-url>",
adm:"<creative markup>" }] }] }
-> 204 No Content # no-bid (no eligible campaign, throttled, or over deadline)
GET <nurl>?price=${AUCTION_PRICE} # exchange fires on win -> our ingestion
Event ingestion (bidder / exchange -> pipeline)
// bid logged by the bidder (for later win-matching + dedup)
{ "type":"BID", "bid_id":"bid_9a", "req_id":"req_7f3", "campaign_id":"c_88",
"user_key":"u_1", "price":4.20, "dc":"ap-south-1", "ts":... }
// win notice (exchange fires nurl)
{ "type":"WIN", "bid_id":"bid_9a", "clearing_price":3.10, "ts":... }
// impression / click beacons (client/exchange)
{ "type":"IMPRESSION", "bid_id":"bid_9a", "ts":... }
{ "type":"CLICK", "bid_id":"bid_9a", "ts":... }
Data stores - the right tool per plane
1. Campaign DB - PostgreSQL (relational, source of truth). Campaigns, budgets, bid strategies, targeting, frequency caps, flights. Relational because these are structured, transactional (budget edits/top-ups must be ACID), moderate in volume (~50K rows), and read by humans for reporting. Written rarely relative to the bid firehose, so relational is perfect. Never on the hot path - compiled and pushed to in-memory indexes.
campaigns(
campaign_id PK, advertiser_id FK, status, objective,
daily_budget numeric, total_budget numeric, spent_total numeric,
bid_strategy jsonb, freq_cap_per_day int,
start_date, end_date, created_at, updated_at )
INDEX (advertiser_id, status), INDEX (status, end_date)
targeting(campaign_id FK, dimension, value, is_exclude bool) -- normalized clauses
INDEX (dimension, value) -- feeds the inverted rule-index build
creatives(creative_id PK, campaign_id FK, cdn_url, format, review_status)
2. Profile Store - sharded in-memory grid (Aerospike / Redis-cluster-style). ~1 TB of ~1B profiles, keyed by user/device, shard = hash(user_key), replicated 2-3x. Read-only sub-millisecond GET on the hot path; fed asynchronously (segments bulk-loaded, frequency/recent fields updated from the event stream). NoSQL/in-memory because access is a single keyed binary record fetch at microsecond latency - no joins, no transactions.
profile:{user_key} -> { segments(packed), freq_caps{cid->n}, recent(ring),
consent_flags, updated_at }
snapshot -> object storage (rebuild on node loss)
3. Campaign Cache + Rule Index - in-memory, on every bidder. The ~500 MB of campaign metadata plus the compiled Boolean-expression targeting index, replicated to every bidder and updated incrementally from the config Kafka stream. In-memory because access is “gather posting lists + fetch campaign records by id” at microsecond latency.
in-memory per bidder:
campaign:{cid} -> { bid_strategy, creatives, freq_cap, status }
index[dim=value] -> posting list of campaign_ids
required[cid] -> int ; exclude[dim=value] -> posting list
4. Budget Cache - Redis, per data center (sharded, in-memory). spent:{cid}, pace:{cid}, and per-DC sub-budget/lease keys, written by the Global Controller + local lease service, read O(1) by bidders (with ~1s local cache). Sharded by campaign_id so a hot campaign’s counter is one shard, not a global hotspot. In-memory and non-durable because it is derived from the durable event stream and rebuildable.
Redis per DC (sharded by campaign_id):
GET spent:{cid} # remaining-budget check
GET pace:{cid} # pacing admission probability
DECR lease:{cid}:{bidder} # atomic local lease decrement
5. Event firehose - Kafka -> Flink -> object storage + columnar warehouse. Bids, wins, impressions, clicks. Kafka for durable, replayable, high-throughput ingestion; Flink for dedup (by bid id), win-to-bid attribution, and profile/budget updates; raw events to object storage (S3/Iceberg), rolled-up aggregates to a columnar warehouse (ClickHouse/BigQuery) for reporting. Append-only, immutable, partitioned by hour. The one plane that must never lose a billable win.
Kafka topics: bids, wins, impressions, clicks
-> Flink: dedup by bid_id, join win<->bid, attribute spend
-> deduped spend -> Global Budget Controller + billing ledger
-> clean events -> profile updates + feature store (training) + warehouse
Why Postgres for config, in-memory grid for profiles, in-memory index for rules, Redis for budgets, Kafka+warehouse for events: the planes have opposite needs. Config is transactional and low-volume - relational. Profiles are a huge keyed dataset needing sub-ms reads - sharded in-memory grid. Rules are a small replicated dataset needing microsecond set-lookups - compiled in-memory index. Budgets are hot, derived, mutable counters needing freshness and sharding - in-memory Redis. Events are an unbounded durable replayable firehose feeding billing/budget/ML - a log plus a warehouse. There is no single “RTB database.”
Bottlenecks & Scaling
Where it breaks first, in order, and the fix for each:
1. The profile lookup latency (breaks first). A disk-backed or remote-service profile fetch cannot meet sub-millisecond inside a ~50ms server budget. Fix: a sharded in-memory profile grid co-located with bidders, single keyed GET, fast miss for cold users, fed entirely asynchronously. This is the lookup the whole 100ms budget is built around.
2. Scanning all campaigns per bid request. O(50K) targeting evaluations * 3M req/sec is impossible and blows the rule-eval slice. Fix: the compiled inverted Boolean-expression rule index in memory on every bidder - gather the profile’s posting lists, match by conjunct count, exclusion pass. 50K campaigns become tens of eligible ones in a couple of milliseconds.
3. Budget enforcement across data centers. A global synchronous counter per bid adds a cross-region round-trip that destroys the deadline; the race overspends. Fix: per-DC sub-budgets reconciled by a Global Controller that sees all spend, enforced locally with leases, plus an in-memory sharded counter read O(1). Accept a small bounded overspend near the cap; shrink sub-budgets/leases and broadcast EXHAUSTED to bound it.
4. Front-loaded, bursty spend. An aggressive campaign exhausts its budget in the first hour. Fix: probabilistic pacing - a feedback controller turns global spend-vs-curve into an admission probability per campaign; bidders flip a coin to admit it into the internal auction. Smooth, coordination-free, cross-DC-friendly delivery.
5. Delayed and duplicated win notices. Wins arrive seconds late and can be duplicated or lost, so spend is wrong and budget drifts. Fix: dedup by bid id (idempotent counting) and win-to-bid attribution in the pipeline; reserve against outstanding bids so a DC does not promise budget it is about to spend. Kafka gives durable, replayable ingestion so no win is lost.
6. Hot campaign / hot key. One broadly-targeted campaign appears in huge posting lists and its budget counter becomes a single-shard hotspot. Fix: shard budget counters by campaign_id so each lives on one shard; for extreme hotness replicate the counter read and reconcile writes async. For huge posting lists, cap candidates per request (sample) and rely on pricing to find the winner among a bounded sample; log truncation so it is visible.
7. Wasted compute on no-bids. We evaluate 3M/sec but bid on ~15%; spending full work on the 85% we no-bid is the silent scaling killer. Fix: order the funnel cheapest-first - cheap context pre-checks (exchange/placement/geo allow-list, deadline admission) before the profile lookup and rule match, so obvious no-bids exit in microseconds. Shed load by no-bidding under overload rather than queuing and answering late.
8. Frequency-cap correctness across exchanges/DCs. The same user is auctioned on multiple exchanges and in multiple DCs, so a per-DC cap count undercounts and blows the global cap. Fix: store the frequency-cap counter on the user profile (which is sharded by user, so all of a user’s caps live on one shard), updated from the global event stream. The cap can be slightly stale (seconds), so accept a small bounded overshoot like the budget - exact global caps would need a per-impression global write.
9. Config propagation lag. A paused or budget-edited campaign keeps bidding on stale bidder caches. Fix: push every config change through the config Kafka stream so all bidders apply it within seconds; broadcast hard EXHAUSTED/paused signals with priority. Budget Cache TTL and lease expiry bound how long a stale decision persists.
10. Single points of failure and graceful degradation. Bidder, profile shard, budget cache, controller. Fix: bidders are stateless behind a load balancer; any bidder answers any request from its replicated index + the shared profile grid. Profile shards are replicated with failover and rebuild from snapshots; on a miss or shard outage the bidder degrades to a context-only bid. Budget Cache is replicated Redis; if budget data is briefly unavailable, fail toward not bidding (conservative) rather than overspending. If the Global Controller is down, DCs keep enforcing their last sub-budget allocation and pace conservatively until it recovers. The whole system fails toward no-bid, which costs only opportunity, never correctness.
11. Multi-region / global as the default, not an afterthought. Fix: deploy bidder + profile grid + Budget Cache per region, co-located with the exchanges that region serves, so the latency-critical path never crosses a region. The Global Budget/Pacing Controller is the only cross-region component, and it is off the hot path, reconciling spend and re-allocating per-DC sub-budgets every couple of seconds. The event firehose replicates asynchronously to a global warehouse for billing and reporting.
Wrap-Up
The trade-offs that define this design:
- An in-memory read-only hot path under a hard network deadline. The 100ms is mostly network, so the server side is a handful of in-memory operations - a sub-millisecond sharded profile GET, a compiled rule-index match, a cached budget read - with an explicit per-stage budget. A request we cannot answer in time becomes a free no-bid, never a late bid.
- The profile store is the keystone. ~1 TB of ~1B profiles held in a sharded in-memory grid, keyed so one user lives on one shard, read-only on the bid path and fed entirely asynchronously, with a fast miss for cold users. Everything about the latency budget is built around that one sub-millisecond lookup.
- Rules are a compiled index, not a scan. 50K campaigns are cut to tens of eligible ones by an inverted Boolean-expression index that gathers the profile’s posting lists and matches by conjunct count - the only way rule evaluation fits the budget at 3M requests/sec.
- Global budget is split, leased, and reconciled - never globally transacted. A controller that is the only thing seeing all spend splits each budget into per-DC sub-budgets, bidders enforce locally with leases and a probabilistic pacing coin flip, and spend is made correct by idempotent dedup on bid id. We accept a small, deliberately bounded overspend as the explicit price of never putting a cross-DC transaction on the bid path.
- Everything fails toward no-bid. A missing profile becomes a context-only bid; an unavailable budget becomes a conservative no-bid; an overloaded bidder sheds load instead of answering late. Failure costs opportunity, never correctness, because we are the bidder, not the page.
One-line summary: a demand-side RTB bidder where each exchange request is answered inside 100ms by joining a sub-millisecond sharded in-memory user profile to a compiled Boolean rule index, pricing the eligible campaigns and submitting one bid, while a global controller splits each budget into per-DC sub-budgets enforced with leases and smoothed by a probabilistic pacing controller, spend is deduplicated on bid id across data centers, and the whole system fails toward a harmless no-bid.
Comments