1. Requirements & Scope (5 min)
Functional Requirements
- Advertisers create campaigns with targeting criteria (demographics, interests, keywords, geography, device type), set budgets (daily/total), and bid on ad placements (CPC, CPM, CPA)
- When a user visits a page, the ad serving system runs a real-time auction among eligible ads, selects the winner(s), and renders the ad within 100ms
- Track ad events (impressions, clicks, conversions) with accurate attribution and provide real-time reporting to advertisers
- Implement budget pacing — spend the daily budget evenly throughout the day rather than exhausting it in the first hour
- Detect and filter fraudulent clicks (bot clicks, click farms, competitor clicking) before charging advertisers
Non-Functional Requirements
- Availability: 99.99% — every failed ad request is lost revenue. At $100M daily revenue, each minute of downtime costs ~$70K.
- Latency: < 100ms end-to-end from ad request to ad response. The ad auction must complete within 50ms to leave time for network and rendering.
- Consistency: Financial data (budgets, billing) must be strongly consistent. Ad serving can tolerate eventual consistency for targeting data (a few minutes of propagation is fine).
- Scale: 10M ad requests/sec globally. 10M active ad campaigns. 1 billion ad impressions/day. 50M click events/day.
- Durability: Every click and impression must be logged durably — this is billing data. Zero data loss for financial events.
2. Estimation (3 min)
Ad Serving Traffic
- 10M ad requests/sec
- Each request: evaluate ~1000 candidate ads → filter to ~50 eligible → score and rank → select top 3-5
- Each ad request: ~2 KB (user context, page context, device info)
- Response: ~5 KB (ad creative URLs, tracking pixels, metadata)
- Bandwidth: 10M × 7 KB = 70 GB/sec — distributed across 50+ edge PoPs
Auction Computation
- 10M auctions/sec
- Each auction evaluates ~50 eligible ads with ML click prediction
- ML inference: ~0.1ms per ad × 50 ads = 5ms per auction (batched inference)
- Total ML compute: 10M × 50 = 500M inferences/sec
- Requires: ~5,000 GPU/TPU instances for ML serving
Storage
- Event logging: 1B impressions/day × 500 bytes = 500 GB/day impressions
- 50M clicks/day × 200 bytes = 10 GB/day clicks
- Total event storage: ~200 TB/year (retained for 2 years)
- Ad campaigns: 10M campaigns × 5 KB = 50 GB — fits in memory
- User profiles (targeting data): 2B users × 2 KB = 4 TB — distributed cache
Revenue Math
- Average CPM (cost per 1000 impressions): $5
- 1B impressions/day × $5/1000 = $5M/day from display
- Average CPC (cost per click): $1
- 50M clicks/day × $1 = $50M/day from search/click ads
- Total: ~$55M/day ≈ $20B/year
3. API Design (3 min)
Advertiser-Facing APIs
// Create a campaign
POST /v1/campaigns
Headers: Authorization: Bearer <advertiser_token>
Body: {
"name": "Summer Sale 2026",
"objective": "clicks", // impressions | clicks | conversions
"daily_budget": 500000, // $5,000.00 in cents
"total_budget": 15000000, // $150,000.00
"bid_strategy": "manual_cpc", // manual_cpc | target_cpa | maximize_clicks
"max_bid": 200, // $2.00 max CPC
"targeting": {
"geo": ["US", "CA"],
"age_range": [25, 54],
"interests": ["technology", "gaming"],
"keywords": ["gaming laptop", "best GPU 2026"],
"devices": ["desktop", "mobile"],
"time_of_day": { "start": 8, "end": 22, "timezone": "America/New_York" }
},
"creatives": [
{ "type": "banner", "size": "300x250", "image_url": "...", "landing_url": "..." },
{ "type": "text", "headline": "50% Off Gaming Laptops", "description": "...", "landing_url": "..." }
],
"start_date": "2026-06-01",
"end_date": "2026-08-31"
}
Response 201: { "campaign_id": "camp_xyz", "status": "pending_review" }
// Get campaign performance
GET /v1/campaigns/{campaign_id}/metrics?date_range=last_7d&granularity=daily
Response 200: {
"campaign_id": "camp_xyz",
"metrics": {
"impressions": 1250000,
"clicks": 37500,
"ctr": 0.03, // 3% click-through rate
"conversions": 1125,
"cpa": 444, // $4.44 cost per acquisition
"spend": 50000000, // $500,000.00
"remaining_budget": 10000000
},
"daily_breakdown": [...]
}
Ad Serving API (internal, called by publisher pages)
// Request ads for a page
GET /v1/ads?slots=3&format=banner_300x250,banner_728x90
&page_url=example.com/tech/reviews
&user_id=anon_abc // hashed, for targeting
&device=mobile&geo=US-CA
Response 200 (< 100ms): {
"ads": [
{
"ad_id": "ad_001",
"creative_url": "https://cdn.ads.com/banner_001.jpg",
"landing_url": "https://advertiser.com/sale?utm_source=...",
"impression_url": "https://track.ads.com/imp?id=ad_001&...",
"click_url": "https://track.ads.com/click?id=ad_001&...",
"bid_price": 150 // $1.50 CPM, for publisher revenue share
}
],
"auction_id": "auc_12345"
}
Event Tracking API
// Impression beacon (fired when ad is rendered)
GET /v1/track/impression?ad_id=ad_001&auction_id=auc_12345&ts=1708632000
Response 204 (no content, fire-and-forget)
// Click redirect (user clicks ad)
GET /v1/track/click?ad_id=ad_001&auction_id=auc_12345
Response 302: Redirect to landing_url
(logs click event before redirect)
// Conversion postback (from advertiser's server)
POST /v1/track/conversion
Body: { "campaign_id": "camp_xyz", "conversion_id": "conv_001", "value": 9999 }
4. Data Model (3 min)
Campaigns (PostgreSQL — sharded by advertiser_id)
Table: campaigns
campaign_id (PK) | varchar(20)
advertiser_id (FK) | varchar(20)
name | varchar(200)
objective | enum('impressions','clicks','conversions')
daily_budget | int -- cents
total_budget | int
spent_total | int -- running total spend
bid_strategy | varchar(30)
max_bid | int
targeting | jsonb
status | enum('draft','pending_review','active','paused','completed','rejected')
start_date | date
end_date | date
created_at | timestamp
Ad Creatives (PostgreSQL)
Table: creatives
creative_id (PK) | varchar(20)
campaign_id (FK) | varchar(20)
type | enum('banner','text','video','native')
size | varchar(20)
content | jsonb -- image_url, headline, description, etc.
landing_url | varchar(500)
status | enum('pending_review','approved','rejected')
quality_score | float -- ML-predicted relevance/quality (0-1)
Ad Index (in-memory, rebuilt every few minutes)
// Inverted index for fast candidate selection during auction
// Loaded into each ad server's memory
keyword_index: Map<keyword, List<campaign_id>>
geo_index: Map<geo_code, List<campaign_id>>
interest_index: Map<interest, List<campaign_id>>
device_index: Map<device_type, List<campaign_id>>
// Each entry: campaign_id + max_bid + remaining_budget + quality_score
// Total size: ~10M campaigns × 100 bytes = 1 GB per ad server (fits in memory)
Event Log (Kafka + ClickHouse)
// Kafka topics: ad_impressions, ad_clicks, ad_conversions
// Retained 7 days in Kafka
// ClickHouse (analytical queries, 2-year retention)
Table: events
event_type | Enum('impression','click','conversion')
event_id | String
ad_id | String
campaign_id | String
advertiser_id | String
user_id | String (hashed)
auction_id | String
timestamp | DateTime
bid_price | UInt32
geo | String
device | String
page_url | String
Budget Tracking (Redis)
Key: budget:{campaign_id}:daily:{date}
Type: Hash
Fields:
spent | int (cents, incremented on each billable event)
limit | int (daily budget)
pacing | float (target spend rate per hour)
Key: budget:{campaign_id}:total
Type: String (remaining total budget in cents)
5. High-Level Design (12 min)
Ad Serving Pipeline (< 100ms total)
User visits a web page
→ Publisher's ad tag (JavaScript) fires ad request
→ CDN/Edge → Ad Server (closest PoP):
Step 1: Parse Request (< 1ms)
Extract: user_id, page context, device, geo, ad slot sizes
Step 2: User Profile Lookup (< 5ms)
Redis/Memcached: get user interests, demographics, behavior segments
If cache miss → use contextual signals only (page content, keywords)
Step 3: Candidate Selection (< 5ms)
Query in-memory ad index:
geo_index[US-CA] ∩ device_index[mobile] ∩ interest_index[gaming]
→ ~1000 candidate campaigns
Filter: active status, within date range, has remaining budget, creative approved
→ ~200 eligible campaigns
Filter: frequency capping (user hasn't seen this ad > 3 times today)
→ ~150 final candidates
Step 4: Bid Calculation + Click Prediction (< 20ms)
For each candidate (batched ML inference):
pCTR = click_prediction_model(user_features, ad_features, context_features)
eCPM = bid × pCTR × 1000 // expected revenue per 1000 impressions
Sort by eCPM descending
Step 5: Auction (< 2ms)
Run generalized second-price auction:
Winner pays the minimum bid needed to beat the second-place ad
price_to_charge = second_place_eCPM / winner_pCTR
Select top 3-5 ads for the available slots
Step 6: Budget Check (< 2ms)
Redis: INCRBY budget:{campaign_id}:daily:{date} {charge_amount}
If new_total > daily_budget → skip this ad, select next candidate
If total_budget exhausted → skip
Step 7: Response (< 1ms)
Return ad creatives, tracking URLs, metadata
Total: ~35ms server-side, ~100ms with network
→ Browser renders ads, fires impression beacons
→ On click: redirect through click tracker → landing page
Event Processing Pipeline
Impression/Click Events:
→ Event Collector (edge servers, receive tracking pixels/redirects)
→ Kafka (durable event stream)
→ Stream Processor (Flink):
1. Fraud Detection: filter invalid clicks (see Deep Dive 3)
2. Deduplication: same impression/click ID within 5-min window
3. Attribution: match clicks to impressions, conversions to clicks
4. Budget Update: INCRBY in Redis for real-time budget tracking
5. Write to ClickHouse for reporting
→ ClickHouse (analytical queries)
→ Billing Service (aggregate billable events → generate invoices)
Components
- Ad Servers (50+ edge PoPs, 100s of instances): Handle ad requests. Run the full auction pipeline in-process. Stateless except for cached ad index and user profiles.
- Ad Index Builder: Reads campaigns from PostgreSQL, builds in-memory inverted indexes, pushes to ad servers every 2-5 minutes. Ensures ad servers have fresh campaign data.
- Click Prediction Service (ML): Serves the pCTR model. Deployed on GPU instances. Batched inference for throughput. Model retrained daily on click/no-click data.
- User Profile Service (Redis/Memcached): Stores user interest segments, demographics, behavioral signals. Updated by a profile pipeline that processes user activity.
- Event Collectors: Edge servers that receive impression beacons and click redirects. Ultra-low latency (< 10ms). Write to Kafka.
- Stream Processor (Flink): Real-time event processing: fraud detection, deduplication, attribution, budget updates.
- Budget Pacing Service: Calculates target spend rate per campaign per hour. Adjusts bid multipliers to pace spending evenly.
- Fraud Detection Engine: ML + rules-based click fraud detection. Filters invalid traffic before billing.
- Reporting Service: Reads from ClickHouse. Serves advertiser dashboards with near-real-time metrics (< 5 min delay).
- Campaign Management Service: CRUD for campaigns, creatives, targeting. Creative review (automated + manual).
6. Deep Dives (15 min)
Deep Dive 1: The Ad Auction Mechanism
Why not a simple highest-bidder-wins auction? If we only ranked ads by bid price, advertisers with deep pockets would always win, regardless of ad quality. Users would see irrelevant, low-quality ads. Click rates would drop. Publishers would earn less long-term.
Google’s innovation: eCPM ranking (quality × bid)
For each candidate ad:
pCTR = predicted probability that this user clicks this ad
bid = advertiser's maximum CPC bid
eCPM = pCTR × bid × 1000
// eCPM = expected revenue per 1000 impressions
Example:
Ad A: bid = $2.00, pCTR = 1% → eCPM = $20.00
Ad B: bid = $1.00, pCTR = 3% → eCPM = $30.00 ← WINS (despite lower bid)
Ad B wins because it's more relevant to the user (higher pCTR).
The user is happier (relevant ad), the publisher earns more ($30 vs $20),
and the advertiser pays less (lower bid but better targeting).
Generalized Second-Price (GSP) auction:
Ads ranked by eCPM. Winner pays the minimum needed to beat 2nd place:
Rank 1: Ad B (eCPM = $30.00, pCTR = 3%, bid = $1.00)
Rank 2: Ad A (eCPM = $20.00, pCTR = 1%, bid = $2.00)
Rank 3: Ad C (eCPM = $15.00, pCTR = 2%, bid = $0.75)
Ad B's charge: (Ad A's eCPM / Ad B's pCTR) + $0.01
= ($20.00 / 0.03) / 1000 + $0.01
= $0.667 + $0.01 = $0.68 per click
(instead of their $1.00 bid — they save $0.32)
Why second-price?
- Encourages truthful bidding. Advertisers bid their true value
because they'll pay less than their bid (next-highest price).
- In a first-price auction, advertisers shade their bids downward
(bid less than true value), making the auction less efficient.
VCG (Vickrey-Clarke-Groves) auction — the theoretically optimal mechanism:
Each winner pays the externality they impose on others:
charge_i = (total value of auction without i) - (total value paid by others with i present)
VCG guarantees:
- Truthful bidding is the dominant strategy
- Maximizes social welfare (sum of all participants' utility)
Trade-off: VCG is more complex and can result in lower revenue for the platform
than GSP. In practice, Google uses a modified GSP that approximates VCG properties
while maintaining higher revenue.
Deep Dive 2: Click Prediction (ML Model)
Why click prediction is the most important ML model in ads:
- Directly determines ad ranking (eCPM = pCTR × bid)
- A 1% improvement in prediction accuracy → millions of dollars in additional revenue
- Must run on every auction (500M inferences/sec) with < 5ms latency
Model architecture:
Input features (three categories):
1. User features (~50 dimensions):
- Demographics: age bucket, gender, income bracket
- Interests: 100+ interest categories, each a probability
- Behavioral: pages visited recently, ads clicked recently, purchase history
- Context: time of day, day of week, device, browser, OS
2. Ad features (~30 dimensions):
- Ad category, quality score, historical CTR
- Creative type (banner, text, video)
- Landing page quality score
- Advertiser trust score
3. Cross features (~20 dimensions):
- User interest × ad category match score
- Historical CTR of this user on this advertiser's ads
- Geographic relevance (user location vs ad targeting)
Total: ~100 features per (user, ad) pair
Model: Wide & Deep neural network
- Wide component: memorizes specific feature combinations
(user_123 + ad_category_gaming → high CTR from historical data)
- Deep component: generalizes across feature interactions
(users who like gaming AND are 25-34 AND on mobile → higher CTR for gaming ads)
Training:
- Data: billions of (impression, clicked/not_clicked) pairs
- Retrained daily with last 7 days of data
- Online calibration: adjust predictions every hour based on
observed vs predicted CTR (prevent model drift)
Serving:
- Model quantized to INT8 for fast inference
- Batched: process 50 ads per auction in a single GPU call
- Latency: 2-5ms for a batch of 50 predictions
- 500M inferences/sec → ~5,000 GPU instances (T4 or similar)
Calibration is critical:
The model outputs a probability (e.g., 0.03 = 3% chance of click).
This probability MUST be well-calibrated because it directly determines pricing.
If the model says 3% but the true rate is 2%:
- eCPM is overestimated by 50%
- Advertiser gets charged for predicted clicks that don't happen
- Short-term: platform earns more
- Long-term: advertisers leave because ROI doesn't match predictions
Calibration technique (isotonic regression):
1. After model training, sort all predictions into bins (0-1%, 1-2%, ..., 99-100%)
2. For each bin, compute the actual CTR from held-out data
3. Build a monotonic mapping: predicted → actual
4. Apply this mapping to all production predictions
Monitor: plot predicted CTR vs actual CTR daily. Alert if divergence > 5%.
Deep Dive 3: Budget Pacing and Fraud Detection
Budget Pacing:
Problem: A campaign has a $5,000 daily budget. Without pacing, it might spend
$5,000 in the first 2 hours (morning traffic) and have no ads for the rest of the day.
This is bad because:
- Evening traffic might have higher conversion rates
- The advertiser misses potential customers in the afternoon
- Traffic distribution is uneven throughout the day
Solution: Spend-rate controller
target_spend_rate = daily_budget / remaining_hours
Every hour, Budget Pacing Service calculates:
ideal_spend = daily_budget × (elapsed_hours / 24)
actual_spend = sum of charges so far today
spend_ratio = actual_spend / ideal_spend
if spend_ratio > 1.1 (overspending):
→ Reduce bid multiplier: effective_bid = max_bid × 0.7
→ Participate in fewer auctions (probabilistic throttling)
if spend_ratio < 0.9 (underspending):
→ Increase bid multiplier: effective_bid = max_bid × 1.3
→ Participate in more auctions
Stored in Redis:
HSET pacing:{campaign_id} bid_multiplier 0.7 participation_rate 0.8
Ad server reads pacing parameters during auction:
effective_bid = advertiser_bid × bid_multiplier
if random() > participation_rate → skip this campaign for this auction
Fraud Detection Pipeline:
Click fraud costs advertisers $100B+/year globally. The platform must filter
invalid clicks before billing.
Layer 1: Rule-Based Filters (real-time, in Flink)
- Duplicate clicks: same user + same ad within 30 seconds → deduplicate
- Click flooding: > 10 clicks from same IP in 1 minute → mark as invalid
- No impression: click event without a preceding impression → invalid
- Bot signatures: known bot User-Agents, headless browser detection
- Click timing: click < 100ms after impression render → physically impossible
(human reaction time minimum ~200ms)
Layer 2: ML Anomaly Detection (near-real-time, batch every 5 minutes)
Features per click:
- Time since impression (too fast = bot)
- Mouse movement entropy (no movement = bot)
- Session depth (clicks without any other page activity = suspicious)
- IP reputation score
- Device fingerprint uniqueness
- Click-through to conversion rate for this source
Model: isolation forest for anomaly detection
- Trained on known-good traffic (verified human clicks)
- Clicks that deviate significantly from normal patterns → flagged
Layer 3: Aggregated Pattern Analysis (batch, hourly)
- Click farms: cluster of IPs clicking the same ads repeatedly
Graph analysis: build IP → ad click graph, detect dense subgraphs
- Competitor clicking: detect if a specific advertiser's competitor
is systematically clicking their ads (identify by cookie/device fingerprint)
- Geographic anomaly: campaign targeting US, but 40% of clicks from
countries known for click farms → flag
Outcome:
- Invalid clicks: not billed to advertiser, not counted in reports
- Suspicious clicks: held for review, billed only if cleared
- Advertisers can dispute clicks via a self-service review tool
- Monthly invalid traffic report provided to advertisers
Target: filter > 95% of fraudulent clicks while maintaining < 0.1% false positives
(legitimate clicks incorrectly filtered).
7. Extensions (2 min)
- Real-Time Bidding (RTB) / Ad Exchange: Instead of running the auction internally, open it to external demand-side platforms (DSPs) via the OpenRTB protocol. On each ad request, send bid requests to 10-20 DSPs simultaneously (< 100ms timeout). DSPs respond with bids. Select the highest bidder. This is how programmatic advertising works — Google Ad Exchange processes 100B+ bid requests/day.
- Retargeting / Remarketing: Track users who visited an advertiser’s site but didn’t convert. When those users visit publisher sites, show them the advertiser’s ads (reminding them to complete the purchase). Requires cross-site tracking (third-party cookies or privacy-preserving alternatives like Topics API / FLEDGE).
- Video ad serving: Pre-roll, mid-roll, post-roll video ads. VAST (Video Ad Serving Template) protocol for standardized ad delivery. Viewability tracking (was the video ad actually watched or auto-played muted?). Completion rate metrics. Video ads have 10-50x higher CPM than display ads.
- Privacy-preserving targeting: With cookie deprecation, implement privacy-safe targeting: on-device interest profiling (Google Topics API), differential privacy for aggregate reporting, secure multi-party computation for conversion attribution without exposing individual user data. Federated learning for click prediction models that train on-device.
- Dynamic creative optimization (DCO): Automatically generate ad variations (different headlines, images, CTAs) and use multi-armed bandit algorithms to identify the best-performing creative for each audience segment. Reduces advertiser workload while improving CTR through continuous optimization.