A CDN sounds like a caching proxy you already know how to build: put an Nginx in front of the origin, set a TTL, done. That intuition survives about one slide. A real CDN is a globally distributed system whose entire job is to put a copy of someone else’s bytes physically close to every human on earth, keep those copies fresh (or provably safe to serve stale), route each request to the right box in the right city without the client knowing any of the plumbing exists, and do all of this while sustaining billions of requests per second and ~100 Tbps of egress without ever letting the origins behind it feel the load. The interesting part is not “cache a file.” It is: thousands of points of presence spread across the planet, each independently caching, each pulling from a shared origin through a tiered hierarchy that collapses cache-fill traffic, all fronted by a routing layer that steers a viewer in Lagos to a box in Lagos, plus an invalidation system that can purge a stale object from every edge on earth in seconds. Let me build it properly.
Everything hard here - request routing, the cache hierarchy, invalidation, origin protection - exists to move two numbers: edge cache hit ratio (what fraction of bytes are served without touching origin) and edge RTT (how physically close the serving box is to the user). Get those right and you have a fast, cheap CDN; get them wrong and you have an expensive, slow reverse proxy that melts its origins. A CDN is not “a cache.” It is a distributed system whose product is the cache hit, measured in nines of hit ratio and milliseconds of latency.
Functional Requirements (FR)
In scope:
- Serve cached content from the edge. A client requests an asset (image, video segment, JS/CSS bundle, API response) by URL; the CDN serves it from an edge location close to the client if cached, otherwise fetches from origin, caches it, and serves it.
- Origin pull (cache-fill). On a cache miss, the edge pulls the object from the customer’s origin server, stores it per its cacheability rules, and serves it. Subsequent requests hit the cache.
- Cache control per object. Honor
Cache-Control,Expires,ETag, and per-customer override rules (TTLs, cache-key composition, whether to cache at all). Support conditional revalidation (If-None-Match/304). - Cache invalidation / purge. A customer can invalidate an object (or a path prefix, or a tag) so every edge on earth stops serving the stale copy - within seconds.
- Request routing. Steer each client to a near, healthy edge (by geography, network, and load), transparently, with automatic failover when an edge or region is down.
- Static and dynamic acceleration. Cache static assets fully; for uncacheable dynamic requests, still route them over the CDN’s optimized backbone to the origin (TCP/TLS termination at the edge, connection reuse to origin) so even a cache miss is faster than going direct.
- TLS termination at the edge. Terminate HTTPS close to the user (customer certs, SNI) so the handshake RTT is short.
- Basic analytics and logging. Per-customer hit ratio, bandwidth, status codes, top URLs - fed from edge logs.
Explicitly out of scope (say it so you own the scope):
- The origin application itself. The CDN sits in front of customer origins; I do not build their app servers or databases. I define the pull contract and how I protect them.
- A DDoS / WAF security product. Real CDNs bundle these and the edge is a natural chokepoint for them, but the security layer is a system of its own; I note the hook point and move on.
- Video transcoding / packaging. A CDN delivers HLS/DASH segments; producing them is the origin’s job (see any video-platform design). I treat segments as opaque cacheable objects.
- Edge compute / functions. Running customer code at the edge (image resize, A/B logic) is a large adjacent product; I flag where it hooks into the request path and leave it.
- The physical network / peering. Buying transit, negotiating peering, laying the backbone - real and expensive, but infrastructure, not the software architecture I am designing.
The one decision that drives everything: this is a read-astronomically-heavy, geographically distributed, cache-hit-ratio-and-latency-critical system where the same immutable-ish bytes are served billions of times from machines close to users, and the origins behind it must be shielded so hard that they see a tiny, roughly constant trickle of traffic no matter how viral the content gets. The whole design is about maximizing edge hit ratio and minimizing edge distance while keeping origins safe and caches coherent.
Non-Functional Requirements (NFR)
- Scale: Global. On the order of billions of requests/sec aggregate at peak across all PoPs, ~100 Tbps of total egress. Thousands of PoPs / hundreds of metros. Hundreds of thousands of edge servers. Petabytes of hot cacheable content; the working set per PoP is terabytes.
- Latency: Edge serving of a cache hit in single-digit milliseconds at the server plus the client-to-edge RTT (target keeping p95 client-to-edge under ~30-50ms by putting a PoP near every population center). TLS handshake terminated at the edge. A cache miss adds the edge-to-origin RTT (mitigated by the backbone + shielding).
- Availability: 99.99%+ on the read/serve path - if the CDN is down, every site behind it is down, so this is load-bearing for a large fraction of the internet. Any single edge, PoP, or region failing must transparently fail over. The control plane (config, purge) can tolerate brief degradation; the data plane (serving) cannot.
- Consistency: Eventual on cache content by design - two edges may briefly serve different versions of an object, and a freshly-pulled object may lag origin by a TTL. The one guarantee that must be tight is purge propagation: after a customer purges, no edge should keep serving the old bytes beyond a few seconds. Config changes (routing, cache rules) propagate globally in seconds-to-minutes and are eventually consistent.
- Durability: The cache is not a system of record - it is disposable. Losing a cached object just causes a re-pull from origin. The origin owns durability. What must be durable is the control-plane state (customer configs, cert material, routing maps, purge log).
- The real budget is edge hit ratio and origin egress. A 1% drop in hit ratio at 100 Tbps is ~1 Tbps of extra origin-bound traffic - it blows up customer origins and the CDN’s own mid-tier bandwidth bill. Hit ratio is a first-order design constraint, not a metric to admire afterward.
Back-of-the-Envelope Estimation (BoE)
Real numbers with the arithmetic, because they dictate the topology.
Request and egress volume:
Target: ~100 Tbps total egress at peak, "billions of req/sec."
Assume avg served object ~ 100 KB (mix of small API/JSON and large media segments).
100 Tbps = 100e12 bits/sec = 12.5e12 bytes/sec = 12.5 TB/sec of egress.
12.5e12 bytes/sec / 100,000 bytes/obj = 1.25e8 = ~125 million objects/sec.
That looks low vs "billions"; the billions count includes tiny requests
(favicons, beacons, 304s, API hits ~1-5 KB). Recompute at ~5 KB avg small-req:
a large share of REQUESTS are small; a large share of BYTES are big media.
Model two classes:
- small reqs (~3 KB): the bulk of the COUNT -> ~2-3 billion req/sec
- large media segments (~500 KB-2 MB): the bulk of the BYTES -> the 100 Tbps
So: ~2-3 B requests/sec drives the request-path design (routing, connection handling);
~100 Tbps drives the bandwidth/topology design. Both are the point.
Per-PoP sizing:
Say ~2,000 PoPs sharing 100 Tbps unevenly (big metros carry more).
Average PoP: 100 Tbps / 2,000 = ~50 Gbps. Large metro PoPs: multiple Tbps.
Requests: ~2.5e9 / 2,000 = ~1.25 M req/sec average per PoP, 10x+ that in big metros.
Per edge server: a tuned box does ~100 Gbps NIC and ~1-2 M req/sec of cache hits.
A big-metro PoP at ~2 Tbps needs ~20-40 edge servers just for line rate,
plus headroom => hundreds of servers in the largest PoPs, a handful in small ones.
Cache sizing (the hit-ratio lever):
Cacheable working set is governed by a power law: a small fraction of URLs
are a huge fraction of requests. Suppose the "hot set" that yields ~90% of hits
is ~a few TB per PoP; the long tail is effectively infinite.
Give each edge box ~10-30 TB of NVMe + ~256-512 GB RAM.
A PoP with 40 boxes * 20 TB = ~800 TB of cache -> holds a deep hot set.
Hit ratio target: ~90% at a single edge, pushed to ~99% for BYTES via the
mid-tier (regional shield) that catches the misses. See the hierarchy below.
Origin egress (what shielding must protect):
If edge hit ratio for bytes is 90%, naive origin egress = 10% of 100 Tbps = 10 Tbps.
Spread across thousands of origins that is still catastrophic for many of them,
AND the same missed object gets pulled independently by every one of 2,000 PoPs.
2,000 independent pulls of one newly-popular object = a self-inflicted DDoS on origin.
The mid-tier shield collapses that: N edges in a region share ONE regional cache,
so a region pulls a cold object from origin ONCE, not once-per-edge.
Target effective origin egress AFTER shielding: << 1% of total => sub-Tbps,
and each origin sees a small, roughly CONSTANT trickle regardless of virality.
Purge scale:
Customers purge constantly (deploys, content updates). Say ~50,000 purge ops/sec
globally at peak, each fanning out to ALL ~2,000 PoPs * hundreds of servers.
A naive "delete from every server" is 50,000 * (thousands of servers) messages/sec
= billions of invalidation messages/sec. Must be a fan-out tree + lazy versioning,
never a synchronous broadcast to every box.
The takeaways: the request path is a 2-3 billion req/sec connection-and-cache problem that must be handled at the edge with almost no compute per hit; the bandwidth is 100 Tbps that only a globally distributed edge fleet can source; hit ratio is the master lever and demands a mid-tier shield so origins see a constant trickle, not a per-PoP stampede; and purge is a fan-out problem that cannot be a synchronous global broadcast. Routing, the cache hierarchy, invalidation, and origin protection are the interview.
High-Level Design (HLD)
The architecture splits into a data plane (the request-serving path: routing -> edge -> mid-tier shield -> origin, replicated in thousands of PoPs, carrying the billions-of-req/sec firehose) and a control plane (config distribution, purge fan-out, cert management, health and analytics aggregation - low-QPS, globally consistent-ish, drives the data plane but is never on the hot serving path). A client resolves the CDN hostname, gets routed to a near edge, and the edge serves from cache or fills through the hierarchy. Purges and config flow from a central control plane out to every edge.
┌──────────────────────── CONTROL PLANE ───────────────────────┐
│ Config store (customer rules, cache keys, TLS certs) │
│ Purge service (accept -> version bump -> fan-out tree) │
│ Routing map builder (geo/latency/health -> DNS + anycast) │
│ Log/analytics aggregation (edge logs -> rollups) │
└───────▲───────────────┬──────────────────────┬────────────────┘
│ edge logs │ push config/purge │ health
───────────────────────────────┼───────────────┼───────────────────────┼───────────────
DATA PLANE (per PoP, x thousands) ▼ │
┌─────────┐ 1. resolve+route ┌───────────────────────────────────────┴───────────┐
│ Client │────────────────────>│ ROUTING LAYER │
│(browser │ (Anycast IP / DNS │ - Anycast BGP + GeoDNS + L4/L7 load balancers │
│ / app) │ steering) │ - picks a NEAR, HEALTHY PoP; LB picks an edge box │
└────┬────┘ └───────────────────────┬───────────────────────────┘
│ 2. GET https://asset (TLS terminated at edge) ▼
│ ┌───────────────────────────────────────────────────┐
│<──── cache HIT (99%) ────│ EDGE SERVER (in the PoP nearest the client) │
│ single-digit ms │ - TLS term, request normalize, build cache key │
│ │ - RAM (hot) + NVMe (warm) tiered cache │
│ │ - on HIT: serve. on MISS: ask the shield ─┐ │
│ └────────────────────────────────────────────┼──────┘
│ │ miss
│ ┌────────────────────────────────▼──────┐
│ │ REGIONAL SHIELD / MID-TIER CACHE │
│ │ (few per region; all edges in the │
│ │ region collapse their misses here) │
│ └───────────────┬────────────────────────┘
│ │ miss (rare) -> ONE pull
│ ▼
│ ┌────────────────────────────────────────┐
│ │ ORIGIN (customer's server / storage) │
│ │ reached over the CDN's optimized │
│ │ backbone; sees a small, steady trickle │
│ └─────────────────────────────────────────┘
Request flow (client fetches an asset):
- The client resolves the asset’s hostname. Routing steers it to a near, healthy PoP - via Anycast (the same IP announced from every PoP; BGP routes the packet to the topologically nearest one) and/or GeoDNS (the DNS resolver hands back the IP of the closest PoP based on the resolver’s location and CDN load/health maps). Inside the PoP, an L4/L7 load balancer picks a specific edge server.
- The edge terminates TLS (short handshake because it is physically close), normalizes the request, and computes a cache key (usually host + path + a subset of query/headers per the customer’s rules).
- The edge looks up the key: RAM first (hottest objects), then local NVMe (warm). On a hit with a still-fresh object, it serves immediately - single-digit ms. This is ~90-99% of requests.
- On a miss (or a stale object needing revalidation), the edge does not hit origin directly. It asks its regional shield (mid-tier cache). If the shield has it, the edge fills from the shield (a fast intra-region hop) and serves. All edges in the region share this shield, so the region pulls a given object from origin only once.
- On a shield miss, the shield pulls from origin over the CDN’s optimized backbone (warm TCP/TLS connections, reused pools), caches it at the shield, hands it to the edge (which also caches it), and serves the client. Conditional revalidation (
If-None-Match) turns most refreshes into cheap304s. - Uncacheable dynamic requests still ride this path: the edge terminates TLS and proxies to origin over the backbone with connection reuse and route optimization, so even a non-cached request beats the client going direct.
Control flow (config and purge):
- A customer changes a cache rule or purges an object. The request hits the control plane, which validates it, records it durably, and fans it out through a tree (central -> regional relays -> PoPs -> edge servers), not a flat broadcast.
- Edges apply config and invalidate objects (via a version bump / lazy check - detailed below) within seconds. Health and load signals flow back up so the routing map can steer away from unhealthy PoPs. Edge logs stream up asynchronously for analytics.
The key insight: the data plane is a stateless-ish, cache-heavy serving path replicated thousands of times and driven by a small central control plane; the serving path never blocks on the control plane, and origins are shielded by a tier that collapses misses so they see a trickle, not the firehose.
Component Deep Dive
The hard parts, naive-first then evolved: (1) request routing - how a client reaches a near, healthy edge; (2) the cache hierarchy and eviction - how an edge maximizes hit ratio without infinite disk; (3) origin protection and cache-fill - how thousands of PoPs pull from one origin without stampeding it; (4) cache invalidation - how you purge one object from every edge on earth in seconds.
1. Request routing: why “one DNS record” fails, then Anycast + GeoDNS
Naive approach: run one data center, point the customer’s DNS CNAME at it, and let everyone connect to that one IP.
Where it breaks:
- Distance is latency you cannot cache away. A user in Sydney hitting a box in Virginia eats ~200ms+ RTT on every connection and TLS handshake. The entire premise of a CDN - be close to the user - is violated. One location is not a CDN, it is a reverse proxy.
- No failover. If that IP/DC is down, every site behind the CDN is down. A single DNS record with a long TTL means clients keep hammering a dead address for the TTL duration.
- Capacity. One location cannot source 100 Tbps or 2 billion req/sec. Physics and NICs say no.
Evolution A: GeoDNS - many PoPs, resolve to the nearest. Announce a PoP in every major metro. When a client resolves the hostname, an authoritative DNS that knows the resolver’s approximate location (and the CDN’s live health/load map) returns the IP of a near, healthy, non-overloaded PoP.
Client in Lagos -> resolver -> CDN authoritative DNS
DNS knows: resolver ~Lagos, Lagos PoP healthy & under capacity
returns: Lagos PoP VIP. Short DNS TTL (e.g. 20-60s) so it can re-steer fast.
If Lagos PoP is overloaded/down -> DNS returns the next-nearest healthy PoP (Accra/Jo'burg).
Why short TTLs: routing must react to failures and load in seconds, so DNS answers are short-lived - the trade-off is more DNS QPS, which is cheap and itself served from an anycast DNS fleet.
Where GeoDNS alone still breaks: DNS only knows the resolver’s location, not the client’s (a client using a distant public resolver gets mis-steered), failover is bounded by the DNS TTL (clients cache the old IP), and it balances at metro granularity, not per-packet.
Evolution B: Anycast for the fast, self-healing layer. Announce the same IP address from every PoP via BGP. The internet’s own routing delivers each client’s packets to the topologically nearest PoP announcing that IP. No DNS lookup of location needed; the network does it.
Anycast IP 203.0.113.10 announced from ALL PoPs via BGP.
Client packet -> internet routes to the nearest announcement -> nearest PoP.
PoP dies -> its BGP announcement withdraws -> traffic reroutes to next-nearest
AUTOMATICALLY, in ~seconds, no DNS TTL wait. Self-healing.
The production answer combines both: Anycast for the entry IPs (instant, packet-level, self-healing failover and near-optimal routing) with GeoDNS / load-aware steering layered on for cases where you need finer control (steer away from an overloaded PoP, honor customer geo rules, split by network). Inside the chosen PoP, L4 load balancers (ECMP / Maglev-style consistent hashing) spread connections across edge servers, and consistent hashing on the cache key means a given object tends to land on the same edge box (so it is cached once per PoP, not once per box).
- Health-aware: PoPs continuously report health/load; unhealthy ones withdraw anycast announcements and get pulled from DNS answers, so traffic drains automatically.
- Anycast caveat handled: long-lived TCP connections could theoretically flap between PoPs if BGP re-converges mid-connection; in practice routes are stable enough and connection-level stickiness plus graceful drain handle it. TLS/QUIC connection migration helps.
The result: a client reaches a near, healthy edge in one network hop with no client-visible machinery, and any edge/PoP/region failure re-routes in seconds without waiting on a DNS TTL.
2. Cache hierarchy and eviction: why “one big LRU” fails, then tiered RAM/NVMe + smart eviction
Naive approach: each edge box keeps one flat LRU cache in RAM; on miss, pull from origin.
Where it breaks:
- RAM is tiny vs the working set. A PoP’s hot set is terabytes; RAM per box is hundreds of GB. A pure-RAM LRU thrashes - it evicts warm objects constantly, hit ratio collapses, and misses stampede origin.
- LRU is fooled by scans. A one-time sweep over many cold objects (a crawler, a batch of unique URLs) evicts the genuinely hot objects to hold junk that will never be requested again. Classic LRU cache pollution.
- Every box caching independently wastes the PoP. If 40 boxes in a PoP each independently cache the same object, you store it 40 times and fill origin 40 times on cold-start. The PoP’s effective cache is 1/40th of its disk.
Evolution A: tiered RAM + NVMe per box. Keep the hottest objects in RAM (microsecond serve) and a much larger warm tier on local NVMe SSD (still sub-ms, terabytes per box). Lookups check RAM then NVMe then go up the hierarchy. This multiplies effective cache size ~100x per box and matches the power-law access pattern: a few objects are white-hot (RAM), a long warm tail lives on disk.
Evolution B: scan-resistant admission + eviction. Replace naive LRU with an algorithm that resists pollution:
- Admission control (e.g. TinyLFU): before caching a new object, check a compact frequency sketch - only admit it if it is likely more valuable than what it would evict. A one-hit-wonder from a scan never displaces a hot object.
- Eviction (LRU/segmented-LRU/S3-FIFO-style): evict by a blend of recency and frequency, not recency alone.
- Size-aware: a 2 GB object and a 2 KB object should not be treated equally; account for size so one huge cold file cannot evict thousands of hot small ones.
Evolution C: intra-PoP sharding by cache key (consistent hashing). Do not let every box cache everything. Consistent-hash the cache key across the PoP’s edge servers so each object has a home box (or a small replica set for the hottest). The L7 layer routes a request for object X to X’s home box. Now the PoP’s total cache is the sum of all boxes’ disks, not one box’s, and an object is filled from origin ~once per PoP, not once per box.
Request for /img/hero.jpg -> hash(key) -> box #17 owns it
box #17: RAM? -> NVMe? -> miss -> shield -> origin. Cached once, on box #17.
Replicate only the SUPER-hot keys to a few boxes to avoid a single-box hot spot
(the hot-key fix, applied inside the PoP).
- Freshness within the cache: each object carries a TTL (from
Cache-Control/Expiresor a customer override). Past TTL, the object is stale; the edge revalidates with origin (If-None-Match/If-Modified-Since) - a304refreshes the TTL cheaply without re-transferring bytes. Stale-while-revalidate lets the edge serve the slightly-stale copy instantly while revalidating in the background, so a TTL expiry never causes a user-facing miss latency. - Negative caching: cache
404s and errors briefly so a flood of requests for a missing object does not repeatedly hit origin.
The result: each edge serves the hot set from RAM in microseconds and the warm tail from NVMe in sub-ms, resists scan pollution so hit ratio stays high, and shards within the PoP so cache capacity and origin-fill efficiency scale with the whole PoP, not one box.
3. Origin protection and cache-fill: why per-PoP pulls stampede origin, then the shield tier
Naive approach: on any cache miss, the edge pulls directly from the customer’s origin.
Where it breaks:
- The per-PoP stampede. A newly-popular (or newly-purged) object is cold at all ~2,000 PoPs at once. If each pulls from origin, that is 2,000 simultaneous origin fetches for one object - a self-inflicted DDoS. Multiply by many objects during a deploy and the origin dies.
- The intra-PoP stampede (thundering herd). Even within one PoP, if 100,000 concurrent requests for the same uncached object each trigger an origin pull, origin gets 100,000 identical requests for one object in a millisecond.
- Origins are small. The whole reason a customer uses a CDN is that their origin cannot serve internet-scale traffic. The CDN must make origin see a tiny, roughly constant load regardless of how viral the content behind it goes.
Evolution A: request coalescing (single-flight) per cache. Within a cache, the first miss for an object acquires a lock and pulls from the next tier; all concurrent requests for the same object wait on that one in-flight fill and are served from its result. 100,000 concurrent misses become one upstream fetch.
100k concurrent misses for /video/seg_42.ts on one edge:
req #1 -> acquires fill-lock -> fetches from shield/origin
reqs #2..100000 -> park on the same in-flight fetch
fetch returns -> all 100k served from the one filled copy. Origin saw 1 request.
Evolution B: the regional shield (mid-tier cache) to collapse cross-PoP pulls. Insert a mid-tier cache tier between edges and origin. Many edges (all PoPs in a region) route their misses through a small number of shield caches. The shield coalesces and caches, so a region pulls a given object from origin exactly once, then serves every edge in the region from the shield.
[Edge PoP A] [Edge PoP B] ... [Edge PoP Z] (many edges in a region)
\ | /
\ | / misses go UP to the shield, not to origin
v v v
┌──────────── REGIONAL SHIELD ────────────┐ (few, large, well-connected)
│ coalesces + caches region's misses │
└───────────────────┬──────────────────────┘
│ ONE pull per object per region
▼
ORIGIN
- Two (or three) tiers: edge -> regional shield -> optionally a global “super-shield” close to origin. Each tier multiplies the coalescing factor. With shielding, effective origin egress drops from ~10% of traffic to well under 1%, and each origin sees a constant trickle - the whole point.
- The optimized backbone: shield-to-origin runs over the CDN’s private backbone / warm connection pools (persistent TCP/TLS to origin, HTTP/2 multiplexing), so even the rare pull is fast and does not pay a cold-handshake tax. This also accelerates dynamic/uncacheable traffic: the edge terminates the client TLS locally and reuses a hot origin connection over an optimized route.
- Origin health and failover: the shield tracks origin health; on origin failure it can serve stale content (“serve-stale-on-error”) rather than propagating a 5xx to users, and can fail over to a backup origin. Customers configure origin pools with health checks.
- Prefetch / push for predictable spikes: for a known event (a product launch, a game patch, a video premiere), the customer or the CDN can pre-warm shields and edges so even the first requests hit a warm cache instead of cold-filling under peak load.
The result: origins see a small, steady load no matter how viral the content gets - request coalescing kills the intra-cache herd, the shield tier kills the cross-PoP stampede, and stale-on-error plus backup origins keep serving even when an origin is down.
4. Cache invalidation: why “delete from every box” fails, then versioned keys + fan-out
Naive approach: on purge, send a “delete object X” message to every edge server on earth and wait.
Where it breaks:
- Fan-out explosion. A purge must reach hundreds of thousands of edge servers across thousands of PoPs. At ~50,000 purges/sec (BoE), a flat broadcast is billions of messages/sec - the control plane becomes the bottleneck and the network drowns in invalidation traffic.
- Synchronous is impossible. You cannot block the customer’s purge API call until every edge on earth has acked. Some edges are momentarily unreachable; waiting for global acks means purge latency is bounded by the slowest box on the planet.
- Consistency vs speed. Miss an edge and it serves stale bytes indefinitely (a customer’s “we pulled that leaked file” purge that silently failed on one PoP is a real incident). But blocking for perfect global consistency is unworkable at this scale.
Evolution A: fan-out tree, not a flat broadcast. Structure propagation as a tree: the central purge service writes the purge to a durable log and pushes to a handful of regional relays; each relay pushes to its PoPs; each PoP pushes to its edge servers. Fan-out is logarithmic in messages per node, and each tier parallelizes. A purge reaches all edges in seconds, and the tree self-heals (a relay retries; an edge that reconnects catches up from the log).
Purge API -> durable purge LOG (ordered, per-customer) -> version bump
│
├──> Region relay 1 ──> PoP ──> edge, edge, edge...
├──> Region relay 2 ──> PoP ──> edge, edge, edge...
└──> Region relay N ──> ...
Each edge applies the purge locally; acks flow back up for observability.
Evolution B: versioned cache keys / generation numbers - make purge O(1) and lazy. Instead of physically deleting bytes from every box, bake a version into the cache key and just bump the version. A purge becomes “increment the generation for this object/path/customer”; edges pick up the new generation via the fan-out tree, and any request now computes a new cache key that misses cleanly and re-fills. The stale bytes are never served (their key is dead) and get evicted lazily by normal LRU pressure - no synchronous global delete needed.
cache_key = hash(host + path + GENERATION[path])
Purge /img/hero.jpg -> GENERATION[/img/hero.jpg] bumped 7 -> 8
Next request keys on generation 8 -> clean miss -> re-fill fresh.
Old generation-7 object is now unreachable; evicted lazily. No global delete storm.
- Purge granularity: single URL, path prefix (
/assets/*), or surrogate tags (origin tags responses withSurrogate-Key: product-123; purging the tag invalidates every object carrying it - the fix for “invalidate everything related to product 123” without listing URLs). Tags map to generation bumps on the tag. - Soft vs hard purge: a soft purge marks objects stale (so
stale-while-revalidateserves the old copy once more while revalidating - zero origin spike, eventual freshness); a hard purge makes them immediately un-servable (for legal/security removals). Customers choose per call. - The immutable-URL escape hatch: the cleanest invalidation is none. Content-hashed URLs (
app.9f3c2a.js) are immutable - a new version is a new URL, so you never purge, you just stop referencing the old one. Static-asset pipelines lean on this so the vast majority of “updates” need no purge at all; purge is reserved for same-URL mutable content. - Consistency model: purge is eventually consistent with a tight SLO (all edges within a few seconds at p99). The durable, ordered purge log is the source of truth, so a reconnecting edge replays any purges it missed - no edge is permanently stale even if it was briefly offline.
The result: a customer purges an object and every edge on earth stops serving it within seconds, via a logarithmic fan-out tree plus lazy versioned keys, with tag-based bulk purge and a soft/hard choice - and immutable content-hashed URLs sidestep purge entirely for most static assets.
API Design & Data Schema
The vast majority of “API” traffic is plain HTTP GETs of assets served by edges (not a custom API); the interfaces below are the customer control-plane API plus the internal contracts that make the data plane work.
Customer-facing control-plane API
# --- Configure a distribution (a customer's CDN property) ---
POST /api/v1/distributions
Body: { "origin":"https://origin.acme.com", "domains":["cdn.acme.com"],
"default_ttl_s": 3600, "cache_key":{"query_allowlist":["v"]},
"tls_cert_id":"cert_88", "shield_region":"eu-west" }
-> 201 { "dist_id":"d_71", "cname_target":"acme.map.cdn.net", "status":"deploying" }
# --- Cache behavior rules (path-scoped overrides) ---
PUT /api/v1/distributions/{id}/behaviors
Body: [ { "path":"/api/*", "cache":false },
{ "path":"/assets/*", "ttl_s": 604800, "immutable": true },
{ "path":"/img/*", "ttl_s": 86400, "stale_while_revalidate_s": 600 } ]
-> 200 { "status":"propagating" } # pushed to all edges via control plane
# --- Purge / invalidation ---
POST /api/v1/distributions/{id}/purge
Body: { "type":"url", "urls":["/img/hero.jpg"] } # single/list
Body: { "type":"prefix", "prefix":"/assets/" } # path prefix
Body: { "type":"tag", "tags":["product-123"], "mode":"soft" } # surrogate key
-> 202 { "purge_id":"p_5c", "scope":"tag:product-123",
"eta_s": 5, "mode":"soft" } # async; returns immediately
GET /api/v1/distributions/{id}/purge/{purge_id}
-> 200 { "status":"completed", "edges_applied": 214113, "edges_total": 214113 }
# --- Analytics ---
GET /api/v1/distributions/{id}/stats?from=...&to=...&granularity=5m
-> 200 { "requests": 9.1e9, "hit_ratio": 0.976, "egress_bytes": 4.2e15,
"status_5xx": 0.0003, "top_urls":[ ... ] }
Edge-served asset request (the hot path, standard HTTP)
GET https://cdn.acme.com/assets/app.9f3c2a.js # served by nearest edge
Request headers: If-None-Match: "9f3c2a" # conditional revalidation
HIT -> 200, Cache-Control: public, max-age=604800, immutable
X-Cache: HIT (edge=LOS1, tier=RAM)
STALE-> edge revalidates with shield/origin; 304 refreshes TTL, or 200 with new body
MISS -> edge -> shield -> origin fill; X-Cache: MISS; then cached for next time
Internal data / config schemas
1. Control-plane config store - strongly consistent, globally replicated (a distributed KV / a small relational store, read-replicated to every region). Small, critical, read-mostly by the data plane. This is the one place that needs real consistency and durability.
Table: distributions (SQL / consistent KV, keyed by dist_id)
dist_id STRING PRIMARY KEY
customer_id STRING INDEX
domains JSON -- CNAMEs this distribution serves
origin_pool JSON -- origins + health-check config + failover order
default_ttl_s INT
cache_key_rule JSON -- which query/headers compose the key
behaviors JSON -- path-scoped rule list (ttl, cache on/off, swr, immutable)
tls_cert_id STRING -- reference into the cert store
shield_region STRING
version BIGINT -- bumped on any change; edges pull deltas by version
Table: purge_log (append-only, ordered per distribution - source of truth)
purge_id STRING PRIMARY KEY
dist_id STRING PARTITION KEY
scope STRING -- url | prefix | tag + value
generation BIGINT -- new generation number assigned to the scope
mode ENUM(soft, hard)
created_at TIMESTAMP
2. Edge cache (per box) - the disposable data plane store. Not a database; an in-process tiered cache (RAM index + NVMe blob store). No cross-box consistency; rebuilt by re-pulling on loss.
key = hash(host + normalized_path + cache_key_subset + generation)
value = { bytes, content_type, etag, ttl_deadline, size, last_access, hit_count }
tiers = RAM (hot LRU/W-TinyLFU) over NVMe (warm, size-aware eviction)
metadata: generation table per (dist, path/tag) -> current version (for lazy purge)
3. Routing / mapping state - the control plane’s view of the world. Drives DNS answers and anycast health withdrawal. Eventually consistent, refreshed every few seconds.
pop_health[pop_id] -> { healthy, load_pct, egress_gbps, withdraw_anycast? }
geo_map[prefix/asn] -> ranked list of nearest healthy PoPs
origin_health[dist_id] -> per-origin up/down for failover + serve-stale decisions
4. Analytics store - write-heavy rollups from edge logs (columnar / time-series OLAP). Edges ship logs asynchronously; a pipeline aggregates to per-distribution, per-time-bucket rollups (requests, hit ratio, bytes, status codes). Never on the serving path.
Why these choices: the config/control plane is small, relational-ish, and correctness-critical - a strongly-consistent, globally-replicated store, because serving the wrong cache rule or missing a purge is a real incident. The edge cache is enormous, hot, and disposable - an in-process RAM+NVMe tier with no consistency guarantees, because the origin owns durability and a lost cache just re-pulls; a SQL database here would be absurd. Routing state is small and fast-changing - an eventually-consistent map refreshed continuously. Analytics is append-heavy and aggregate-read - a columnar/OLAP store fed by rollups. Each layer gets exactly the consistency and cost model its access pattern demands; forcing one store to do all four is the naive mistake.
Bottlenecks & Scaling
Where it breaks first, in order, and the fix for each:
1. Origin overload on cold/viral content (breaks first). A newly-popular or freshly-purged object is cold everywhere; naive per-PoP, per-box pulls stampede the origin. Fix: the whole point of the design - request coalescing (single-flight) collapses concurrent misses within a cache to one upstream fetch; the regional shield tier collapses cross-PoP misses so a region pulls each object from origin once; serve-stale-on-error and backup origins keep serving during origin trouble; prefetch/pre-warm for known spikes. Effective origin egress stays well under 1% of total and roughly constant regardless of virality.
2. Edge cache hit ratio (the master lever). A 1% hit-ratio drop at 100 Tbps is ~1 Tbps of extra origin-bound and mid-tier traffic. Fix: tiered RAM + NVMe to hold a deep working set; scan-resistant admission (W-TinyLFU) so one-hit-wonders do not evict hot objects; intra-PoP consistent-hash sharding so the PoP’s whole disk is one logical cache and objects fill once per PoP; stale-while-revalidate so TTL expiry never causes user-facing miss latency; encourage immutable content-hashed URLs so the hot set caches forever.
3. Hot object / hot key (a single URL goes viral). Consistent hashing pins it to one edge box, which then melts under all of that object’s traffic. Fix: replicate the super-hot keys across several boxes in the PoP (detect via per-key hit counters) and let the L7 layer spread requests - the classic hot-key fan-out. RAM-tier serving makes a hot object nearly free per hit. Across PoPs the object is naturally replicated (each PoP caches its own copy), so virality spreads load geographically by construction.
4. Purge fan-out storm. 50,000 purges/sec times hundreds of thousands of edges is a message explosion. Fix: logarithmic fan-out tree (central -> regional relays -> PoP -> edge) instead of a flat broadcast; versioned/generation cache keys so purge is an O(1) version bump applied lazily, not a synchronous global delete; surrogate tags so bulk invalidation is one bump; a durable ordered purge log so reconnecting edges replay missed purges. Purge reaches every edge in seconds without drowning the control plane.
5. Request routing under PoP/region failure. A dead PoP must not black-hole traffic. Fix: Anycast self-heals via BGP withdrawal in seconds (no DNS TTL wait); GeoDNS with short TTLs and health-aware answers steers away from unhealthy/overloaded PoPs; L4 LB (consistent hashing / Maglev) inside the PoP survives individual box loss; graceful connection drain on planned maintenance.
6. Connection and TLS overhead at billions of req/sec. Handshakes and syscalls dominate before bytes ever move. Fix: TLS termination at the edge (short RTT), session resumption / 0-RTT (TLS 1.3, QUIC), HTTP/2 and HTTP/3 multiplexing to amortize connections, kernel-bypass / tuned network stacks (io_uring, DPDK-style) and sendfile/zero-copy from NVMe to NIC so a cache hit barely touches the CPU. Keep-alive pools to origin so misses do not pay cold handshakes.
7. Backbone / mid-tier bandwidth between edge and shield and origin. Cache-fill and shield traffic is real bandwidth. Fix: shielding minimizes it (fill each object per region once); the optimized private backbone with connection reuse and better routing than the public internet carries it; compression (Brotli/gzip at the edge) and range requests for large media reduce bytes moved; collapse forwarding so simultaneous fills of a big object share one transfer.
8. Shard keys, stated plainly. Intra-PoP cache placement -> cache key (host+path+key-subset+generation) via consistent hashing, so an object has a home box and the PoP disk is one logical cache; replicate only hot keys. Config/control store -> dist_id (a distribution and its rules/purges co-locate). Purge log + analytics -> dist_id (a customer’s purges are ordered together; stats aggregate per distribution). Routing map -> geo/ASN prefix -> nearest PoPs. Sharding the cache by anything but the cache key would break locality and re-fill objects redundantly; sharding config by customer keeps a distribution’s state together.
9. Single points of failure. Edge servers, PoPs, and regions are all redundant by construction - anycast + DNS route around any loss. The control plane is the one centralized brain, so it is globally replicated (consistent store with quorum + failover), and critically the data plane never blocks on it: edges serve from their last-known config and cache even if the control plane is briefly unreachable, catching up on config/purge deltas when it returns. The purge log is durable and ordered so nothing is permanently missed. Origins have health-checked pools with failover and serve-stale-on-error.
10. Multi-region / global correctness. The system is global by design - every PoP is independent and locally sufficient for serving. Config and purge propagate from the central control plane to regional relays and are eventually consistent with a tight SLO (seconds). No serving decision depends on a remote region being up; only the freshness of a just-changed config or a just-issued purge depends on propagation, and that is bounded to seconds. Analytics aggregate asynchronously and are allowed to lag.
Wrap-Up
The trade-offs that define this design:
- Distribute everything, centralize only control. The data plane is thousands of independent, locally-sufficient PoPs so that serving survives any failure and every user is close to a box; the only centralized component is a small control plane for config and purge, and the serving path never blocks on it - trading perfect global config consistency (seconds of propagation lag) for a data plane that is always up and always local. One data center was rejected as “not a CDN.”
- The cache hierarchy is the origin’s shield, not just a speedup. Request coalescing plus a regional mid-tier collapse billions of edge misses into a trickle of origin pulls, so origins see a small, constant load regardless of virality - trading an extra network hop on a miss for the survival of every origin behind the CDN. Direct edge-to-origin pulls were rejected as a self-inflicted DDoS.
- Routing is the network’s job, layered with the CDN’s. Anycast gives packet-level, self-healing, near-optimal routing for free; GeoDNS and load-aware steering add the control anycast lacks - trading some routing complexity for seconds-scale, TTL-free failover. A single DNS record was rejected for having neither locality nor failover.
- Invalidate lazily, fan out logarithmically, and prefer never invalidating. Versioned cache keys make purge an O(1) generation bump applied lazily; a fan-out tree reaches every edge in seconds without a broadcast storm; immutable content-hashed URLs sidestep purge entirely for most assets - trading a bounded window of eventual freshness for a purge system that scales. A synchronous global delete was rejected as impossible at this fan-out.
- Consistency spent where it matters. Strong consistency only for the control plane (configs, certs, the ordered purge log); everything on the serving path (cache content, routing, analytics) is eventually consistent - which is exactly what buys the cacheability, the locality, and the scale.
One-line summary: a CDN is thousands of independent edge PoPs placed close to users, each serving a tiered RAM+NVMe cache with a high hit ratio, fronted by anycast + GeoDNS routing that steers every client to a near healthy edge and self-heals on failure, backed by a regional shield tier and request coalescing that collapse billions of misses into a constant trickle of origin pulls, and kept coherent by a logarithmic purge fan-out over versioned cache keys - all driven by a small strongly-consistent control plane that the always-local data plane never has to wait on, so ~100 Tbps of egress and billions of requests per second are served fast, cheap, and without ever letting the origins behind it feel the load.
Comments