A DNS resolver sounds like a lookup table: a client asks “what is the IP for example.com,” you return an A record, done. That intuition survives about one query. A real recursive resolver is a globally distributed system whose whole job is to turn a name into an address by walking a delegation tree of servers it does not own, cache the answer for exactly as long as the record’s TTL allows and not one second more, hand back the geographically right address so a user in Mumbai gets the Mumbai edge of the site they asked for, and do all of this in under 50ms at p99 while sustaining trillions of queries per day. The interesting part is not “look up a name.” It is: thousands of resolver nodes spread across the planet behind one anycast IP, each running a recursive engine that chases NS delegations from the root down while a multi-tier cache absorbs 90%+ of traffic, each honoring per-record TTLs so answers are fresh without hammering authoritative servers, and each returning ECS-aware answers so the address you get is close to you - all while a single slow or dead authoritative server upstream must never stall the box. Let me build it properly.

Everything hard here - the recursive walk, the cache, TTL correctness, geo-awareness - exists to move two numbers: cache hit ratio (what fraction of queries never touch an authoritative server) and resolution latency (how fast a full recursion completes on a miss). Get those right and you have a fast resolver that barely loads the DNS hierarchy; get them wrong and you have a slow proxy that DDoSes the root and TLD servers. A resolver is not “a hash map of names to IPs.” It is a distributed system whose product is the cache hit, measured in nines of hit ratio and milliseconds of tail latency.

Functional Requirements (FR)

In scope:

  • Recursive resolution. A client (stub resolver on a laptop, phone, or server) sends a query for a name and record type; the resolver returns the answer, doing the full recursion itself - root, then TLD, then authoritative - if the answer is not cached.
  • All the common record types. A (IPv4), AAAA (IPv6), CNAME (alias), MX (mail), NS (delegation), TXT, SRV, PTR (reverse), SOA, CAA. Follow CNAME chains to a terminal address.
  • Caching with TTL respect. Cache every answer for exactly its TTL. Serve from cache while fresh; on expiry, re-resolve. Cache negative answers (NXDOMAIN, NODATA) too, bounded by the SOA minimum TTL. Never serve a record past its TTL.
  • Geo-aware / client-subnet answers. Many authoritative servers return different addresses depending on where the client is (CDN steering, geo load balancing). Support EDNS Client Subnet (ECS) so the resolver forwards a truncated client prefix upstream and caches answers keyed by that prefix, so a client gets a nearby address.
  • Transport support. Classic UDP/53 and TCP/53 (fallback for large answers and truncated TC responses), plus modern encrypted transports - DoT (DNS-over-TLS, 853), DoH (DNS-over-HTTPS, 443), and DoQ (DNS-over-QUIC).
  • DNSSEC validation. Optionally validate the signature chain (RRSIG/DNSKEY/DS from the root trust anchor down) so forged answers are rejected; return SERVFAIL on validation failure.
  • Resilience to bad upstreams. A slow, dead, or lying authoritative server must not stall the resolver: timeouts, retries across the NS set, and serve-stale on upstream failure.

Explicitly out of scope (say it so you own the scope):

  • Authoritative DNS hosting. I am building the resolver (the recursive side that clients query), not the authoritative servers that own zones. I define how I talk to authoritative servers; hosting example.com’s zone is a different system.
  • The root and TLD infrastructure. The 13 root server letters and the TLD registries are given; I consume them via a bundled root hints file. I do not run them.
  • Domain registration / registrar. Buying and managing domain names is a commerce system, unrelated to resolution.
  • A full recursive-vs-forwarding home router. I am designing an internet-scale public resolver (think 1.1.1.1 / 8.8.8.8 class), not a CPE forwarder, though the caching ideas overlap.
  • Content filtering / parental controls as a product. Real public resolvers bundle malware/family-filter variants; that is a policy layer bolted onto the answer path. I note the hook point and move on.

The one decision that drives everything: this is a read-astronomically-heavy, latency-critical, globally distributed system where the same small set of popular names is queried trillions of times, the answers are short-lived (TTL-bounded) and location-dependent, and the upstream authority servers I depend on are owned by other people and must be shielded from my traffic so hard that they see a tiny, roughly constant trickle no matter how much load I take. The whole design is about maximizing cache hit ratio and minimizing recursion latency while honoring TTLs exactly and returning the geographically correct answer.

Non-Functional Requirements (NFR)

  • Scale: Global, public resolver. On the order of trillions of queries/day - call it ~5 trillion/day, which averages ~50-60 million QPS and peaks 2-3x that. Hundreds of PoPs / metros, thousands of resolver nodes, all behind a handful of well-known anycast IPs.
  • Latency: Cache hit answered in single-digit milliseconds at the server plus the client-to-PoP RTT; target p99 end-to-end under 50ms. A cache miss requires a full recursion (root -> TLD -> authoritative, 2-4 upstream RTTs) and is the tail we fight to keep rare and bounded (aggressive parallelism, prefetch, cached delegations so we rarely start at the root).
  • Availability: 99.99%+ on the query path. If the resolver is down, every device configured to use it loses the internet, so this is load-bearing. Any node, PoP, or region failing must transparently fail over via anycast.
  • Consistency: Eventual and TTL-bounded by design. DNS is fundamentally eventually consistent - after an authoritative record changes, resolvers worldwide may serve the old value until its TTL expires. The one correctness guarantee that must be tight is never serve past TTL (except explicit, bounded serve-stale on upstream failure). Two PoPs may briefly hold different cached values for the same name; that is expected.
  • Durability: The cache is disposable - losing it just causes re-resolution from the hierarchy. Nothing on the query path is a system of record. What must be durable is small control-plane state: the root hints, DNSSEC trust anchors, config, and blocklists. The authoritative servers own the truth.
  • The real budget is cache hit ratio and upstream QPS. At 50M+ QPS, even a 1% miss rate is 500K+ recursions/sec fanning out to root/TLD/authoritative servers. Hit ratio is a first-order design constraint, and protecting the upstream hierarchy (which I do not own and cannot overload) is as important as protecting my own nodes.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the topology and the cache sizing.

Query volume:

Target: ~5 trillion queries/day.
   5e12 / 86,400 s = ~5.8e7 = ~58 million QPS average.
   Peak at ~2.5x average = ~145 million QPS.
Reads only, effectively: a resolver's "write" is a cache fill from a miss,
   not a client-driven write. So this is a ~99.99% read system by request count.

Read vs “write” (miss) split - the hit-ratio lever:

DNS traffic is a brutal power law: a tiny set of names (google.com, the big CDNs,
   ad/analytics domains) are queried constantly; a huge tail is rare.
Assume a steady-state cache HIT ratio of ~90% across a node
   (higher for bytes-of-popular-names, lower counting the long unique tail).
   Misses = 10% of 58M = ~5.8M recursions/sec globally that touch upstream.
Each miss may need root + TLD + authoritative lookups, BUT root and TLD
   delegations are themselves cached with long TTLs (days), so in practice a
   miss usually starts at the authoritative server, not the root.
   Effective root QPS from us: near zero (root NS/glue cached for days).
   Effective TLD QPS: low (TLD NS cached for hours-days).
   The real upstream load is authoritative lookups for freshly-expired names.

Per-PoP / per-node sizing:

Say ~1,000 resolver nodes across ~150 PoPs sharing 145M peak QPS.
   Average node: 145e6 / 1,000 = ~145K QPS. Big-metro nodes carry 5-10x more.
A tuned resolver box (mostly in-memory cache, small UDP packets) handles
   ~200K-500K QPS of cache hits per box. So a busy PoP is tens of nodes.
   Small packets (~50-100 bytes query, ~100-300 bytes answer) => the bottleneck
   is packets-per-second and syscalls, NOT bandwidth. ~145K QPS * ~200 B
   = ~29 MB/s = trivial bandwidth; it is a PPS and CPU problem.

Cache sizing:

Distinct names in the hot working set that yield ~90% of hits: on the order of
   a few million to tens of millions of cached RRsets per node.
   Each cached entry: name + type + value + TTL deadline + metadata ~ a few hundred bytes.
   10M entries * ~300 B = ~3 GB. With ECS (per-prefix variants of popular names)
   it multiplies - say 5-10x for the top names -> tens of GB.
   Give each node ~64-128 GB RAM; the cache lives in memory. Easy fit.
The long tail is effectively infinite and uncacheable-useful; we do not chase it.

Negative cache:

Junk queries (typos, random subdomains, malware beaconing to dead C2, misconfigured
   clients) are a LARGE fraction of traffic - often 20-40%.
   Caching NXDOMAIN/NODATA (bounded by SOA minimum TTL, capped at e.g. 5 min)
   is not optional: without it, every garbage query re-recurses and hammers TLDs.

The takeaways: this is a ~58M QPS average / ~145M peak small-packet system where the bottleneck is packets-per-second and cache hit ratio, not bandwidth; the cache is a tens-of-GB in-memory structure that must hold the power-law hot set plus per-prefix ECS variants; delegation caching (root/TLD NS with long TTLs) means a miss almost never starts at the root; and negative caching is mandatory because a huge chunk of traffic is garbage. Recursion, caching+TTL, geo-awareness, and upstream protection are the interview.

High-Level Design (HLD)

The architecture splits into a data plane (the query-answering path: anycast entry -> node -> cache -> recursive engine -> upstream authority, replicated across thousands of nodes, carrying the 58M+ QPS firehose) and a control plane (root hints and trust-anchor distribution, config and blocklist push, health/anycast management, telemetry aggregation - low-QPS, drives the data plane but is never on the hot query path). A client sends a query to an anycast IP; the nearest PoP receives it, answers from cache if fresh, otherwise recurses through the hierarchy, caches the result, and replies.

                       ┌──────────────────────── CONTROL PLANE ─────────────────────────┐
                       │  Root hints + DNSSEC trust anchors (bundled, rarely changing)   │
                       │  Config / blocklist / policy push  (variants: filtering IPs)    │
                       │  Anycast health + route management (withdraw dead PoPs)          │
                       │  Telemetry / query-log aggregation (sampled, privacy-scrubbed)   │
                       └───────▲────────────────┬───────────────────────┬─────────────────┘
                               │ metrics/health │ push config/anchors    │ health
   ─────────────────────────────┼──────────────┼───────────────────────┼──────────────────
   DATA PLANE (per PoP, x hundreds)             ▼                       │
   ┌─────────┐  1. query to      ┌──────────────────────────────────────┴────────────┐
   │ Stub    │  anycast IP       │  ANYCAST ENTRY + L4 LOAD BALANCE                   │
   │ resolver│──────────────────>│  - one IP announced from every PoP (BGP)          │
   │ (OS/app)│  UDP/TCP/DoT/DoH  │  - packet lands at NEAREST healthy PoP; LB -> node │
   └────┬────┘  /DoQ            └────────────────────────┬──────────────────────────┘
        │  2. "A? www.site.com"                          ▼
        │                        ┌──────────────────────────────────────────────────┐
        │<─── cache HIT (~90%) ──│  RESOLVER NODE                                     │
        │     single-digit ms    │  - parse query, normalize, build cache key        │
        │                        │  - CACHE lookup (RAM): fresh? -> serve            │
        │                        │  - (optional) DNSSEC validate                     │
        │                        │  - on MISS -> hand to RECURSIVE ENGINE ─┐         │
        │                        └─────────────────────────────────────────┼─────────┘
        │                                                                   │ miss
        │                                    ┌──────────────────────────────▼─────────┐
        │                                    │  RECURSIVE ENGINE                       │
        │                                    │  - find deepest cached delegation (NS)  │
        │                                    │  - walk down: [root]->TLD->authoritative│
        │                                    │  - parallel-query the NS set, pick fast │
        │                                    │  - coalesce duplicate in-flight queries │
        │                                    └──────────────┬──────────────────────────┘
        │                                                   │ UDP/TCP to authority
        │                                    ┌──────────────▼──────────────────────────┐
        │                                    │  UPSTREAM HIERARCHY (not owned by us)    │
        │                                    │  root NS  ->  TLD NS  ->  authoritative  │
        │                                    │  (root/TLD cached for days; rarely hit)  │
        │                                    └──────────────────────────────────────────┘

Query flow (client resolves a name):

  1. The client’s stub resolver sends the query to the resolver’s anycast IP (e.g. 1.1.1.1). BGP routes the packet to the topologically nearest PoP announcing that IP. An L4 load balancer (consistent hashing / Maglev) inside the PoP picks a specific resolver node.
  2. The node parses the query (name + type + class), normalizes the name (lowercase, strip trailing dot handling), and builds a cache key = (qname, qtype, ECS-prefix?).
  3. The node checks its in-memory cache. If a fresh answer exists (TTL not expired), it serves immediately - single-digit ms. This is ~90% of queries. If DNSSEC is enabled, the cached answer’s validation state is remembered so it is not re-validated on every hit.
  4. On a miss (or expired entry), the node hands the query to the recursive engine. The engine does not blindly start at the root: it finds the deepest cached delegation for the name (e.g. it already has the .com NS set cached), and starts the walk there. It queries the authoritative name servers for the zone, following NS referrals and CNAME chains until it gets the answer.
  5. To keep the tail bounded, the engine queries multiple name servers in the NS set in parallel (or in fast succession) and takes the first good answer, applies short timeouts with retry, and coalesces duplicate concurrent misses for the same name into one in-flight recursion.
  6. The engine caches every RRset it learns (the answer, plus any delegation NS/glue) each with its own TTL, optionally DNSSEC-validates the chain, and returns the answer to the client. Subsequent queries hit the cache.

Control flow (config, anchors, health):

  1. Root hints and DNSSEC trust anchors are bundled and updated rarely via the control plane; config, policy (filtering variants), and blocklists are pushed to all nodes.
  2. Nodes report health/load; unhealthy PoPs withdraw their anycast announcement so traffic drains to the next-nearest PoP automatically. Sampled, privacy-scrubbed telemetry flows up for capacity and abuse analysis.

The key insight: the data plane is a stateless-ish, cache-heavy answering path replicated across thousands of nodes, each locally sufficient; the answering path never blocks on the control plane, delegation caching means a miss rarely touches the root, and the upstream hierarchy I do not own is shielded by the cache so it sees a trickle, not the firehose.

Component Deep Dive

The hard parts, naive-first then evolved: (1) the recursive engine - how you resolve a name without starting at the root every time and without one slow authority stalling you; (2) the cache and TTL - how you honor TTLs exactly while keeping hit ratio high and the tail latency low; (3) geo-aware answers - how ECS gives the right address without exploding the cache; (4) upstream protection and abuse - how trillions of queries do not turn into a DDoS on the DNS hierarchy.

1. The recursive engine: why “start at the root every time” fails, then delegation caching + parallel NS

Naive approach: on every miss, start at a root server, ask for the name, follow each referral down (root -> TLD -> authoritative), querying one server at a time and waiting for each to answer.

Where it breaks:

  • Root/TLD stampede. If every miss starts at the root, trillions of queries/day means millions of QPS hitting the 13 root letters and the TLD servers - a self-inflicted DDoS on infrastructure you do not own and cannot overload. The root operators would (rightly) treat you as an attacker.
  • Latency stacks up. Root RTT + TLD RTT + authoritative RTT, serially, on every miss. Three round trips before you can answer. At internet distances that is easily 150-300ms - blowing the 50ms budget on every miss.
  • One slow server stalls everything. A zone’s NS set has multiple servers; if you always ask the first and it is slow or dead, you wait for a timeout (seconds) before trying the next. Tail latency explodes.

Evolution A: cache the delegations, so a miss rarely starts at the root. Every referral you get - the root’s list of .com name servers, .com’s list of example.com name servers - is itself a cacheable RRset with a (usually long) TTL. Cache them. Now a miss for blog.example.com starts from the deepest cached delegation you already hold:

Cache already has: root NS (TTL days), .com NS + glue (TTL ~2 days).
   Miss for blog.example.com:
     - start NOT at root, but at the cached .com name servers
     - ask a .com server -> get example.com's NS set -> cache it
     - ask example.com's server -> get the A record -> cache + answer
   Root was not touched. TLD touched only because example.com's NS wasn't cached yet;
   next time, that's cached too and only the authoritative server is hit.

This is why real resolver upstream QPS to the root is near zero: the root’s own delegation data barely changes and caches for days.

Evolution B: query the NS set in parallel, with timeouts and retry. A zone has several authoritative servers. Instead of serial “ask #1, wait, timeout, ask #2,” the engine fires at several name servers concurrently (or with a very short stagger) and takes the first valid answer, tracking per-server RTT so it prefers historically fast servers (SRTT-based selection, like bind/unbound).

example.com NS = { ns1 (RTT 20ms), ns2 (RTT 200ms), ns3 (dead) }
   Engine prefers ns1 (lowest smoothed RTT); if no answer in ~T ms, fan out to ns2/ns3.
   First good answer wins; dead/slow servers get their RTT penalized for next time.
   Hard cap on total recursion time -> return SERVFAIL rather than hang.

Evolution C: query coalescing (single-flight) for concurrent misses. When a popular name’s TTL expires, thousands of concurrent client queries can all miss at once. Without coalescing, that is thousands of identical recursions hitting the authoritative server in the same millisecond.

TTL for cdn.bigsite.com expires; 10,000 concurrent client queries arrive:
   query #1 -> starts the recursion, marks name "in flight"
   queries #2..10000 -> park on the same in-flight recursion
   recursion returns -> all 10,000 answered from the one result; cache filled once.
   Authoritative server saw exactly ONE query.

Evolution D: CNAME chain following and glue handling. The engine follows CNAME/DNAME aliases to a terminal address in one logical resolution (returning the full chain), and uses glue records (the A/AAAA addresses of name servers shipped inside a referral) to avoid a separate lookup just to find the NS’s own IP - and guards against out-of-bailiwick glue to prevent cache poisoning.

The result: a miss almost never starts at the root, resolves in usually one or two upstream RTTs thanks to cached delegations, is bounded in tail latency by parallel NS querying with RTT-aware selection and hard timeouts, and never stampedes an authority thanks to single-flight coalescing.

2. Cache and TTL: why “one map, evict on expiry” fails, then TTL-exact serving + prefetch + serve-stale

Naive approach: one hash map of name -> (value, expiry); on lookup, if now < expiry serve it, else delete and re-resolve.

Where it breaks:

  • The expiry cliff = a latency spike and a herd. The instant a popular name’s TTL hits zero, the next query (and the thousands behind it) all miss and must wait for a full recursion. Users see a sudden latency spike every TTL period on exactly the most popular names. And they stampede the authority (mitigated by coalescing, but the latency cliff remains).
  • Ignoring per-RRset TTLs corrupts freshness. A CNAME might have TTL 3600 while the A it points to has TTL 30. If you cache the whole answer under one TTL you either serve the A stale (wrong) or re-fetch the CNAME needlessly. Each RRset must carry and be evicted by its own TTL.
  • No negative caching = garbage traffic re-recurses forever. Every typo and every malware beacon to a dead domain re-runs a full recursion to prove it is still NXDOMAIN, hammering TLDs.
  • Unbounded memory. A pure “cache everything forever until expiry” fills RAM with a long tail of one-hit names, evicting the hot set.

Evolution A: per-RRset TTL, stored as an absolute deadline. Cache each RRset (not each answer) keyed by (name, type) with ttl_deadline = now + record_TTL. On lookup, remaining TTL = deadline - now, and you return that decremented TTL to the client (so downstream caches also expire correctly). Never serve a record whose deadline has passed. Clamp TTLs to sane bounds (e.g. min 0-5s, max ~1-7 days, and a hard cap so a hostile authority can’t pin junk in your cache forever).

Evolution B: prefetch popular names before they expire. Kill the expiry cliff: for names that are queried frequently, re-resolve them in the background shortly before their TTL expires (e.g. when a query arrives with <10% of TTL remaining on a hot record). The refresh happens off the query’s critical path, so clients keep getting cache hits and never see the miss latency.

cdn.bigsite.com, TTL 60s, very hot.
   At ~54s (90% elapsed) a query arrives -> serve cached answer instantly (hit)
      AND kick off a background re-resolution.
   Background recursion completes, cache updated with fresh 60s TTL.
   No client ever hit the expiry cliff. The authority saw one refresh, not a herd.

Evolution C: serve-stale on failure (RFC 8767). If the upstream authority is down or times out when a TTL has expired, serving SERVFAIL breaks the internet for that name. Instead, keep expired records for a bounded grace window and, only when a fresh resolution fails, serve the stale answer (with a short TTL) rather than an error - trading a bounded staleness for availability.

Evolution D: negative caching, bounded by SOA. Cache NXDOMAIN and NODATA responses too, using the zone’s SOA minimum TTL (capped, e.g. at 5 min per RFC 2308). Now a flood of queries for a non-existent name is absorbed by the cache instead of re-recursing.

Evolution E: bounded, scan-resistant eviction. The cache is a fixed-size in-memory store (tens of GB). Evict by a blend of recency, frequency, and remaining TTL (LRU with frequency awareness / W-TinyLFU-style admission) so a burst of unique junk names cannot evict the genuinely hot set. Aggressively age out negative and one-hit entries.

The result: TTLs are honored to the second (absolute deadlines, decremented on serve), the hot set never hits an expiry cliff (prefetch), the resolver stays up when authorities do not (bounded serve-stale), garbage traffic is absorbed (negative caching), and memory stays bounded and pollution-resistant (scan-resistant eviction).

3. Geo-aware answers: why “cache one answer per name” fails, then EDNS Client Subnet

Naive approach: cache exactly one answer per (name, type) and hand it to everyone.

Where it breaks:

  • CDN steering breaks. Big sites use DNS to steer users to a nearby edge: the authoritative server returns different A records depending on where the query appears to come from. A recursive resolver’s own IP is what the authority sees - so if a user in Mumbai queries a resolver node that egresses from, or is perceived as, one location, everyone behind that resolver gets the same CDN edge, possibly far from many of them.
  • One global answer is wrong for a location-dependent record. Caching a single answer and serving it worldwide defeats geo load balancing entirely - the whole point of the authority returning location-specific addresses is lost.

Evolution A: EDNS Client Subnet (ECS). The resolver includes a truncated prefix of the client’s IP (e.g. /24 for IPv4, /56 for IPv6 - truncated for privacy) in the upstream query via the ECS EDNS option. The authoritative server uses that prefix to pick a nearby address and returns a scope prefix saying “this answer is valid for this prefix length.” The resolver then caches the answer keyed by (name, type, client-prefix) and only serves it to clients in that prefix.

Client 203.0.113.45 in Mumbai queries A? cdn.bigsite.com
   Resolver forwards upstream with ECS = 203.0.113.0/24
   Authority returns: A = <Mumbai edge IP>, scope = /24
   Resolver caches: (cdn.bigsite.com, A, 203.0.113.0/24) -> Mumbai edge, TTL honored
   A client in 198.51.100.0/24 (London) gets a SEPARATE cache entry -> London edge.

Where ECS alone strains: it multiplies the cache - a popular name now has one entry per client prefix instead of one entry total, potentially thousands of variants. And it leaks client location upstream (a privacy cost). So:

  • Cache-key by the returned scope, not the sent prefix. The authority tells you how broad the answer is (the scope). If it returns scope /0, the answer is global - cache it once. If /24, cache per /24. Keying by the scope keeps variants only as granular as the authority actually needs.
  • Aggregate cold names, use ECS only where it matters. Only forward ECS to authorities that use it (learn this per-zone), and for names that are not geo-steered, keep a single global entry. Do not blow up the cache for names that return the same answer everywhere.
  • Privacy-truncate and allow opt-out. Truncate to /24/56 max, and respect authorities and clients that disable ECS (privacy-focused public resolvers often send a /0 ECS meaning “do not use my location”).
  • Anycast already helps. Because the resolver itself is anycast, a Mumbai client already hits a Mumbai PoP; combined with ECS, the answer and the resolver are both local.

The result: clients get a geographically appropriate address (CDN steering works), the cache does not explode because entries are keyed by the authority-declared scope and only geo-steered names get per-prefix variants, and privacy is bounded by prefix truncation and opt-out.

4. Upstream protection and abuse handling: why “just forward everything” fails, then shielding + rate control

Naive approach: for anything not cached, recurse upstream; accept and answer every client query as it comes.

Where it breaks:

  • The DNS hierarchy is not yours to overload. Root, TLD, and authoritative servers are third-party infrastructure. Trillions of client queries with a naive cache would translate into a crushing upstream load. You must make upstream see a small, roughly constant trickle regardless of your intake.
  • Junk and abuse are a huge fraction of traffic. Random-subdomain floods (a known attack: <random>.victim.com forces the resolver to recurse to the victim’s authority because each name is unique and uncacheable), reflection/amplification attempts using your resolver as a weapon, and malware beaconing all target the recursion path.
  • A single misbehaving upstream can wreck you. A lying or looping authority (NS records that point at each other) can send the engine into an infinite delegation loop.

Evolution A: the cache + coalescing + delegation caching is the shield. Sections 1-2 already collapse most of it: ~90% hit ratio means only ~10% recurses; delegation caching keeps root/TLD QPS near zero; single-flight coalescing means one recursion per name no matter how many concurrent clients. Negative caching absorbs the garbage-name floods. This is the primary protection.

Evolution B: per-target rate limiting and loop/depth guards. Cap outbound QPS per authoritative server / per zone so a random-subdomain attack against one victim’s zone cannot make your resolver flood that zone (you protect the victim and yourself). Bound recursion depth and total time, detect NS loops (a delegation that does not descend), and cap the number of upstream queries per client query.

Attack: 1M queries/sec for <random-uuid>.victim.com (each name unique -> always misses)
   Without guards: 1M recursions/sec hammer victim.com's authoritative servers.
   With guards: per-zone outbound cap (e.g. a few hundred QPS to victim.com);
                excess client queries answered SERVFAIL/dropped, not forwarded.
   Victim's authority is protected; our recursion path is not saturated.

Evolution C: anti-spoofing and cache-poisoning defense. UDP DNS is spoofable, so the engine randomizes the query source port and the transaction ID (0x20 case randomization on the qname too), and only accepts an answer whose port/ID/case/question match. It rejects out-of-bailiwick records (an authority for evil.com cannot inject a record for bank.com). DNSSEC validation, where enabled, cryptographically rejects forged answers by checking the RRSIG chain from the root trust anchor down.

Evolution D: abuse controls on the ingress. Per-client-IP rate limiting (with care - many users share NAT’d IPs), Response Rate Limiting to blunt amplification, and dropping obviously malformed or reflection-shaped traffic. Encrypted transports (DoT/DoH/DoQ) both improve privacy and make it harder to spoof/abuse the resolver.

The result: the upstream DNS hierarchy sees a small, roughly constant trickle no matter how much traffic or abuse arrives - the cache and coalescing shield it, per-zone rate limits and loop guards contain random-subdomain floods, and port/ID/case randomization plus DNSSEC keep the cache from being poisoned.

API Design & Data Schema

The vast majority of “API” traffic is the DNS wire protocol itself (UDP/TCP/53, DoT/853, DoH/443, DoQ) - not a custom REST API. The interfaces below are the wire query contract plus the internal cache/config schemas.

The query interface (DNS wire protocol, shown logically)

# Classic UDP/TCP (RFC 1035 message): a query and its answer, logically:
QUERY:
  { id, flags:{ RD=1 (recursion desired) }, question:{ qname, qtype, qclass=IN },
    edns:{ udp_size:1232, DO=1 (DNSSEC ok)?, ECS:{ family, prefix, source_len:24 }? } }

RESPONSE:
  { id (echoed), flags:{ QR=1, RA=1 (recursion available), AA=0, RCODE },
    answer:[ { name, type, ttl (decremented), rdata } ... ],
    authority:[ NS ... ], additional:[ glue A/AAAA, edns:{ ECS:{ scope_len } } ],
    RCODE in { NOERROR, NXDOMAIN, SERVFAIL, REFUSED } }

# DoH (RFC 8484): the same DNS message over HTTPS
GET  /dns-query?dns=<base64url(DNS message)>        Accept: application/dns-message
POST /dns-query   Content-Type: application/dns-message   Body: <raw DNS message>
  -> 200 application/dns-message  <raw DNS response>   # cache-control mirrors min TTL

# DoT (RFC 7858): DNS messages over a TLS connection on port 853 (length-prefixed).
# DoQ (RFC 9250): DNS messages over QUIC on port 853.

Example resolution of www.example.com:

Client -> Resolver:  A? www.example.com  RD=1  ECS=203.0.113.0/24  DO=1
Resolver (cache miss) -> .com NS:  NS? example.com   (starts from cached .com delegation)
   <- referral: example.com NS = ns1.example.com (+ glue A)
Resolver -> ns1.example.com:  A? www.example.com  ECS=203.0.113.0/24
   <- CNAME www.example.com -> web.cdn.net ; then A web.cdn.net = <nearby edge> scope=/24
Resolver caches each RRset by its own TTL, keyed with ECS scope; DNSSEC-validates chain.
Resolver -> Client:  200 NOERROR  CNAME + A (ttl decremented)  RA=1  AD=1 (validated)

Internal data / config schemas

1. Resolver cache (per node) - the disposable data-plane store. Not a database; an in-memory, concurrent, sharded map with TTL eviction. No cross-node consistency; rebuilt by re-resolving on loss.

Cache entry, keyed by (qname_lowercased, qtype, ecs_scope_prefix?):
  rrset          : [ rdata ... ]              -- the records
  ttl_deadline   : absolute timestamp         -- now + record TTL; serve remaining = deadline-now
  original_ttl   : int                        -- for prefetch threshold + stale window
  rcode          : NOERROR | NXDOMAIN | NODATA -- negative entries cached too (SOA-bounded)
  dnssec_state   : secure | insecure | bogus  -- remembered so hits skip re-validation
  ecs_scope_len  : int?                        -- how broadly this answer applies
  last_access    : ts   ;  hit_count : int     -- for eviction (LFU/LRU) + prefetch hotness
  server_srtt    : per-NS smoothed RTT (in a sibling table) -- for parallel-NS selection

Structure: sharded by hash(qname) across CPU cores (lock-per-shard); fixed byte budget;
           eviction = W-TinyLFU admission + LRU/TTL; negative + one-hit entries aged fast.
Delegation entries (root NS, TLD NS + glue) live in the SAME cache with long TTLs.

2. Control-plane config store - small, strongly consistent, globally replicated. Read-mostly by the data plane; correctness-critical. A distributed KV / small relational store replicated to every region.

root_hints        : the 13 root server names + addresses (bootstrap; rarely changes)
trust_anchors     : DNSSEC root KSK(s) (RFC 5011 rollover) -- correctness-critical
config[version]   : timeouts, TTL clamps (min/max), ECS policy, cache byte budget,
                    per-zone/per-client rate limits, prefetch thresholds, serve-stale window
policy_variants   : blocklists per resolver-IP variant (malware / family filter / unfiltered)
                    -> answers overridden to NXDOMAIN or a sinkhole for blocked names

3. Routing / anycast health state - the control plane’s view of PoPs. Drives BGP announcement withdrawal. Eventually consistent, refreshed continuously.

pop_health[pop_id] -> { healthy, load_pct, qps, node_count, withdraw_anycast? }
node_health[node]  -> { up, cache_hit_ratio, p99_ms, upstream_error_rate }

4. Telemetry / query-log store - sampled, privacy-scrubbed, write-heavy (columnar/OLAP). Nodes ship sampled and anonymized logs asynchronously; a pipeline rolls up QPS, hit ratio, RCODE distribution, top qnames, abuse signatures. Never on the query path; heavily privacy-constrained (public resolvers commit to minimal logging).

Why these choices: the cache is enormous, hot, and disposable - an in-memory sharded structure with no consistency guarantees, because the authoritative servers own the truth and a lost cache just re-resolves; a durable database here would be absurd and slow. The config/trust-anchor store is tiny and correctness-critical - a strongly-consistent, globally-replicated store, because a wrong trust anchor or a stale blocklist is a real security incident. Anycast/health state is small and fast-changing - eventually consistent, refreshed every few seconds. Telemetry is append-heavy and aggregate-read - a columnar/OLAP store fed by sampled rollups, deliberately minimal for privacy. Each layer gets exactly the consistency, durability, and cost model its access pattern demands; forcing one store to do all four is the naive mistake. Notably there is no relational database on the query path at all - DNS resolution is a caching-and-recursion problem, not a transactional one.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Cache hit ratio (the master lever, breaks first if wrong). At ~58M QPS, each 1% of miss rate is ~580K recursions/sec fanning out to authorities. Fix: hold a deep hot set in tens of GB of RAM per node; delegation caching so misses rarely start at root/TLD; prefetch hot names before expiry to avoid the cliff; negative caching to absorb garbage; scan-resistant eviction (W-TinyLFU) so junk floods do not evict the hot set. Hit ratio is designed for, not admired afterward.

2. Upstream (authoritative/TLD/root) overload. The DNS hierarchy is third-party and must see a trickle. Fix: the cache + single-flight coalescing (one recursion per name regardless of concurrent client count) + delegation caching collapse the firehose to a trickle; per-zone outbound rate limits and loop/depth guards contain random-subdomain attacks so a victim’s authority (and your own recursion path) is protected. Effective root QPS stays near zero; per-authority QPS stays low and roughly constant.

3. The expiry cliff / thundering herd on popular names. A hot name’s TTL expiring causes a synchronized miss-and-recurse for thousands of concurrent clients. Fix: prefetch refreshes hot names off the critical path before expiry; coalescing ensures even a genuine simultaneous miss is one upstream query; serve-stale-on-error keeps answering if the refresh fails. Clients never see the cliff.

4. Recursion tail latency (the p99 killer). A slow or dead authoritative server can add seconds to a miss. Fix: parallel querying of the NS set with RTT-aware (SRTT) server selection, short timeouts with retry across servers, hard caps on total recursion time (return SERVFAIL rather than hang), and cached delegations so most misses are one RTT, not three. Prefetch keeps popular names out of the miss path entirely.

5. Hot key / hot name (a single name goes hyper-viral or is attacked). Consistent-hash sharding could pin one blistering name to one CPU shard or node. Fix: hot names are almost always cache hits (RAM-served, nearly free per query), so a single name rarely bottlenecks. Within a node, shard by hash(qname) across all cores; across a PoP, the L4 LB spreads client connections and each node caches its own copy of the hot name. Across PoPs, anycast spreads load geographically by construction. For an attack (unique random subnames under one zone), the per-zone rate limit and negative cache contain it.

6. Packets-per-second and syscall overhead at 145M peak QPS. DNS is tiny packets; the bottleneck is PPS and per-packet CPU, not bandwidth. Fix: kernel-bypass / batched I/O (XDP/eBPF, recvmmsg/sendmmsg, io_uring, DPDK-style stacks) to answer cache hits with minimal CPU; multi-core sharding so each core owns a cache shard lock-free-ish; UDP fast path for the common small query. Bandwidth is a non-issue; PPS is the design constraint.

7. Anycast routing under PoP/region failure. A dead PoP must not black-hole queries. Fix: anycast self-heals via BGP withdrawal in seconds (no DNS TTL wait, since clients target a fixed anycast IP); unhealthy nodes are pulled from the L4 LB; the client’s stub resolver also has a secondary resolver configured as a last resort. Anycast plus stateless UDP means a re-route mid-flight just retries harmlessly.

8. Shard keys, stated plainly. Cache placement within a node -> hash(qname) across CPU shards (a name’s records co-locate; ECS variants hang off the same key). Across nodes there is no shared shard - each node has its own full cache (shared-nothing), because the cache is disposable and re-derivable; you trade some duplicate upstream fills for zero cross-node coordination. Config/trust store -> replicated globally (tiny, read-mostly). Telemetry -> partition by time + PoP. Sharding the cache across nodes (a distributed cache) was rejected: the coordination and network hop would cost more than the occasional duplicate recursion it saves at these packet rates.

9. Single points of failure. Nodes, PoPs, and regions are redundant by construction - anycast routes around any loss, and every node is locally sufficient (its own cache + recursion). The control plane is the one centralized brain, so it is globally replicated (consistent store with quorum + failover), and the data plane never blocks on it: nodes serve from their last-known config, cache, root hints, and trust anchors even if the control plane is briefly unreachable, catching up on config/blocklist deltas when it returns. The upstream hierarchy’s own redundancy (multiple root letters, multiple NS per zone) is used via parallel NS querying.

10. Global correctness and consistency. The system is eventually consistent by nature: after an authoritative record changes, resolvers worldwide may serve the old value until its TTL expires - this is DNS working as designed, not a bug. The tight guarantees are never serve past TTL (absolute deadlines, TTL decremented on serve) except a bounded serve-stale window on upstream failure, and DNSSEC validation where enabled so answers are authentic. Two PoPs holding briefly different cached values for the same name is expected and fine. No answering decision depends on a remote region being up.

Wrap-Up

The trade-offs that define this design:

  • Distribute everything, centralize only control. The data plane is thousands of shared-nothing, locally-sufficient resolver nodes behind anycast, so answering survives any failure and every user hits a near box; the only centralized component is a small control plane for config, trust anchors, and blocklists, and the query path never blocks on it - trading duplicate upstream fills and seconds of config propagation lag for a data plane that is always up and always local. A single central resolver was rejected as neither local nor survivable.
  • The cache is the hierarchy’s shield, not just a speedup. A high hit ratio plus single-flight coalescing plus delegation caching collapse trillions of client queries into a trickle of upstream lookups, so the root, TLDs, and authoritative servers I do not own see a small, constant load - trading TTL-bounded staleness for the survival of infrastructure I cannot overload. Starting every miss at the root was rejected as a self-inflicted DDoS.
  • Honor TTLs exactly, but never let expiry hurt users. Absolute per-RRset deadlines mean I never serve stale (except a bounded serve-stale-on-error window for availability); prefetch refreshes hot names before the cliff; negative caching absorbs garbage - trading a little background re-resolution and a bounded staleness for zero user-facing expiry spikes. A naive “delete on expiry, recurse on next query” was rejected for its latency cliff and herd.
  • Geo-awareness via ECS, keyed by the authority’s declared scope. Forwarding a truncated client prefix gets users a nearby CDN edge; keying the cache by the returned scope (not always the client) keeps the per-prefix cache blow-up only as granular as the authority actually needs - trading some cache multiplication and a privacy cost (bounded by truncation and opt-out) for correct geo-steering. Caching one global answer per name was rejected for breaking CDN steering.
  • Consistency spent where it matters. Strong consistency only for the tiny control plane (config, DNSSEC trust anchors, blocklists); everything on the query path (cache content, routing, telemetry) is eventually consistent and TTL-bounded - which is exactly what buys the cacheability, the locality, and the scale.

One-line summary: a global DNS resolver is thousands of shared-nothing nodes behind a few anycast IPs, each running a recursive engine that starts from cached delegations and queries authoritative NS sets in parallel, fronted by an in-memory cache that honors per-RRset TTLs exactly (with prefetch, negative caching, and bounded serve-stale) and keys geo-steered answers by EDNS Client Subnet scope, coalescing concurrent misses and rate-limiting per zone so the DNS hierarchy it does not own sees only a trickle - all driven by a small strongly-consistent control plane the always-local data plane never waits on, so trillions of queries a day are answered in under 50ms at p99 without ever overloading the root, the TLDs, or the authorities behind them.