Everyone starts a web crawler the same way: a queue of URLs, a while loop, fetch, parse out the links, push them back on the queue, repeat. It works on ten pages. Then the interviewer turns it into the real problem: crawl the whole web - billions of pages - politely (never hammer one server), without downloading the same page twice, without getting stuck in an infinite maze of dynamically generated URLs, across hundreds of machines, and refresh it all continuously because the web changes under you.

A web crawler is deceptive because the naive while loop is correct - it just does not survive contact with scale, hostile servers, or the shape of the real web. Almost every hard part hides inside one component: the URL frontier, the thing that decides what to crawl next. Get the frontier right - politeness, prioritization, dedup, and trap resistance all live there or right next to it - and the rest is plumbing. Get it wrong and you either get banned from every site you touch, drown in duplicate work, or wander forever inside a single crawler trap.

Let me do it properly.

Functional Requirements (FR)

In scope:

  • Fetch pages from a seed set outward. Given a set of seed URLs, download each page, extract the hyperlinks, and recursively crawl the discovered links - the classic breadth-first (or priority-first) graph traversal of the web.
  • Store the raw content. Persist the fetched page (HTML, and optionally other content types) so downstream consumers (an indexer, a search engine, an ML training pipeline) can process it. The crawler produces content; it does not index it.
  • Respect politeness. Never overload a single host. Obey robots.txt, honor crawl-delay, and rate-limit per domain so we look like a well-behaved bot, not a DDoS.
  • Deduplicate. Do not crawl the same URL twice, and detect when two different URLs return the same content (mirrors, session IDs, tracking params).
  • Prioritize. Not all pages are equal. High-value pages (popular, frequently updated, high PageRank-ish) should be crawled sooner and refreshed more often than a dead forum from 2009.
  • Recrawl / freshness. The web changes. Pages must be revisited on a schedule that reflects how often they actually change, so the corpus does not go stale.
  • Be resilient to traps. Survive infinite URL spaces (calendars, faceted search, session-ID loops), spider traps, and hostile or broken servers without getting stuck or crashing.

Explicitly out of scope (say this so you control the scope):

  • Indexing / ranking / search. We produce a corpus of raw pages. Building an inverted index and serving search queries is a separate system entirely.
  • JavaScript rendering. Full headless-browser rendering of SPA content is an expensive add-on (a render farm). The base design fetches raw HTTP responses; we note where a render tier grafts on.
  • Deep parsing / entity extraction. We extract links (and enough to dedup and prioritize). Structured extraction, NLP, and content classification are downstream.
  • Auth / paywalled / logged-in content. We crawl the public web. Anything behind a login is out.

The one decision that drives everything: a web crawler is a massive, long-running, distributed graph traversal where the scarce resources are politeness (per-host request budget) and dedup state (which URLs and which content we have already seen), not raw fetch throughput. Bandwidth is cheap; not getting banned and not doing redundant work are hard. So the design is dominated by the URL frontier - the component that decides what to fetch when, at what priority, without violating politeness or repeating itself.

Non-Functional Requirements (NFR)

  • Scale: target ~10B pages in the corpus, refreshed continuously. Sustained fetch rate on the order of ~1B pages/month, which is a few thousand to tens of thousands of pages/second aggregate. Hundreds of worker machines. The frontier holds billions of pending URLs; the “seen” set tracks tens of billions of URLs.
  • Throughput over latency: this is a batch/streaming system, not a request/response one. No user waits on a single fetch. We optimize aggregate pages/second and machine efficiency, not p99 of one fetch. A single page taking 30s to time out is fine as long as it does not block others.
  • Politeness (a hard constraint, not a nice-to-have): at most a bounded request rate per host (e.g. 1 request every few seconds per domain, or whatever robots.txt crawl-delay says). Violating this gets us IP-banned and is the fastest way to kill a crawler.
  • Availability: the crawler should run continuously and survive machine failures without losing frontier state or re-crawling everything. It is not user-facing, so brief pauses are acceptable, but losing the frontier or the seen-set is catastrophic (you would restart from seeds).
  • Consistency: loose/eventual everywhere. It is fine if two workers momentarily both consider the same URL; dedup catches it. It is fine if freshness lags. There is no strict correctness contract - a slightly stale or slightly duplicated crawl is acceptable; a banned IP or an infinite loop is not.
  • Durability: high for the frontier and the seen-set (lose them and you lose all crawl progress and politeness memory) and for the stored content (it is the product). The in-flight state of a single worker is disposable - if a worker dies mid-fetch, its URLs go back on the frontier.

Back-of-the-Envelope Estimation (BoE)

Let me ground the design in numbers.

Crawl rate and fetch throughput:

Target corpus:                 10,000,000,000 pages (10B)
Recrawl the whole corpus:      once per month (avg; hot pages far more often)

Pages to fetch per month = 10B (one full refresh)
Seconds per month        = 30 * 86,400 = 2,592,000 s

Sustained fetch rate = 10B / 2.59M s ≈ 3,860 pages/sec average
Peak (say 3x)                              -> ~12,000 pages/sec

So ~4K pages/sec average, ~12K peak, sustained, forever. That is the engine’s baseline.

How many worker machines / connections:

Avg fetch time (network + slow servers): ~500ms wall clock per page
A single worker with C concurrent connections does C / 0.5s pages/sec.
With 500 concurrent connections per worker:  ~1,000 pages/sec/worker.

Workers needed for 4K/sec avg  ≈ 4  (but politeness spreads load,
  and we want headroom + geographic spread), so realistically dozens
  of fetcher workers, hundreds at peak, most connections idle-waiting
  on slow hosts. The bottleneck is politeness + I/O wait, not CPU.

The insight: fetching is I/O-bound and mostly waiting. You need lots of concurrent connections, not lots of CPU, and politeness caps how many of those connections can point at any one host.

Storage:

Avg page size (compressed HTML):  ~50 KB (raw HTML ~100KB, ~2x gzip)
10B pages * 50 KB = 500 TB for one snapshot of content.

Keep a few historical versions + non-HTML -> low petabytes.
Object storage (S3/GCS-class), not a database.

Metadata per URL (URL, hash, last-crawled, status, priority): ~200 bytes
10B URLs * 200 bytes = 2 TB of URL metadata. Fits in a sharded store.

The seen-set (the dedup memory - this is the interesting number):

Distinct URLs ever discovered: easily 50B+ (the frontier explores far
  more URLs than it keeps; many are dupes, traps, dead links).

Storing 50B URLs as full strings (avg 80 bytes) = 4 TB just to answer
  "have I seen this URL?" - and we must answer it billions of times/day.

A Bloom filter for URL membership:
  50B items at 1% false-positive rate needs ~9.6 bits/item
  = 50B * 9.6 / 8 bytes ≈ 60 GB.  Fits in RAM (sharded).
Compare: 4 TB of strings vs 60 GB Bloom filter -> ~70x smaller.

That 60 GB (vs 4 TB) is the whole reason we reach for a Bloom filter for the seen-set: the exact set is too big to keep hot, and we can tolerate the occasional false positive (skipping a page we have not actually seen).

Bandwidth:

4,000 pages/sec * 100 KB raw = 400 MB/sec ingest average.
Peak ~12K pages/sec * 100 KB = 1.2 GB/sec = ~9.6 Gbps.
Non-trivial but a handful of well-provisioned machines handle it.
Egress (writing to object storage) is similar. Plan network capacity.

Bandwidth is real but manageable. The scarce, design-shaping resources are the per-host politeness budget (you cannot buy your way past a server’s tolerance) and the dedup state (60 GB Bloom filter + content-hash store queried billions of times).

High-Level Design (HLD)

A crawler is a loop, but a distributed, stateful one. The heart is the URL frontier: the store and scheduler that decides which URL each fetcher pulls next, honoring priority and politeness. Everything else feeds it or drains it. Fetchers pull URLs from the frontier, download pages, hand content to storage and links back through dedup into the frontier.

                         ┌──────────────┐
                         │  Seed URLs   │
                         └──────┬───────┘
                                │
                    ┌───────────▼────────────────────────────────┐
                    │              URL FRONTIER                    │
                    │  ┌────────────────┐   ┌──────────────────┐  │
                    │  │ Priority queues│──►│ Per-host queues  │  │
                    │  │ (front, F1..Fn)│   │ (back, B1..Bm)   │  │
                    │  └────────────────┘   └────────┬─────────┘  │
                    │      prioritizer         politeness/host    │
                    └──────────────────────────────┬─────────────┘
                                                    │ pull next URL
                                                    │ (host allowed now?)
              ┌─────────────────────────────────────▼──────────────┐
              │                  FETCHER WORKERS                     │
              │  robots.txt check ─► DNS resolve ─► HTTP GET         │
              │  (each worker = many concurrent connections)         │
              └───────┬───────────────────────────────────┬─────────┘
                      │ raw page                           │ extracted links
            ┌─────────▼─────────┐              ┌───────────▼───────────┐
            │  Content Store    │              │   Link Extractor      │
            │ (object storage,  │              │  parse HTML, get <a>  │
            │  compressed HTML) │              │  normalize URLs       │
            └─────────┬─────────┘              └───────────┬───────────┘
                      │ content hash                       │ candidate URLs
            ┌─────────▼─────────┐              ┌───────────▼───────────┐
            │  Content Dedup    │              │   URL Dedup           │
            │ (simhash/hash of  │              │  Bloom filter +       │
            │  page body seen?) │              │  exact seen-set store │
            └───────────────────┘              └───────────┬───────────┘
                                                           │ new URLs only
                                                           ▼
                                              back into the URL FRONTIER

     ┌──────────────────────────────────────────────────────────────┐
     │  Shared services: DNS resolver+cache, robots.txt cache,        │
     │  trap detector, scheduler (recrawl timing), coordinator        │
     └──────────────────────────────────────────────────────────────┘

Request flow - one page, end to end (the crawl loop):

  1. A fetcher worker asks the URL frontier for the next URL. The frontier hands back a URL whose host is currently allowed to be hit (politeness satisfied) and that has the highest available priority.
  2. The worker checks the robots.txt cache for that host (fetching and caching robots.txt if it is not cached), and confirms the URL is allowed. If disallowed, drop it.
  3. The worker resolves the host via the DNS resolver + cache (DNS is a hidden bottleneck at this scale, covered later), opens an HTTP connection, and does the GET.
  4. On a successful response, the worker writes the raw, compressed page to the content store (object storage) and records metadata (URL, status, fetch time, content hash).
  5. Content dedup: hash/simhash the page body and check whether we have already stored identical/near-identical content under another URL. If it is a dup, we still record the URL mapping but do not re-process or re-store the body.
  6. The link extractor parses the HTML, pulls out all hyperlinks, and normalizes each (resolve relative URLs, strip fragments, canonicalize).
  7. Each candidate URL goes through URL dedup: check the Bloom filter (and, on a Bloom hit, the exact seen-set) for “have we already discovered this?” New URLs are marked seen and enqueued back into the frontier with a computed priority. Known URLs are dropped.
  8. The scheduler independently decides when already-crawled pages are due for a recrawl and re-injects them into the frontier at their scheduled time.

Failure handling in the loop: if a fetch times out or errors, the worker releases the host lease and either retries later (with backoff) or marks the URL dead after N failures. If a worker dies mid-fetch, its leased URLs time out and return to the frontier - nothing is lost because the frontier state is durable, not in the worker.

The key architectural insight: the frontier is the brain, workers are dumb muscle. Workers just fetch what they are told; all the intelligence - what to crawl next, how to stay polite, how to prioritize, how not to repeat - lives in the frontier and the dedup/scheduler services around it. That separation is what lets you scale fetchers horizontally without any of them needing global knowledge.

Component Deep Dive

Four hard parts: (1) the URL frontier with prioritization and politeness, (2) dedup of both URLs and content at scale, (3) distributed crawling and coordination, and (4) trap avoidance. I will walk each from naive to scalable.

1. The URL Frontier: Priority + Politeness in One Structure

The frontier answers one question billions of times: “what should the next fetcher fetch?” The answer must honor two competing goals at once - crawl high-priority pages first, and never hit any single host too fast. Those goals fight each other, and resolving that fight is the crux of the whole design.

Approach A: One global FIFO queue (the naive instinct)

A single queue. Workers pop the front, push discovered links to the back. Classic BFS.

[ url1, url2, url3, url4, ... ]  <- workers pop front, push back

Where it breaks - two ways, both fatal:

  • No politeness. A popular page links to 200 pages on the same host. They all land near each other in the queue, so a burst of workers pops them and hammers that one host with 200 near-simultaneous requests. The host rate-limits or bans us. A single FIFO has no notion of “slow down for this host,” and per-host rate is the hard constraint we cannot violate.
  • No priority. A brand-new spam page and a frequently-updated news homepage sit in the same line, first-come-first-served. We waste our finite crawl budget on junk while important pages wait.

A single queue conflates “order discovered” with “order to crawl,” and those must be different.

Approach B: Split priority from politeness (two-layer frontier)

The standard solution (the Mercator design) is a two-stage queue: a front set of queues by priority, and a back set of queues by host.

FRONT: priority queues            BACK: per-host queues
  F1 (highest) ─┐                   B_hostA: [u,u,u]  ← one host per queue
  F2           ─┼─► router ──►      B_hostB: [u,u]
  F3           ─┤   (assigns to     B_hostC: [u,u,u,u]
  ...          ─┘    a host queue)  ...
  Fn (lowest)  ─┘                   each back queue = FIFO for ONE host

A "host heap" tracks, per back-queue, the earliest time that host may
be hit next (now + crawl_delay). Workers pull from the host whose
next-allowed time is soonest and has passed.
  • Front queues (priority): when a new URL is enqueued, a prioritizer assigns it to one of N priority bands (F1 highest … Fn lowest) based on a computed score (PageRank-ish, update frequency, domain trust, depth). Higher bands are drained preferentially.
  • Router: moves URLs from front queues into back queues, one back queue per active host, biased toward higher-priority front queues so important URLs reach a host queue sooner.
  • Back queues (politeness): exactly one host per back queue, FIFO within it. A min-heap of hosts keyed by “next allowed fetch time” lets a worker instantly find a host that is due. When a worker takes a URL for host H, H’s next-allowed time is pushed to now + crawl_delay(H), so no other worker will pull H until then. This structurally enforces politeness - a host physically cannot be pulled faster than its delay.

This solves both problems: priority is honored by the front, politeness is guaranteed by the back plus the host-timing heap. A worker’s operation becomes: pop the host-heap for a host that is due -> take the front URL of that host’s back queue -> fetch -> reschedule the host.

Where it strains: it is now a stateful, in-memory structure holding billions of URLs. That does not fit on one machine, and if the machine dies you lose the frontier. That pushes us to the distributed version.

Approach C: The distributed, durable frontier (the fix)

Partition the frontier across many machines, and shard by host so that all URLs for a given host live on the same frontier partition:

  • Shard key = hash(host). Every URL for example.com routes to the same frontier shard. This is essential: politeness is a per-host property, so all of a host’s URLs and its next-allowed-time must live together, or two shards would independently schedule the same host and blow the rate limit. Sharding by host makes politeness enforceable locally with no cross-shard coordination.
  • Durability. Back the queues with a durable, append-friendly store (e.g. a partitioned log like Kafka per priority band, or a persistent queue store) plus a metadata DB for host timing, so a frontier node can crash and recover its queues rather than restarting from seeds.
  • Leasing, not deleting. When a worker takes a URL, the frontier leases it with a timeout instead of removing it. If the worker finishes, it acks and the URL is removed; if the worker dies, the lease expires and the URL becomes available again. This gives at-least-once crawling (dedup makes the rare double-crawl harmless) and no lost work.

Now the frontier scales horizontally (add shards), survives failures (durable + leases), and keeps politeness correct (host-sharded so each host is scheduled by exactly one owner).

Frontier design Politeness Priority Scales Survives crash Verdict
Single global FIFO none (bans you) none no no Rejected
Front/back two-layer (1 machine) yes yes no no Right idea, wrong scale
Host-sharded distributed + leases yes (local per host) yes yes yes The design

2. Dedup: URLs and Content at Tens of Billions

Two different dedup problems, both essential. URL dedup stops us from re-enqueuing the same link the web points at millions of times. Content dedup stops us from storing/processing the same page body served under many different URLs (mirrors, session IDs, ?utm_source= noise). Skip either and you drown in redundant work.

URL dedup: naive -> Bloom filter

Naive: a database SELECT per candidate URL. For every extracted link, SELECT 1 FROM seen WHERE url = ?. Correct, but we extract hundreds of thousands of links per second, and this is a random point lookup against a table of tens of billions of rows. The DB becomes the bottleneck instantly; disk-seek-per-URL cannot keep up.

Naive-plus: a giant in-memory hash set of URL strings. O(1) lookups, but the BoE showed 50B URLs at ~80 bytes each is ~4 TB of RAM. Too expensive to keep hot even sharded.

Better: a Bloom filter as the first-line seen-set. A Bloom filter answers “have I seen this URL?” probabilistically in ~10 bits/URL: ~60 GB for 50B URLs (vs 4 TB), fits in RAM, sharded by URL hash.

  • No false negatives, tunable false positives. If the filter says “not seen,” it is definitely new - enqueue it. If it says “seen,” it is probably seen (small chance of a false positive). A false positive means we wrongly skip a genuinely new URL - an acceptable loss for a crawler (we miss one page out of billions), which is exactly why the probabilistic structure is the right trade here.
  • Flow: extract URL -> normalize -> check Bloom. If absent, it is new: add to Bloom, enqueue to frontier. If present, drop (already seen). This makes the hot-path dedup a couple of in-RAM bit checks, not a disk lookup.
  • When you cannot tolerate the miss: keep an exact seen-set in a sharded key-value store (RocksDB/Cassandra-class) as a second tier, consulted only on a Bloom “hit” to confirm before dropping - trading a lookup on the (rarer) hit path for zero missed pages. Most crawlers accept the tiny false-positive loss and skip this.
  • URL normalization first. Before any dedup, canonicalize: lowercase host, remove default ports, strip URL fragments (#...), sort/strip query params, resolve ./.., decode redundant encodings. Without this, Example.com/A, example.com/a#top, and example.com/a?utm=x look like three URLs and defeat dedup.

Content dedup: same body, different URL

Even after URL dedup, many distinct URLs return identical or near-identical content. Two levels:

  • Exact dup: hash the page body (e.g. SHA-256 or a 64-bit hash). Keep a set of seen content hashes. If a new page’s hash is already present, it is a byte-identical dup (a mirror, a session-URL of the same page) - record the URL -> canonical mapping but do not re-store or re-extract. Cheap and catches exact mirrors.
  • Near-dup: pages that differ only in a timestamp, an ad, or a session token are not byte-identical but are effectively the same. Detect with SimHash (or MinHash): compute a fingerprint such that near-identical documents have fingerprints within a small Hamming distance. Compare a new page’s SimHash against seen ones (bucketed for efficient near-neighbor lookup); if within threshold, treat as a near-dup. This is what stops us from storing a million trivially-different variants of the same templated page.

Content dedup also feeds trap detection (section 4): a host generating endless near-identical pages under different URLs is a strong trap signal.

3. Distributed Crawling and Coordination

One machine cannot do 4K-12K pages/sec sustained, so crawling is distributed across many fetcher workers and many frontier shards. The coordination questions are: how do workers get work without stepping on each other, and how do we avoid a central bottleneck?

Approach A: Central master hands out URLs (naive)

A single coordinator holds the frontier and hands URLs to workers on request.

Where it breaks: the master is a single point of failure and a throughput ceiling - every one of hundreds of thousands of URL-dispatches per second funnels through it, plus it must track per-host politeness globally. It becomes the bottleneck the moment you add workers. Rejected as the primary data path.

Approach B: Host-partitioned frontier, workers pull from their partition

Because the frontier is already sharded by host (section 1), distribution falls out naturally:

  • Each frontier shard owns a disjoint set of hosts and enforces politeness for exactly those hosts locally. No two shards ever schedule the same host, so there is no global politeness lock - the hard coordination problem is eliminated by the shard key, not solved by locking.
  • Fetcher workers pull from frontier shards (many workers per shard for throughput). A worker pulling a URL takes a lease; the shard’s host-heap guarantees the host is not handed out again until its delay passes, even across the many workers hitting that shard.
  • A lightweight coordinator (built on ZooKeeper/etcd) handles only membership and assignment: which frontier shards exist, which are alive, and rebalancing host ranges when nodes join or leave - a low-frequency control-plane job, not on the per-URL data path. This keeps the hot path decentralized while still having a brain for failover.

Consistent hashing for host assignment

Assign hosts to frontier shards via consistent hashing so that when a shard is added or removed, only a small fraction of hosts remap (rather than reshuffling everything). Each host’s URLs, seen-state, and politeness timing move together to the new owner. This keeps rebalancing cheap and preserves the “one owner per host” invariant during scaling events.

DNS: the hidden bottleneck

At thousands of fetches/sec, DNS resolution becomes a real constraint - a naive getaddrinfo per URL adds latency and can overwhelm resolvers. Fixes:

  • A dedicated, caching DNS resolver layer with a large TTL-respecting cache. Most fetches hit hosts we have resolved recently.
  • Prefetch / async DNS so resolution overlaps with other work rather than blocking a fetch thread.
  • Cache negative results (NXDOMAIN) too, so dead hosts do not get re-resolved endlessly.

Geographic distribution

Run fetcher clusters in multiple regions and route a host’s crawl to the geographically closest cluster where sensible - lower latency, and it spreads egress. The frontier sharding can incorporate a region dimension. This is an optimization, not core, but worth stating.

4. Trap Avoidance and Freshness

The open web is adversarial and infinite in places. Two things will wreck a naive crawler: traps (infinite or near-infinite URL spaces) and staleness (a corpus that never refreshes). Both are frontier-adjacent policy problems.

Crawler traps

A trap is any structure that generates unbounded URLs or feeds you garbage forever:

  • Calendar/pagination traps: ?date=2026-07-08 -> “next day” -> ?date=2026-07-09 -> forever. Faceted search (?color=red&size=m&sort=price&page=999) generates a combinatorial explosion of URLs.
  • Session-ID / infinite redirect loops: every link carries a fresh session ID, so the same page looks like infinitely many URLs; or A -> B -> A redirect cycles.
  • Deliberate spider traps: pages that link to auto-generated junk to trap or poison crawlers.

Defenses (layered, because no single one catches all):

  • Max crawl depth per site. Cap how deep from the seed/homepage we will follow links on one host. A calendar going 10,000 days deep gets cut off at depth N. Simple and effective against depth-based traps.
  • Per-host URL budget / quotas. Cap the number of URLs crawled per host per time window. Even an infinite URL space cannot consume more than the budget, so a trap wastes a bounded amount of crawl and then we move on. This is the strongest general defense.
  • URL pattern / duplicate-content detection. If a host produces thousands of near-identical pages (content dedup from section 2 flags them) or many URLs matching a repeating parameter pattern, down-rank or stop crawling that pattern. High near-dup rate = likely trap.
  • Query-parameter limits. Cap URL length and the number of query parameters; canonicalize away known tracking params. Combinatorial faceted URLs blow past these limits and get dropped.
  • Redirect-chain limits. Follow at most N redirects, then give up - kills redirect loops.

The unifying idea: bound everything per host - depth, URL count, redirects, near-dup ratio. A trap can be infinite, but our expenditure on any one host must be finite. That per-host budget lives right next to the frontier’s per-host queue, so it is natural to enforce there.

Freshness and recrawl scheduling

The web changes; a one-shot crawl rots. But recrawling all 10B pages uniformly wastes budget on pages that never change. The fix is adaptive recrawl:

  • Estimate change frequency per page. Track how often a page’s content hash actually changes across crawls. A news homepage changes hourly; an archived PDF never changes.
  • Schedule recrawl proportional to change rate (and importance). Frequently-changing, high-value pages get short recrawl intervals (minutes to hours); static, low-value pages get long ones (weeks to months). This is a direct application of the idea that recrawl frequency should track observed update frequency, weighted by page importance.
  • The scheduler re-injects due URLs into the frontier at their scheduled time, at a priority reflecting importance. It runs continuously alongside the discovery crawl - the frontier interleaves fresh discovery with scheduled recrawls.

This turns the crawler from a batch job into a continuous, self-adjusting refresh loop: spend crawl budget where the web is actually changing and where it matters.

API Design & Data Schema

A crawler is mostly internal, but it has clear interfaces: an admin/control plane, the frontier’s internal contract, and the data stores.

Control-plane API (admin)

POST /api/v1/seeds                 # inject seed URLs / a whole seed list
     { "urls": ["https://a.com", ...], "priority": "high" }
     -> normalized, deduped, enqueued to the frontier

POST /api/v1/crawl/policy          # set/adjust crawl policy
     { "max_depth": 20, "per_host_qps": 0.3,
       "per_host_daily_budget": 50000, "respect_robots": true }

GET  /api/v1/status                # crawl health/metrics
     -> { "pages_per_sec": 4120, "frontier_size": 3.2e9,
          "hosts_active": 1.1e7, "avg_recrawl_lag_hours": 40 }

POST /api/v1/host/{host}/pause     # stop crawling a host (e.g. abuse complaint)

Frontier internal contract (data plane)

frontier.enqueue(url, priority, discovered_from, depth)
   -> normalize, URL-dedup (Bloom), assign priority band, route to host queue

frontier.lease(worker_id)          # worker asks for next fetchable URL
   -> returns a URL whose host is due (politeness satisfied), leased with TTL
   -> { url, host, lease_id, crawl_delay }

frontier.ack(lease_id)             # worker finished (success or terminal fail)
   -> remove URL from queue, advance host next-allowed time

frontier.nack(lease_id, retry_after)   # transient failure, requeue with backoff

Data model - the right store for each job

Four very different stores, because the access patterns are wildly different.

1. URL frontier / queues (durable log + queue store):

frontier_url
  url            STRING       # normalized URL
  host           STRING       # shard/route key (shard by hash(host))
  priority_band  INT          # F1..Fn
  depth          INT          # distance from seed (trap defense)
  discovered_at  INT64
  lease_expiry   INT64 NULL   # set when leased to a worker
  PARTITIONED BY hash(host)   # all of a host's URLs co-located

Backed by a partitioned log (Kafka-style per priority band) for throughput plus a small metadata store for host timing. Not a relational DB - the access pattern is enqueue/lease/ack streaming, not queries.

2. Seen-set (Bloom filter + optional exact store):

In RAM: sharded Bloom filter, ~60 GB total, key = hash(normalized_url)
Optional exact tier (only checked on Bloom "hit"):
  seen_url  (RocksDB/Cassandra)
    url_hash   BINARY(16)  PK    # 128-bit hash of normalized URL
    first_seen INT64
    -- sharded by url_hash; point lookups only

3. Host / politeness metadata (key-value, sharded by host):

host_state
  host              STRING  PK
  next_allowed_at   INT64         # politeness: earliest next fetch time
  crawl_delay_ms    INT           # from robots.txt or default
  robots_rules      BLOB          # parsed robots.txt, cached with TTL
  robots_fetched_at INT64
  urls_crawled_today INT          # per-host budget (trap defense)
  ip_addresses      LIST<STRING>  # cached DNS result
  INDEX (next_allowed_at)         # find hosts that are due

4. Content store (object storage) + page metadata (wide-column):

Object storage:  s3://crawl-corpus/{host}/{url_hash} -> gzipped body

page_meta   (Cassandra/Bigtable-class, wide-column)
  url_hash        BINARY(16)  PK   # partition key
  url             STRING
  content_hash    BINARY(16)       # for exact content dedup
  simhash         BINARY(8)        # for near-dup detection
  http_status     INT
  fetched_at      INT64
  content_length  INT
  change_count    INT              # how many times body changed (recrawl)
  next_recrawl_at INT64            # adaptive freshness schedule
  INDEX (next_recrawl_at)          # scheduler pulls due pages

Why these choices: the frontier is a streaming queue, so a log/queue system, not SQL. The seen-set is a huge membership test, so a Bloom filter, not a table scan. Host state and page metadata are massive key-value/wide-column workloads with point access by host/url_hash, so Cassandra/Bigtable-class stores that shard linearly - not a single relational DB that would bottleneck on 10B+ rows and hundreds of thousands of writes/sec. Content is large blobs read/written whole, so object storage, not a database. No single database anywhere on the hot path - each store is chosen for one access pattern.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Politeness violations (breaks first, gets you banned). A single FIFO or naive parallelism hammers one host and we get IP-banned - the fastest way to kill a crawler. Fix: the host-sharded frontier with per-host back queues and a next-allowed-time heap structurally caps per-host rate; one shard owns each host so no global coordination is needed, and per-host budgets bound total load on any site. Politeness is enforced by the data structure, not by hoping.

2. URL dedup lookups (hundreds of thousands/sec against tens of billions). A DB SELECT per candidate link cannot keep up. Fix: sharded in-RAM Bloom filter (~60 GB for 50B URLs) as the first-line seen-set - a couple of bit checks per URL, no disk. Accept the tiny false-positive loss; add an exact KV tier only if zero missed pages matters.

3. Frontier size and durability (billions of pending URLs). An in-memory single-machine frontier neither fits nor survives a crash. Fix: shard the frontier by hash(host) across many nodes, back queues with a durable partitioned log, and lease URLs so a dead worker’s URLs return automatically. Consistent hashing keeps rebalancing cheap when nodes change.

4. DNS resolution. Per-fetch DNS at thousands of fetches/sec overwhelms resolvers and adds latency. Fix: a dedicated caching DNS layer with TTL-respecting cache, async prefetch, and negative caching. Cache resolved IPs in host_state.

5. Crawler traps (infinite URL spaces). Calendars, faceted search, and session-ID loops can consume unbounded crawl on one host. Fix: bound everything per host - max depth, per-host URL budget, redirect-chain limit, URL length / param caps, and near-dup-ratio detection (via content SimHash) to stop crawling pattern-generating hosts.

6. Hot host / skew. A few giant sites (a social network, a wiki) have vastly more URLs than the long tail, overloading their owning shard. Fix: since one shard owns a host, give hot hosts more resources - allow a very large host to be split into sub-partitions by URL-path prefix while keeping a single politeness scheduler for the host (the scheduler stays the coordination point, storage scales out). Balance shard assignment by measured host size, not by count.

7. Content storage and near-dup bloat. Storing every fetched byte, including a million near-identical templated pages, wastes petabytes. Fix: content dedup - exact-hash to drop byte-identical mirrors, SimHash to detect and drop near-duplicates, and compress everything. Store one canonical copy plus URL -> canonical mappings.

8. Staleness. A one-shot crawl rots; a uniform recrawl wastes budget on static pages. Fix: adaptive recrawl - track per-page change frequency, schedule recrawl proportional to observed change rate and page importance, and let the scheduler continuously re-inject due pages into the frontier. The crawler becomes a continuous refresh loop.

9. Single points of failure. A central master or a single coordinator would cap throughput and risk total stall. Fix: the data path is fully decentralized (host-sharded frontier, stateless workers); only a lightweight coordinator (ZooKeeper/etcd) handles membership and rebalancing off the hot path, and it is itself replicated. Workers and shards fail independently without global impact.

Wrap-Up

The trade-offs that define this design:

  • Politeness as a structural invariant, not a checked rule. We shard the frontier by host and give each host a back queue plus a next-allowed-time schedule, so a host physically cannot be pulled faster than its delay. We give up the simplicity of one global queue to make the hardest constraint (not getting banned) impossible to violate by accident.
  • Probabilistic dedup over exact. A 60 GB Bloom filter answers “seen this URL?” in RAM instead of a 4 TB exact set on disk. We accept an occasional false positive (skipping one genuinely-new page out of billions) in exchange for dedup that keeps up with hundreds of thousands of extracted links per second. For a crawler, that trade is free.
  • Bound everything per host. Depth, URL budget, redirects, near-dup ratio - all capped per host. The web has infinite corners (traps); our spend on any one corner is finite. This single principle handles most trap classes at once and lives naturally beside the per-host frontier queue.
  • Decentralize the data path, centralize only control. The per-URL hot path touches no global lock and no master - host sharding eliminates the coordination instead of solving it. A lightweight coordinator handles only membership and rebalancing. That is what lets fetchers scale to hundreds of machines linearly.
  • Continuous adaptive refresh over one-shot batch. We schedule recrawls by observed change frequency and importance rather than uniformly, trading a bit of scheduling machinery for spending crawl budget exactly where the web is actually changing.

One-line summary: a distributed, host-sharded URL frontier enforces politeness and priority structurally, a Bloom-filter seen-set plus SimHash content dedup kills redundant work at tens of billions of URLs, per-host budgets and depth limits contain traps, and an adaptive recrawl scheduler keeps the corpus fresh - trading exactness and central control for a crawler that fetches billions of pages politely, without repeating itself, and without ever getting stuck.