Reverse image search looks like a search problem, and it is - but not the kind you get from an inverted index of words. A user hands you a picture, not a query string, and asks “where does this appear, and what looks like it.” There are no keywords to match. The only thing you have is pixels, and pixels are the wrong unit: the same photo re-saved as a JPEG, cropped by 5%, watermarked, or scaled to a thumbnail is byte-for-byte a completely different file, yet to a human it is obviously “the same image.” So the core of the problem is not string matching or even byte matching - it is turning an image into a representation where visual similarity becomes geometric closeness, and then finding the closest points to a query among 100 billion of them, in under a second.

That single sentence is the spine of this design. The whole thing decomposes into two hard questions and everything else is plumbing. First: how do you represent an image so that “looks similar” becomes “is nearby in some space” - a perceptual hash for near-identical copies, a learned embedding vector for visual/semantic similarity. Second: how do you search 100 billion of those vectors for the nearest few in sub-second time, when an exact scan of 100B vectors is physically impossible and even a linear scan of one shard is too slow. Get those two right - the representation and the approximate-nearest-neighbour (ANN) index over it - and the rest (crawling the web for images, an ingestion pipeline that vectorizes them, an API, caching) follows. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Search by uploaded image. A user uploads an image (or pastes a URL). The system returns (a) pages on the web where that exact image appears, (b) visually similar images, and (c) a best-guess description/entity (“this is the Eiffel Tower”).
  • Find identical and near-identical copies. The re-compressed, cropped, resized, watermarked, or lightly-edited versions of the query must be found and grouped as “the same image.”
  • Find visually similar images. Not the same photo, but semantically/visually alike - another photo of the same landmark, the same product, the same breed of dog.
  • Rank and return fast. Results ranked by similarity and by the quality/authority of the hosting page, returned in under a second.
  • Index the web’s images. A crawling/ingestion pipeline that discovers images across ~100B images on the web, fetches them, and indexes them for search.

Explicitly out of scope (say this so you own the scope):

  • Training the embedding model. I assume a pre-trained vision embedding model (a CNN/ViT that maps an image to a fixed-length vector such that similar images map to nearby vectors) exists. Designing/training it is an ML problem, not a systems one; I consume it as a black-box embed(image) -> vector.
  • General text-to-image search. “Find pictures of red cars” from a text query is a related but different retrieval problem (text query embedding against image embeddings). Here the query is an image.
  • The web crawler in full. A general-purpose crawler (URL frontier, politeness, dedup, robots.txt) is its own system-design problem. I consume a stream of “here is an image URL found on page P” from an existing crawl pipeline and focus on the image-specific indexing.
  • Face recognition / identifying people. Deliberately excluded - it is a legal, ethical, and privacy minefield and a separate design.
  • Image hosting / thumbnails CDN. I reference a blob store and a CDN for thumbnails but do not design the media-serving stack.

The one decision that drives everything: this is a read-optimized, index-heavy nearest-neighbour system where the unit of work is not “match a keyword” but “map an image to a vector and find its nearest neighbours among 100B vectors.” Unlike a text search engine built on inverted indexes and TF-IDF, there is no vocabulary and no exact term match; the entire difficulty is building and querying an approximate-nearest-neighbour index over high-dimensional vectors at a scale where exact search is impossible. That is why the vectorization step and the ANN index, not a crawler or a SQL store, are the heart of the design.

Non-Functional Requirements (NFR)

  • Scale: 100B images indexed. Assume ~100M reverse-image searches/day from users, with a read-heavy skew (searches vastly outnumber index writes at steady state, though the initial backfill of 100B images is a massive one-time write).
  • Latency: end-to-end query under ~1s (p99), of which the ANN lookup itself must be tens of milliseconds. Embedding the query image is ~50-200ms on an accelerator. The page-annotation/ranking join adds the rest.
  • Availability: 99.9%+ on the query path. This is not life-critical; a failed search can be retried. The index-build path can tolerate lag - an image indexed an hour late is fine.
  • Consistency: eventual consistency. The index reflecting a newly-crawled image minutes-to-hours late is perfectly acceptable. There is no transaction, no “read your write” requirement, no ordering constraint. The query path reads a slightly stale index and that is fine.
  • Durability: the durable source of truth is the mapping image_id -> {source_url, page_urls, hash, metadata} and the raw vectors. The ANN index itself is a derived artifact - it can be rebuilt from the vectors, so it need not be independently durable, only recoverable.
  • Recall vs latency trade-off: because search is approximate, we accept that we may miss a true nearest neighbour occasionally in exchange for speed. Target high recall (e.g. 95%+ of true top-k found) at sub-second latency. This trade is the defining NFR of the whole system.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

The index size (the number that defines the problem):

Images to index: 100,000,000,000 (100B)
Embedding vector: 512 dimensions, float32 = 512 * 4B = 2,048 B/vector

Raw vector storage (uncompressed):
  100B * 2,048B = 2.048 * 10^14 B ≈ 200 TB just for the raw vectors.

200 TB of raw vectors will not fit in the RAM of any single machine, and you cannot afford to scan them linearly per query. Both facts drive the design: compress the vectors, and index them so you never scan all of them.

With Product Quantization (PQ) compressing each vector to ~64 bytes:
  100B * 64B = 6.4 TB.
  Now the compressed index fits across a modest fleet of RAM-heavy nodes.

Query throughput:

100M searches/day / 86,400s ≈ 1,160 searches/sec average.
Peak (5x) ≈ ~6,000 searches/sec.

Only ~6K QPS at peak - modest. The challenge is emphatically not query volume; it is the cost of a single query against 100B vectors. Each query must embed an image and probe a 6.4TB index for nearest neighbours in tens of milliseconds.

Why an exact scan is impossible:

Exact nearest neighbour = compute distance to all 100B vectors per query.
  100B distance computations * ~512 multiply-adds each ≈ 5 * 10^13 ops/query.
Even at 10^12 ops/sec/machine, that is ~50s/query on one machine.
Sharded over 1,000 machines: ~50ms of raw compute - but 1,000 machines
  hammered by EVERY single query does not scale to 6K QPS.

So exact search is off the table. We need an index that lets each query touch a tiny fraction of the 100B vectors - this is what ANN buys us.

Ingestion (the one-time backfill and the steady state):

Backfill: 100B images to embed. At, say, 50K images/sec across a GPU fleet,
  100B / 50,000 = 2,000,000s ≈ 23 days of continuous embedding.
  This is a batch job; it runs once and then trickles.
Steady state: the web adds images continuously; assume ~1B new/day
  = 1B / 86,400 ≈ ~11,600 images/sec to embed and index. Very manageable.

Storage for metadata:

Per image: image_id 8B, source_url ~100B, page_urls (list) ~300B,
           phash 8B, dims/format ~20B, crawl_ts 8B ≈ ~450B, round to 512B.
  100B * 512B = 51.2 TB of metadata (durable, in a sharded KV store).
Thumbnails (for the results grid): ~10KB each in blob store + CDN:
  100B * 10KB = 1 PB of thumbnails. Blob storage, served via CDN, not our concern to build.

Bandwidth: query ingress is trivial (~6K uploads/sec * ~500KB ≈ 3 GB/s at peak, edge-handled and downscaled immediately). The heavy bandwidth is internal, during backfill, fetching 100B source images once.

The takeaways: 200TB of raw vectors compressed to ~6.4TB with quantization; exact search is physically impossible (~50s/query), so the whole design hinges on an ANN index that touches a fraction of a percent of the vectors per query; query volume is low (~6K QPS) but per-query cost is the enemy; and ingestion is a 23-day one-time GPU batch plus an easy ~12K/sec trickle. The vector index, not throughput, is the design driver.

High-Level Design (HLD)

The architecture splits cleanly into two paths that share nothing on the hot path: an offline ingestion path (crawl -> fetch -> embed -> hash -> write to the durable store -> build/update the ANN index) and an online query path (upload -> embed the query -> probe the ANN index -> fetch metadata for the hits -> rank -> return). The ANN index is the artifact the ingestion path produces and the query path consumes.

                        OFFLINE INGESTION PATH
  ┌───────────┐   image URLs   ┌──────────────┐   raw image  ┌──────────────┐
  │ Web Crawl  │ ─────────────▶ │ Image Fetcher │ ───────────▶ │  Embedder     │
  │ (upstream) │                │ + dedup(phash)│              │  (GPU fleet)  │
  └───────────┘                └──────┬───────┘              └──────┬───────┘
                                       │ phash                       │ 512-d vector
                              ┌────────▼─────────────────────────────▼────────┐
                              │  Ingestion Writer                              │
                              │   - write image_id -> {urls, phash, meta}      │
                              │   - append vector to vector store              │
                              │   - enqueue for index build                    │
                              └───────┬───────────────────────┬────────────────┘
                                      │                        │
                            ┌─────────▼────────┐     ┌─────────▼──────────────┐
                            │ Metadata Store    │     │  ANN Index Builder      │
                            │ (sharded KV,      │     │  (builds IVF/HNSW+PQ    │
                            │  image_id keyed)  │     │  shards from vectors)   │
                            └─────────┬─────────┘     └─────────┬──────────────┘
                                      │                         │ ships shards
  ============================================================ │ =============
                        ONLINE QUERY PATH                       ▼
  ┌────────┐  upload   ┌──────────────┐  vector  ┌───────────────────────────┐
  │ User    │ ────────▶ │ Query Service │ ───────▶ │   ANN Search Fleet         │
  │ (image) │ ◀──────── │ (embed query, │ ◀─────── │  (N index shards, each     │
  └────────┘  results   │  scatter/gath,│  top-k   │   holds ~1/N of vectors)   │
                        │  rank)        │  per     └───────────────────────────┘
                        └──────┬───────┘  shard
                               │ hydrate hits (image_id -> urls, page, thumb)
                        ┌──────▼─────────┐     ┌──────────────────────────────┐
                        │ Metadata Store  │     │ Thumbnail Blob Store + CDN    │
                        └────────────────┘     └──────────────────────────────┘

Query flow (the hot path):

  1. The user uploads an image to the Query Service at the edge, which immediately downscales/normalizes it (fixed size, color space) - a thumbnail is all the model needs, and it caps ingress bandwidth.
  2. The Query Service computes the query embedding by calling the same Embedder model used at index time (a GPU/accelerator inference call, ~50-200ms). It also computes the query’s perceptual hash for the exact/near-duplicate lane.
  3. It scatters the query vector to all ANN Search shards (the index is sharded; each shard holds ~1/N of the 100B vectors). Each shard runs an ANN probe over its local vectors and returns its local top-k candidates with distances.
  4. The Query Service gathers the per-shard top-k, merges them into a global top-k by distance, and in parallel checks the phash lane for exact/near-identical copies (a separate cheap hash lookup).
  5. It hydrates the winning image_ids from the Metadata Store (source URL, the pages where the image appears, thumbnail URL) and ranks them - combining visual similarity (vector distance), duplicate confidence (phash), and page authority.
  6. It returns the ranked results (exact-match pages, then visually similar images) with thumbnails served from the CDN.

Ingestion flow (offline):

  1. The upstream crawler emits “image at URL U found on page P.” The Image Fetcher downloads U, normalizes it, and computes its perceptual hash.
  2. A phash dedup check: if this image (or a near-identical one) is already indexed, we do not re-embed - we just append P to that image’s page_urls. This collapses the web’s massive duplication (the same stock photo on 10,000 pages) into one vector.
  3. If new, the Embedder (GPU fleet, batched) computes the 512-d vector. The Ingestion Writer persists image_id -> {source_url, page_urls, phash, dims, ts} to the Metadata Store, appends the vector to the vector store, and enqueues it for the index.
  4. The ANN Index Builder periodically (or continuously) folds new vectors into the ANN index shards and ships updated shards to the Search Fleet.

The key insight: the expensive, batchy work (crawling, embedding, index building) is entirely offline and eventually consistent; the online path only embeds one query and probes a pre-built index. The two paths meet only at the ANN index artifact and the metadata store, so we can scale, rebuild, and fail them independently. Nothing on the query path writes anything durable.

Component Deep Dive

The hard parts, naive-first then evolved: (1) representing an image so similarity becomes geometry, (2) searching 100B vectors in sub-second time - the core problem, (3) sharding and updating the vector index, (4) the exact-duplicate lane and web-scale deduplication.

1. Representing an image so “similar” becomes “nearby”

Naive approach: compare raw pixels / file bytes. Store each image’s bytes; to match a query, compare byte-for-byte, or diff pixel arrays.

Where it breaks:

  • Byte comparison finds only literally-identical files. Re-save the same JPEG at a different quality and every byte changes. Crop 2 pixels, add a watermark, resize to a thumbnail - all “the same image” to a human, all completely different bytes. Byte match has near-zero recall.
  • Pixel diffing does not scale and is not robust. Comparing pixel arrays is O(pixels) per pair and still brittle to resize, rotation, and lighting. And you cannot pixel-diff a query against 100B images.

First evolution: perceptual hashing (pHash). Shrink the image to a tiny grayscale thumbnail (say 32x32), run a DCT, keep the low-frequency coefficients, and threshold them into a 64-bit hash. Similar images produce hashes with small Hamming distance. This is robust to re-compression, resizing, and minor edits, and comparison is a cheap XOR + popcount.

pHash is excellent for near-duplicate detection - “is this the same photo, possibly re-encoded.” But it has a ceiling:

  • It captures appearance, not semantics. Two different photos of the same landmark, or two photos of the same product from different angles, have very different pHashes. pHash cannot find “visually/semantically similar but not the same image.”
  • Hamming search over 64-bit hashes at 100B scale still needs its own index (multi-index hashing / LSH on Hamming space), though it is cheaper than vector search.

The answer: a two-representation model - perceptual hash for the exact/near-duplicate lane, and a learned embedding vector for the visual-similarity lane.

  • Learned embedding. A pre-trained CNN or Vision Transformer maps an image to a fixed-length vector (say 512-d) such that visually and semantically similar images land close together under cosine/L2 distance. This is what lets us return “other photos of the Eiffel Tower,” not just re-saved copies of one photo. It is the primary representation for similarity search.
  • Perceptual hash. Runs alongside, cheaply, purely to catch exact and near-identical copies with high precision and to power “find where this exact image appears on the web,” which is a huge fraction of real reverse-image-search intent.
On both index and query:
  norm_img  = downscale + normalize(image)         # fixed size, color space
  vector    = embed(norm_img)                       # 512-d float, similarity lane
  phash     = perceptual_hash(norm_img)             # 64-bit, duplicate lane
Store both; search both; merge results.

We run the two lanes in parallel and merge: the phash lane gives high-precision “same image, here are its pages,” the vector lane gives high-recall “here is what looks like it.” This dual representation is the foundation everything else sits on.

2. Searching 100B vectors in sub-second time (the core problem)

Given the query vector, find its nearest neighbours among 100B vectors. This is where the design lives or dies.

Naive approach: exact brute-force k-nearest-neighbour. For each of the 100B stored vectors, compute the distance to the query, keep the smallest k.

Where it breaks:

  • Compute. As the BoE showed, one exact query is ~5 * 10^13 ops - ~50s on a single machine. Even sharded across 1,000 machines, every query lighting up all 1,000 machines cannot sustain thousands of QPS. Exact search is physically off the table.
  • Memory. 200TB of raw float32 vectors will not sit in RAM affordably, and reading them from disk per query is hopeless.
  • The curse of dimensionality. Classic spatial trees (kd-trees, R-trees) degrade to near-linear scan in high dimensions (512-d). They do not save you.

First evolution: Locality-Sensitive Hashing (LSH). Hash vectors with functions that make nearby vectors collide with high probability (e.g. random hyperplane projections). At query time, hash the query, look only in the matching buckets, and rank the (few) candidates there exactly.

LSH is a real improvement - it turns search into a bucket lookup plus a small exact rerank - but at 100B scale it strains:

  • Memory and tuning. Getting high recall needs many hash tables, each a copy of the pointers; the tables get large and the recall/memory trade-off is fiddly.
  • Uneven buckets. Real data clusters, so some buckets are huge (still a big scan) and some empty. Recall is inconsistent.

Second evolution: IVF (inverted file / coarse quantization). Cluster all vectors into, say, 1,000,000 centroids with k-means. Each vector is assigned to its nearest centroid, forming an inverted list per centroid. At query time, find the query’s few nearest centroids (nprobe, e.g. 32) and search only the vectors in those lists.

Build: centroids = kmeans(sample_of_vectors, K=1,000,000)
       for each vector v: list[nearest_centroid(v)].append(v)
Query: probe_lists = top_nprobe centroids nearest to query   # e.g. 32 of 1M
       candidates  = concat(list[c] for c in probe_lists)     # ~0.003% of data
       return exact-rerank top-k over candidates

This is the breakthrough: instead of scanning 100B vectors we scan ~nprobe/K of them - with K=1M and nprobe=32, about 0.003% of the data, i.e. ~3M vectors instead of 100B. nprobe is the recall/latency knob: more probes = higher recall, more compute.

Compressing the vectors: Product Quantization (PQ). Even the candidate vectors need to be compact. PQ splits each 512-d vector into, say, 8 subvectors of 64 dims, quantizes each against a small per-subspace codebook (256 entries = 1 byte), so a vector becomes 8 bytes… or with more subquantizers, ~64 bytes for good accuracy. Distances are computed approximately from precomputed lookup tables. This is what turns 200TB into ~6.4TB and keeps the index in RAM.

Third evolution: HNSW (graph-based ANN) for the in-memory tier. HNSW builds a navigable small-world graph: each vector links to its near neighbours across multiple layers, and search greedily walks the graph toward the query. It gives excellent recall at very low latency and is often the best in-memory ANN structure - but it is memory-hungry (stores the graph edges) and harder to update/shard at 100B scale than IVF.

The composite answer: IVF for coarse partitioning + PQ for compression, with HNSW as the fast centroid selector, sharded across a fleet. In practice at this scale you combine them - IVF partitions and PQ compresses so the whole thing fits in RAM and each probe is cheap, and a graph/HNSW structure can accelerate finding the right centroids or serve as the fine index inside a shard.

Per query per shard:
  1. select nprobe nearest centroids     (small, fast)
  2. scan the PQ codes in those lists     (~few M codes, table-lookup distances)
  3. return local top-k with distances
MethodQuery costMemoryRecallUpdate costFit at 100B
Exact brute forceO(N) - 50s200TB rawperfecttrivialimpossible
LSHbucket + rerankhigh (many tables)tunable, uneveneasystrained
IVF + PQO(nprobe/K * N)~6.4TBhigh, tunablemoderateprimary design
HNSWvery lowhigh (graph edges)very highrebuild-ishgreat in-memory, hard to shard

The elegant part: we never search 100B vectors. IVF confines each query to a fraction of a percent of the data by only probing the few nearest clusters, PQ compresses the vectors so those candidates live in RAM and distances are table lookups, and nprobe is a single dial that trades recall against latency. That collapse from 100B to ~millions of candidates per query is what makes sub-second search possible at all.

3. Sharding and updating the vector index

One machine cannot hold 6.4TB in RAM or serve every query. The index must be sharded, and it must absorb ~12K new vectors/sec without a full rebuild.

Naive approach: one giant index, rebuilt nightly. Keep the whole IVF+PQ index on a big machine (or a few), rebuild it from scratch each night to fold in the day’s new images.

Where it breaks:

  • It does not fit and does not scale reads. 6.4TB in RAM and 6K QPS is beyond one node; you need a fleet.
  • Nightly rebuild is too slow and too coarse. Re-clustering 100B vectors is a days-long job; you cannot do it nightly, and new images would be invisible for up to a day.
  • k-means centroids drift. As the corpus grows, the original centroids become unbalanced (hot clusters), degrading recall.

The answer: shard the index by document (random/hash partition), scatter-gather queries, and update via segment merges rather than full rebuilds.

  • Document-partitioned sharding. Partition the 100B vectors randomly (by hash(image_id)) across N shards, each holding ~100B/N vectors with its own IVF+PQ index in RAM. This is the standard search-sharding trick: every query must hit every shard (each shard might hold a true nearest neighbour), so we scatter the query to all N shards and gather their local top-k. Latency is bounded by the slowest shard, and total memory spreads across the fleet. Contrast with term-partitioning (used in text search), which does not apply here because there are no terms - every vector is a potential neighbour of every query.
  • Replicate each shard. Each shard is replicated R times for availability and read throughput; a query fans out to one replica per shard. Query QPS scales by adding replicas; corpus size scales by adding shards.
  • Segment-based incremental updates (LSM-style). Rather than mutating a monolithic index, each shard is a set of immutable segments. New vectors accumulate in a small, fresh, easily-searched segment (or an HNSW graph for the recent tier). Queries search all segments in the shard and merge. Periodically a background compaction merges small segments into larger IVF+PQ segments and re-quantizes, exactly like an LSM tree compacts SSTables. This gives near-real-time indexing (new image searchable within minutes) without ever blocking on a full rebuild.
Shard layout:
  shard_i = [ big_segment(IVF+PQ, ~months old, immutable),
              medium_segments...,
              hot_segment(recent, HNSW or flat, fast to append) ]
  ingest: append vector to hot_segment
  query : search every segment, merge local top-k
  compact: background-merge small -> large, re-run PQ, atomically swap in
  • Periodic global recluster (rare). The IVF centroids are refreshed occasionally (weekly/monthly) from a fresh sample, as an offline job producing a new index generation that is shipped to the fleet and hot-swapped. This is decoupled from ingestion, so the drift problem is handled without touching the hot path.
  • Derived, not durable. The index shards are rebuildable from the durable vector store, so a lost shard replica is re-derived, not recovered from a backup of the index itself.

The lesson: at 100B vectors you document-partition the index and scatter-gather every query (there is no way to prune shards, since any shard may hold a neighbour), you replicate for QPS and availability, and you make ingestion cheap with LSM-style immutable segments and background compaction so new images appear in minutes without a days-long rebuild.

4. The exact-duplicate lane and web-scale deduplication

A huge fraction of reverse-image-search intent is “where does this exact image appear,” and the web is drowning in duplicates - the same stock photo on tens of thousands of pages. Both facts demand a dedicated duplicate path.

Naive approach: treat every crawled image as unique; rely on the vector index to cluster copies at query time. Embed and index every one of the 10,000 copies of a stock photo separately.

Where it breaks:

  • Index bloat. Indexing 10,000 near-identical vectors wastes 10,000x the memory for one visual concept and pollutes results (the top-k fills with copies of the same image instead of diverse pages).
  • Wasted embedding compute. Re-embedding 10,000 identical images burns GPU time on the most expensive step for zero new information.
  • Weak “exact match” precision. Vector distance can conflate “very similar but different” with “identical”; users asking “is this my photo, and who is using it” want a precise same-image answer, not a fuzzy neighbour.

The answer: a perceptual-hash dedup gate at ingestion that collapses copies into one indexed image with many page URLs, plus a phash index for the exact lane at query time.

  • Dedup at ingest. When the Fetcher pulls an image, compute its pHash. Look it up in a phash index. If an existing image is within a small Hamming threshold, this is a duplicate: do not re-embed or add a new vector - just append the new page_url to the existing image_id’s list. One vector, one embed, N pages.
on_fetch(url, page):
    ph = phash(image)
    dup = phash_index.query(ph, hamming <= T)     # near-duplicate lookup
    if dup:
        metadata[dup.image_id].page_urls.append(page)   # cheap, no embed
    else:
        v = embed(image); id = new_id()
        metadata[id] = {source_url:url, page_urls:[page], phash:ph, ...}
        vector_store.append(id, v); phash_index.add(ph, id)
  • Searching the phash index (Hamming ANN). A linear Hamming scan over 100B 64-bit hashes per lookup is ~100B XOR+popcount - too slow. Use multi-index hashing: split the 64-bit hash into b bands (say 4 bands of 16 bits); two hashes within Hamming distance T must match exactly in at least one band (pigeonhole). Index each band separately; a lookup probes only the candidates sharing a band, then verifies full Hamming distance. This turns near-duplicate search into a handful of exact bucket lookups.
  • Two lanes, merged at query time. The query runs both: the phash lane returns high-precision exact/near-identical matches (“this exact image appears on these pages”), and the vector lane returns high-recall visual similars. The ranker presents exact matches first (with their page list), then visually similar images. This split is exactly why we keep two representations.
  • Grouping in results. Even within the vector lane, we cluster returned copies (by phash) so the results grid shows distinct images, not 20 copies of one.

The point: web-scale duplication is not noise to tolerate - it is collapsed at ingestion by a phash gate so we embed and index each distinct image once while tracking all the pages it appears on, and a multi-index-hashing phash index serves the high-precision “find this exact image” intent that vector similarity alone handles poorly.

API Design & Data Schema

The query path is a single upload-and-search call; ingestion is internal.

Query API (client-facing)

POST /api/v1/search/image
  body: multipart image file  OR  { "image_url": "https://..." }
  query params: ?limit=20&include=exact,similar
  -> 200 {
       "query_id": "q_9f3",
       "exact_matches": [
         { "image_id":"img_88", "thumbnail_url":"https://cdn/.../88.jpg",
           "pages":[ {"url":"https://site-a/...","title":"..."},
                     {"url":"https://site-b/...","title":"..."} ],
           "match":"exact", "confidence":0.99 } ],
       "similar_images": [
         { "image_id":"img_12", "thumbnail_url":"...", "source_url":"...",
           "similarity":0.87, "page":{"url":"...","title":"..."} }, ... ],
       "best_guess": { "label":"Eiffel Tower", "entity_id":"kg:/m/02j81" }
     }

GET /api/v1/image/{image_id}
  -> metadata + full page list for one indexed image (drill-down)

Internal ingestion API

POST /internal/ingest/image
  body: { "source_url":"https://...", "found_on_page":"https://page..." }
  -> { "image_id":"img_88", "action":"deduped" | "indexed" }
  // Fetcher/Writer path: fetch -> phash -> dedup-or-embed -> persist

Data stores - the right tool per plane

1. Vector store (durable source of truth for vectors) - sharded blob/columnar, by image_id. The raw (or PQ-compressed) vectors, from which the ANN index is (re)built. Append-only, read in bulk by the index builder, not on the query hot path.

vectors  (partition = hash(image_id))
  image_id   BIGINT   KEY
  vector     BINARY   -- 512-d float32 (2KB) or PQ code (64B)
  created_at TIMESTAMP
  Shard by: hash(image_id). Append-only; index is derived from this.

2. Metadata store (durable) - sharded NoSQL KV, by image_id. The image_id -> {source_url, page_urls[], phash, dims, format, crawl_ts} mapping. Read on the hot path to hydrate hits. Read-mostly, no cross-key transactions, shards cleanly on image_id - NoSQL is the right call; relational buys nothing.

Table: images              (partition key = image_id)
  image_id    BIGINT   PARTITION KEY
  source_url  STRING
  page_urls   LIST<STRING>   -- every page the image was found on (grows on dedup)
  phash       BIGINT (64-bit)
  width/height/format  ...
  crawl_ts    TIMESTAMP
  Query: hydrate a hit = one point read by image_id.

3. ANN index (derived, in-memory) - the Search Fleet. IVF+PQ (plus HNSW hot tier) segments, sharded by hash(image_id), replicated. Not independently durable - rebuilt from the vector store. This is what queries actually probe.

Per shard (in RAM):
  centroids[K]                      -- IVF coarse quantizer
  inverted_lists[centroid] -> [ (image_id, pq_code) ]   -- PQ-compressed
  segments: [big immutable ... hot recent], background-compacted

4. pHash index (derived, in-memory/KV) - multi-index hashing. 64-bit perceptual hashes split into bands for near-duplicate Hamming lookup, keyed for the exact lane and for ingest dedup.

phash_bands  (b tables, one per 16-bit band)
  band_table_i[ band_bits ] -> [ image_id, ... ]
  lookup(ph): union candidates across bands, verify full Hamming <= T

5. Thumbnail blob store + CDN. image_id -> thumbnail.jpg, ~1PB, served via CDN for the results grid. Referenced, not designed here.

6. Query cache - Redis. Keyed by the query image’s phash (identical uploads recur - people search the same viral image). Caches the full result set with a short TTL.

Why NoSQL/in-memory throughout and no relational store on the hot path: every hot-path operation is either an ANN probe (a custom in-memory index) or a point read of an image’s metadata by image_id (a KV lookup). There is no transaction, no join across users, no ordering constraint, no “assign exactly one.” The corpus is enormous, read-mostly, and shards trivially by image_id. Relational’s strengths (ACID transactions, joins, foreign keys) are exactly the things this system never needs, and its costs (single-writer bottlenecks, hard horizontal scaling) are exactly what would kill it. So: a sharded KV metadata store, a custom sharded vector index, and a phash index - eventual consistency everywhere, because an image indexed minutes late is fine.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Exact search is impossible (breaks before you even start). A brute-force k-NN over 100B vectors is ~50s/query. Fix: approximate nearest neighbour with IVF + PQ. IVF confines each query to the few nearest clusters (~0.003% of vectors), PQ compresses 200TB to ~6.4TB so it lives in RAM with table-lookup distances. nprobe dials recall against latency. This is the single optimization that makes the system exist.

2. The index does not fit on one machine. 6.4TB in RAM and every query touching all of it will not run on one node. Fix: document-partition the index across N shards (by hash(image_id)), scatter-gather each query to all shards, merge the per-shard top-k. Memory and per-query compute spread across the fleet; latency is the slowest shard.

3. Read QPS and tail latency. 6K QPS scatter-gathered to every shard means each shard sees full query load, and one slow shard drags the whole query. Fix: replicate each shard R times (QPS scales with replicas), route each fan-out to one replica per shard, and use hedged requests / speculative retries to a second replica when a shard is slow, so the p99 is not held hostage by one straggler.

4. Ingesting new images without a full rebuild. Re-clustering 100B vectors nightly is a days-long job; new images would be invisible for a day. Fix: LSM-style immutable segments - new vectors land in a small hot segment (searchable in minutes), background compaction merges segments into IVF+PQ, and a rare offline global recluster refreshes centroids as a new index generation that is hot-swapped. Ingestion never blocks on a rebuild.

5. Web-scale duplication bloating the index. The same stock photo on 10,000 pages would be 10,000 vectors. Fix: phash dedup gate at ingestion collapses copies into one indexed image with a growing page_urls list - one embed, one vector, N pages. Cuts index size, embedding compute, and result pollution.

6. Embedding throughput (the ingestion bottleneck). Embedding 100B images is a 23-day GPU batch; embedding is the most expensive step. Fix: batch inference on a GPU/accelerator fleet, dedup before embedding (skip the duplicates entirely), and treat backfill as a throttled batch job decoupled from the query path. Steady-state ~12K/sec is trivial for the same fleet.

7. Hot clusters / skewed IVF lists. Real images cluster (lots of sky, lots of faces, lots of white-background product shots), so some IVF lists are enormous - probing them is slow, and recall is uneven. Fix: balanced/hierarchical clustering and a larger K so no single list dominates; split hot lists into sub-lists; cap the candidates scanned per list and rely on multiple probes for recall. Monitor list-size distribution and recluster when it skews.

8. The scatter-gather fan-out cost. Every query hits every shard; more shards means more fan-out, more chances for a straggler. Fix: keep shard count as low as memory allows (fatter shards, more RAM per node), use a two-level fan-out (a query aggregator per rack that fans out locally and returns a partial merge) to bound the coordinator’s fan-out degree, and cache aggressively.

9. Repeated/viral queries. The same viral image gets searched millions of times. Fix: query cache keyed by the query’s phash - identical and near-identical uploads hit the cache and skip embedding and the ANN probe entirely. A short TTL keeps results fresh as the index updates.

10. Single points of failure and durability. Fix: the ANN index is derived from the durable vector store, so a lost shard/replica is re-derived, not restored. The vector store and metadata store are replicated and multi-region. The query path is stateless behind a load balancer. No single node whose loss stops search; at worst a shard replica rebuilds while others serve.

Wrap-Up

The trade-offs that define this design:

  • Approximate over exact, deliberately. Exact nearest-neighbour is physically impossible at 100B vectors, so we accept missing a true neighbour occasionally (95%+ recall) in exchange for tens-of-milliseconds search. Recall-vs-latency, dialed by nprobe, is the defining trade of the whole system.
  • Two representations over one. A perceptual hash for high-precision exact/near-duplicate matches and a learned embedding for high-recall visual similarity - because neither alone serves both “where does this exact image appear” and “what looks like it.”
  • Compress and partition over store-and-scan. Product Quantization turns 200TB into 6.4TB of RAM-resident codes, and IVF confines each query to a fraction of a percent of the data - together they make a sub-second probe of 100B vectors possible.
  • Offline heavy, online light. All the expensive, batchy work (crawl, embed, cluster, build) is offline and eventually consistent; the online path only embeds one query and probes a pre-built, derived index. The two meet only at the index artifact and the metadata store, so they scale and fail independently.
  • Dedup at the source over dedup at query time. A phash gate collapses the web’s massive duplication into one vector per distinct image with a page list, saving embedding compute, index memory, and result quality all at once.

One-line summary: a read-optimized nearest-neighbour system where each image is turned into a perceptual hash and a learned embedding once at ingestion, the web’s duplicates are collapsed by a phash gate, the 100B compressed vectors live in a document-sharded IVF+PQ ANN index that every query scatter-gathers in tens of milliseconds, and a two-lane merge of exact (phash) and similar (vector) results is ranked and returned in under a second - approximate, eventually consistent, and derived-not-durable throughout.