Design Google Translate - System Design

Google Translate looks like a text box: you type “hello,” you get “hola,” done. That intuition survives one short sentence. The moment you ask “hello” to become fluent Japanese, or you paste a 40-page PDF, or two people hold a live spoken conversation across a language barrier, the box turns into a distributed system whose real job is to run large neural machine translation (NMT) models on GPUs/TPUs at sub-second latency, across 100+ languages meaning 10,000+ possible language pairs, while translating billions of words a day - and to do that without lighting a datacenter’s worth of accelerators on fire. The interesting part is not “map word A to word B.” It is: a stateless serving tier that turns arbitrary input (a phrase, a whole document, a web page’s DOM, a stream of speech) into segments, batches those segments across users to keep expensive accelerators busy, routes each segment to the right model for its language pair, caches the enormous repeat-translation traffic so most requests never touch a GPU, and streams partial results back fast enough that a conversation feels live. Let me build it properly.

Everything hard here - batching, model routing, caching, streaming - exists to move two numbers: cost per translated word (dominated by accelerator time, because NMT inference is expensive) and latency per request (dominated by model forward-pass time plus queueing). Get those right and you serve billions of words a day on a bounded fleet of accelerators at sub-second latency; get them wrong and you either bankrupt yourself on idle GPUs or make every user wait two seconds behind a poorly batched queue. Translate is not “a dictionary.” It is a model-serving system whose product is a fast, cheap forward pass, measured in milliseconds of tail latency and fractions of a cent per thousand words.

Functional Requirements (FR)

In scope:

• Text translation. Given source text, an optional source language (auto-detect if absent), and a target language, return the translated text. This is the core path - short phrases up to a few paragraphs.
• Language auto-detection. If the caller does not specify the source language, detect it from the text (a separate, cheap classifier model) before translating.
• 100+ languages / 10,000+ pairs. Support any-to-any across 100+ languages. Not every pair has a direct model; some route through a pivot language (usually English) - xx -> en -> yy.
• Document translation. Upload a document (PDF, DOCX, PPTX, HTML), translate it while preserving layout and formatting, and return a translated document of the same type. This is an async, multi-segment job, not a single request.
• Web page translation. Translate a live web page in place: extract translatable text nodes from the DOM, translate them in bulk, and re-inject, preserving structure, links, and inline markup.
• Conversation / real-time mode. Two people speaking different languages: transcribe speech to text (ASR), translate, optionally synthesize speech (TTS) in the target language, streaming partial results so it feels live. I focus on the translate core and treat ASR/TTS as adjacent services with defined interfaces.
• Alternatives and detail. For short inputs, offer alternative translations, transliteration, and dictionary-style definitions for individual words.
• Feedback loop. Let users suggest better translations; capture that signal for offline model improvement (not applied live).

Explicitly out of scope (say it so you own the scope):

• Training the NMT models. I am designing the serving system - how trained models are deployed, batched, routed, and cached to answer live traffic at scale. The offline training pipeline (data mining, parallel-corpus alignment, model architecture, TPU training clusters) is a separate system. I consume trained model artifacts; I do not produce them.
• The ASR and TTS engines themselves. Speech-to-text and text-to-speech are their own model-serving systems with the same batching/GPU concerns. For conversation mode I define the interface to them and focus on the translation core.
• Building the model architecture. Transformer/seq2seq internals, tokenization schemes, and quality metrics (BLEU) are ML concerns; I treat a model as a black box with a known latency and throughput profile.
• Client apps and the editor UI. The browser extension, the mobile app, the camera/AR overlay for images - these are clients of the API.

The one decision that drives everything: this is a compute-bound (not I/O-bound), read-heavy, cacheable model-serving system where the expensive resource is accelerator time, the same phrases are translated millions of times, latency budgets are sub-second for interactive text and streaming for conversations, and correctness is “good-enough fluent output” rather than a single exact answer. The whole design is about keeping accelerators busy (batching) while keeping latency low (fast forward pass + cache) and cost bounded (cache + right-sizing models).

Non-Functional Requirements (NFR)

• Scale: Billions of words translated per day. Call it ~30 billion words/day. If an average request is ~15 words, that is ~2 billion requests/day, averaging ~23K requests/sec and peaking 2-3x, so design for ~60K peak QPS at the API tier. Document and web-page jobs fan a single user action into hundreds or thousands of segment-level model calls, so the internal segment QPS is much higher than the request QPS.
• Latency: Interactive text translation target p99 under 300ms end to end for a short phrase (a few hundred ms of model forward pass plus queueing plus network); the “sub-second” bar. Web-page and conversation modes stream: first partial result in under 200ms, remaining segments flowing continuously. Document jobs are async - seconds to a couple minutes depending on size, with progress.
• Availability: 99.9%+ on the interactive path. Translate is a utility people rely on mid-task; an outage is very visible. The serving tier and cache are the load-bearing pieces.
• Consistency: Eventual and best-effort. There is no single “correct” translation, and a model version rollout changing an output slightly is acceptable. The cache may briefly serve an output from an older model version after a rollout; that is fine. What must be consistent within a request is that a document/web page uses one model version throughout so the output is coherent.
• Durability: The cache is disposable (losing it just re-runs inference). What must be durable: document jobs and their outputs (a user uploaded a file and expects it back), model artifacts (versioned, in a model registry / blob store), and user feedback (training signal). The models themselves and the training corpus are the systems of record for quality.
• The real budget is accelerator utilization and cost per word. At billions of words/day, inference cost dominates. A high cache hit ratio (most traffic is repeat/common phrases) and high batch efficiency (accelerators near full utilization) are first-order design constraints - they decide the size of the GPU/TPU fleet and therefore the bill.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the fleet size and the caching strategy.

Request and word volume:

Target: ~30 billion words/day translated.
   Avg request ~15 words -> ~2e9 requests/day.
   2e9 / 86,400 s = ~23,000 requests/sec average.
   Peak at ~2.5x = ~58,000 requests/sec. Call it ~60K peak QPS.
Words/sec: 30e9 / 86,400 = ~347,000 words/sec average; ~870K/sec peak.

Cache hit ratio - the master lever:

Translation traffic is a brutal power law. A huge fraction is short, common
   phrases ("hello", "thank you", "where is the bathroom", UI strings, menu items,
   repeated web-page boilerplate). Assume a steady-state cache HIT ratio of ~70%
   on the interactive text path (higher for short common phrases, near-zero for
   long unique sentences).
   Misses that actually hit a model: ~30% of 60K = ~18K model-inference req/sec.
   BUT each miss may be multiple segments (a paragraph splits into sentences),
   so segment-level inference QPS is higher - call it ~40-60K segment inferences/sec.

Accelerator fleet sizing (the cost driver):

A single NMT forward pass for one short sentence: ~20-80ms on a modern
   accelerator, but you NEVER run batch size 1 in production - you batch.
   Say one accelerator, well-batched, sustains ~500-1,500 sentence-translations/sec
   depending on model size and sequence length. Take ~1,000/sec as a planning number.
   Needed for ~50K segment inferences/sec at peak: 50,000 / 1,000 = ~50 accelerators
   for the hot path steady state. Add headroom, the long tail of rare language
   pairs (each pair needs its model resident somewhere), redundancy, and document/
   web-page burst load -> a fleet on the order of hundreds to low-thousands of
   accelerators across regions. Cache is what keeps this from being 10x bigger.

Model footprint and the 10,000-pair problem:

100+ languages -> 100 * 100 = 10,000+ directed pairs. You do NOT ship 10,000 models.
   - A multilingual model handles many languages in one artifact (shared encoder/decoder).
   - Rare pairs route through a PIVOT (xx -> en -> yy), so you mainly need
     "to-English" and "from-English" capacity plus a big multilingual model.
   Each model artifact: hundreds of MB to a few GB. A handful of large multilingual
   models + specialized high-traffic pair models covers the space. They must be
   RESIDENT in accelerator memory; cold-loading a model per request is fatal to latency.

Document / web-page fan-out:

A 40-page document ~ 20,000 words ~ 1,300 sentences -> 1,300 segment inferences
   from ONE user action. A web page ~ 200-2,000 text nodes -> hundreds of segments.
   These bulk jobs dwarf single-phrase requests in internal inference count, which
   is why segment-level QPS >> request QPS and why bulk jobs run async/streamed
   and get batched aggressively.

Storage:

Cache: hot phrases keyed by (src_lang, tgt_lang, model_version, hash(text)).
   Tens to hundreds of millions of hot entries; each ~a few hundred bytes to a few KB.
   ~100M entries * ~500 B = ~50 GB hot cache per region (in-memory tier),
   backed by a larger distributed cache. Trivially affordable vs the GPU bill.
Document jobs + outputs: durable blob store, TTL'd (e.g. delete after 30 days).
Feedback + logs: append-heavy, sampled, into an OLAP/warehouse for training.

The takeaways: this is a ~60K peak request/sec, compute-bound system where the bottleneck is accelerator time, not bandwidth or storage; the cache absorbs ~70% of interactive traffic so only ~18K req/sec (fanning to ~50K segment inferences/sec) actually run a model; the fleet is hundreds to low-thousands of accelerators kept busy by batching; and the 10,000-pair space is collapsed by multilingual models + pivot routing rather than 10,000 artifacts. Batching, caching, model routing, and streaming are the interview.

High-Level Design (HLD)

The architecture splits into a serving plane (the request-answering path: API gateway -> language detect -> cache -> segmenter -> batching/inference -> model on accelerator, carrying the 60K QPS firehose and all the segment fan-out) and a control/offline plane (model registry and rollout, the training pipeline that produces new model versions, document-job orchestration, feedback and telemetry aggregation - not on the interactive hot path). A client sends text; the nearest region detects the language, checks the cache, and on a miss segments the input, batches segments across many users, runs the right model on an accelerator, caches the result, and returns it - streaming for bulk and conversation modes.

                    ┌──────────────────── OFFLINE / CONTROL PLANE ─────────────────────┐
                    │  Training pipeline (parallel corpora -> new model versions)        │
                    │  Model registry + rollout (versioned artifacts, canary, pin)       │
                    │  Document-job orchestrator (async, durable, progress)              │
                    │  Feedback + telemetry aggregation (sampled -> warehouse)           │
                    └──────▲───────────────┬───────────────────────┬─────────────────────┘
                           │ metrics       │ push model versions    │ enqueue doc jobs
   ──────────────────────────┼─────────────┼───────────────────────┼──────────────────────
   SERVING PLANE (per region, x several)   ▼                       │
   ┌─────────┐  1. POST /translate   ┌─────────────────────────────┴────────────────────┐
   │ Client  │  {text, tgt, src?}    │  API GATEWAY + L7 LB                               │
   │ (app /  │──────────────────────>│  - auth, quota, rate limit, request routing        │
   │  web /  │  stream for web/convo │  - route to nearest healthy region                 │
   │  convo) │                       └───────────────────────┬───────────────────────────┘
   └────┬────┘                                               ▼
        │                            ┌──────────────────────────────────────────────────┐
        │<── cached translation ─────│  TRANSLATION FRONTEND (stateless)                  │
        │    (~70% of text traffic)  │  - LANGUAGE DETECT (cheap classifier) if src=auto  │
        │                            │  - CACHE lookup: (src,tgt,model_ver,hash(text))    │
        │                            │  - on hit: return; on miss: SEGMENT the input      │
        │                            │  - MODEL ROUTER: pick model / pivot for (src,tgt)  │
        │                            └───────────────┬──────────────────────────────────┘
        │                                            │ miss: segments to translate
        │                            ┌───────────────▼──────────────────────────────────┐
        │                            │  INFERENCE / BATCHING SERVICE                     │
        │                            │  - collect segments across users into a batch     │
        │                            │  - pad/bucket by length; dispatch to model replica │
        │                            │  - stream partial results back as they complete    │
        │                            └───────────────┬──────────────────────────────────┘
        │                                            │ batched tensor
        │                            ┌───────────────▼──────────────────────────────────┐
        │                            │  MODEL SERVERS (GPU/TPU)                          │
        │                            │  - resident multilingual + per-pair NMT models    │
        │                            │  - forward pass; return translated tokens          │
        │                            │  - autoscaled replicas per model / language group  │
        │                            └───────────────────────────────────────────────────┘

Request flow (short text translation):

1. The client sends POST /translate with the text, a target language, and an optional source language. The API gateway authenticates, applies quota/rate limits, and routes to the topologically nearest healthy region.
2. The translation frontend (stateless) runs language detection if the source is auto (a small, fast classifier - cheap compared to translation).
3. It builds a cache key = (src_lang, tgt_lang, model_version, hash(normalized_text)) and checks the cache. On a hit (~70% of short-text traffic), it returns immediately - no accelerator touched. This is the primary cost and latency win.
4. On a miss, the frontend segments the input into sentences (or smaller units), because models translate segment-by-segment and segmentation enables batching and partial streaming. The model router picks the model for the (src, tgt) pair, or a pivot route (src -> en -> tgt) if no direct model exists.
5. Segments go to the inference/batching service, which collects segments from many concurrent users, groups them by length bucket, forms a padded batch, and dispatches it to a model server replica that has the required model resident on its accelerator.
6. The model server runs the forward pass and returns translated tokens. The batching service demultiplexes results back to the right requests, the frontend caches each segment’s translation, reassembles the segments in order, and returns the result. For streaming modes, each segment is emitted as soon as it completes.

Bulk flow (document / web page):

1. Document: client uploads a file; the document orchestrator stores it durably, creates an async job, extracts text with layout metadata, and fans the segments into the same inference path (aggressively batched), reassembles the translated segments back into the original format, stores the output, and reports progress. The client polls or gets notified.
2. Web page: the client (extension/app) extracts translatable DOM text nodes and sends them as a batch translate call; results stream back and are re-injected into the DOM in place, preserving markup.

Control/offline flow (models, feedback):

1. The training pipeline produces new model versions offline; the model registry versions them and the rollout controller canaries a new version to a slice of traffic, then promotes it. model_version is part of the cache key so a rollout does not serve stale-model outputs as if they were new.
2. Sampled feedback and telemetry flow to a warehouse for the next training cycle. None of this is on the interactive path.

The key insight: the serving plane is a stateless, cache-fronted, batch-everything model-serving path where accelerators are the scarce resource kept busy by cross-user batching, most interactive traffic never reaches a model because the cache absorbs the power-law of common phrases, the 10,000-pair space collapses to a few resident multilingual models plus pivot routing, and bulk/streaming modes reuse the exact same inference core with different front-ends.

Component Deep Dive

The hard parts, naive-first then evolved: (1) the batching/inference service - how you keep expensive accelerators busy without adding latency; (2) the cache - how you absorb the repeat-translation power law without serving wrong or stale-model output; (3) model routing across 100+ languages - how you cover 10,000 pairs without 10,000 models; (4) documents/web pages/conversations - how you preserve structure and stream fast on top of the same core.

1. Batching and inference: why “one request, one forward pass” fails, then dynamic batching

Naive approach: each incoming translation request runs its own model forward pass immediately, batch size 1, on whatever accelerator is free.

Where it breaks:

• Accelerators sit idle and cost explodes. GPUs/TPUs are throughput machines: a batch of 32 sentences costs almost the same wall-clock as a batch of 1, because the accelerator is massively parallel. Running batch size 1 wastes ~90%+ of the hardware. At billions of words/day, that is the difference between hundreds and thousands of accelerators - a direct multiple on the bill.
• Throughput collapses. One-at-a-time inference caps you at maybe a few hundred req/sec per accelerator instead of one to two thousand. You would need an order of magnitude more hardware for the same load.
• No way to prioritize or bound latency. With immediate per-request dispatch there is no queue to reason about; a burst just thrashes the accelerators.

Evolution A: dynamic (adaptive) batching. Put a batching service in front of the model servers. Incoming segments from many users land in a queue; the batcher forms a batch when either the batch reaches a target size or a small max-wait timer (e.g. 5-10ms) elapses - whichever comes first. This trades a few ms of latency for a huge throughput gain.

Segments arriving from many users -> batch queue (per model, per length bucket):
   accumulate until batch_size = 32  OR  10ms elapsed, then dispatch.
   Low load: timer fires first -> small batches, still low latency.
   High load: size fills first -> full batches, max throughput.
   One forward pass now serves 32 users; accelerator utilization jumps from ~10% to ~80%+.

Evolution B: length bucketing to kill padding waste. Transformers process a batch at the length of its longest sequence; padding a 3-word sentence to sit in a batch with a 60-word sentence wastes compute on padding tokens. Bucket segments by length and batch within a bucket, so batches are roughly uniform length and padding is minimal.

Buckets: [1-8 tokens], [9-16], [17-32], [33-64], [65-128] ...
   A short "thank you" batches with other short segments (tiny, fast pass);
   a long clause batches with other long segments. No 3-word sentence dragged
   through a 128-token forward pass. Effective throughput per accelerator rises again.

Evolution C: continuous batching / in-flight batching for autoregressive decode. NMT decoding is autoregressive (token by token); different sentences finish at different lengths. Static batches stall on the slowest sentence. Continuous batching lets a finished sequence leave the batch and a new one enter mid-flight, so the accelerator never idles waiting for the longest decode. This is the same technique modern LLM servers use.

Evolution D: replica pools per model, autoscaled on queue depth. Each model (or language group) has a pool of accelerator-backed replicas. The batching service load-balances across replicas and the pool autoscales on queue depth / batch-wait latency, not CPU. Hot language pairs (en<->es, en<->zh) get many replicas; rare pairs share a smaller multilingual pool.

Evolution E: model right-sizing and quantization. Not every request needs the biggest model. Use quantized (int8) or distilled smaller models for latency-critical short text, reserving the large high-quality model for documents where quality matters more than latency. Smaller/quantized models mean bigger batches fit in accelerator memory and forward passes are faster.

The result: accelerators run at high utilization via dynamic batching, padding waste is minimized by length bucketing, autoregressive decode never stalls on the slowest sequence via continuous batching, hot pairs scale independently via per-model replica pools, and latency-critical paths use right-sized/quantized models - collapsing the fleet size (and the bill) by roughly an order of magnitude versus batch-size-1.

2. Caching: why “no cache” fails, then a model-version-keyed multi-tier cache

Naive approach: every translation request runs the model; no caching, since “translations are just computed.”

Where it breaks:

• You pay for the same translation millions of times. Translation traffic is a brutal power law: “hello,” “thank you,” UI strings, product names, menu items, common sentences, and repeated web-page boilerplate are translated over and over. Without a cache you burn accelerator time (the expensive resource) recomputing identical outputs. Cache hit ratio is the single biggest lever on cost.
• Latency is worse than it needs to be. A cache hit is a memory lookup (sub-millisecond) versus a batched forward pass (tens to hundreds of ms plus queue). Caching the common case makes the p50 tiny.
• Serving after a model upgrade gets subtle. If you cache by text alone and then roll out a better model, users keep getting old outputs forever, and a document could mix old and new translations.

Evolution A: cache keyed by (src, tgt, model_version, hash(normalized_text)). Normalize the text (trim, collapse whitespace, optionally case-fold for lookup with care), hash it, and key the cache by the full tuple including model version. This makes cache entries correct across rollouts: a new model version is a new keyspace, so upgrades do not serve stale outputs, and old entries age out naturally.

key = (src="en", tgt="es", model_ver="nmt-2026.07", sha256("thank you"))
   Hit  -> return "gracias" in <1ms, no accelerator touched.
   Miss -> translate, then SET the key with a TTL. Next identical request hits.
Rolling out nmt-2026.08 changes model_ver -> fresh keyspace; no stale answers.

Evolution B: multi-tier cache (in-process -> regional distributed -> nothing). A small in-process LRU on each frontend catches the hottest phrases with zero network hop; behind it a regional distributed cache (Redis/Memcached-class) holds the tens-to-hundreds of millions of hot entries. A miss at both tiers goes to inference. Write-through on inference completion populates both.

Frontend in-process LRU (hottest ~thousands of phrases, ~0 hop)
   -> miss -> regional distributed cache (~100M entries, ~1ms)
   -> miss -> segment + infer -> write result to both tiers.

Evolution C: segment-level caching, not just whole-request. Cache at the segment (sentence) level, not only the full input. A paragraph that differs by one sentence still hits the cache on its other sentences, so only the novel sentence runs a model. This dramatically raises the effective hit ratio for documents and web pages, which share boilerplate.

Input paragraph = [S1, S2, S3]. S1, S2 cached, S3 novel.
   Only S3 goes to inference; S1/S2 served from cache. Reassemble in order.
   Web-page boilerplate ("Home", "About", "Sign in") is ~always a segment hit.

Evolution D: negative/low-confidence handling and TTLs. Cache successful translations with a TTL (weeks); do not cache low-confidence or user-edited-pending outputs. Bound cache memory with LRU + TTL eviction so a flood of unique long sentences cannot evict the hot short-phrase set. Keep the cache scan-resistant (frequency-aware admission) so one-hit long sentences do not pollute it.

Evolution E: bypass cache for privacy-sensitive/large unique text. Very long unique passages are near-zero hit rate; there is no point caching them (and they may be sensitive). Skip caching above a length threshold or when the caller flags no-store; spend the cache budget on the cacheable short-phrase power law.

The result: ~70% of interactive traffic is served from memory in sub-millisecond time without touching an accelerator, upgrades are safe because the model version is in the key, documents and web pages get high effective hit ratios via segment-level caching, and memory stays bounded and pollution-resistant - the cache is the accelerator fleet’s shield, not just a speedup.

3. Model routing across 100+ languages: why “one model per pair” fails, then multilingual + pivot

Naive approach: train and deploy a dedicated model for every language pair, and route each request to its pair’s model.

Where it breaks:

• 10,000+ models is untenable. 100+ languages means 100 * 100 = 10,000+ directed pairs. Training, storing, versioning, and keeping 10,000 models resident in accelerator memory is impossible - accelerator RAM holds a handful of models, not thousands. Cold-loading a rare-pair model per request would add seconds of latency.
• Rare pairs have little training data. Icelandic-to-Zulu has almost no parallel corpus; a dedicated model would be poor quality even if you could afford it.
• Fleet fragmentation. Ten thousand tiny models means terrible batching (each pair’s traffic is thin), so accelerator utilization craters for the long tail.

Evolution A: multilingual models. A single multilingual NMT model with a shared encoder/decoder handles many languages at once (you tell it the target language via a token). One artifact covers a large slice of the pair space, benefits from transfer learning (high-resource pairs help low-resource ones), and batches well because all its traffic shares one model.

One multilingual model handles {en, es, fr, de, it, pt, ...} -> {any target in set}.
   Request tags the target language: <2es> "hello" -> "hola".
   Dozens of languages, one resident artifact, one replica pool, great batching.

Evolution B: pivot (bridge) translation through English. For pairs with no direct model and little data, route through a pivot language (usually English): src -> en -> tgt. You only need robust “to-English” and “from-English” capacity plus the multilingual core, not a direct model for every exotic pair.

Basque -> Zulu, no direct model:
   Basque -> English  (to-English model)
   English -> Zulu    (from-English model)
   Two hops, each well-supported and well-batched. Quality slightly lower than
   direct, but the pair is served at all - and cheaply.

Evolution C: a routing table + tiered models. The model router consults a config that maps each (src, tgt) to a strategy: direct dedicated model (high-traffic pairs like en<->es, en<->zh get their own tuned, possibly larger model), multilingual model (mid-traffic), or pivot (long tail). High-traffic pairs get quality and dedicated capacity; the tail gets coverage via multilingual/pivot.

route(en, es) -> direct "en-es" model (high traffic, tuned, many replicas)
route(en, sw) -> multilingual model  (medium traffic)
route(eu, zu) -> pivot: (eu->en multilingual) then (en->zu multilingual)

Evolution D: keep models resident, autoscale by demand. Pin the multilingual and high-traffic direct models permanently resident; keep rarer specialized models resident on a smaller shared pool, loaded/evicted by demand with hysteresis (do not thrash). The router load-balances within each model’s replica pool. Language detection failures fall back to the multilingual model, which is robust to source-language ambiguity.

The result: the 10,000-pair space collapses to a few large multilingual models plus dedicated models for the highest-traffic pairs and pivot routing for the long tail - so every pair is covered, high-traffic pairs get quality and capacity, accelerator memory holds a handful of resident models instead of thousands, and batching stays healthy because traffic concentrates on few models.

4. Documents, web pages, and conversations: why “just translate the blob” fails, then structure-aware pipelines

Naive approach: for a document or web page, extract all the text, translate it as one big string, and hand it back.

Where it breaks:

• You destroy the layout. A PDF/DOCX has structure (headings, tables, columns, fonts, images with captions); a web page has DOM structure, links, and inline markup (<b>, <a>). Flattening to one string and translating loses all of it - you get a wall of translated text, not a translated document.
• One giant request blows the latency and memory budget. A 40-page document is thousands of sentences; a single synchronous call would time out and cannot stream progress.
• Conversations need to be live. Waiting for a full utterance to finish before translating makes dialog feel broken; you need partial, streaming results.

Evolution A: extract-translate-reinject with structure preserved. For documents, parse into a structured representation (blocks, runs, cells) and translate the text runs while keeping the structural skeleton; then rebuild the document of the same type, re-inserting translated runs into their original positions with formatting intact. For web pages, walk the DOM, collect translatable text nodes (skipping code, scripts, and marked no-translate regions), translate them as a batch, and re-inject each into its original node so links and inline tags survive.

Web page: DOM -> [textNode1, textNode2, ...] (skip <script>, <code>, translate="no")
   -> batch-translate segments (segment-level cache hits on boilerplate)
   -> re-inject each translated string into its original node, markup preserved.
Document: DOCX -> {paragraphs, runs, tables} -> translate runs -> rebuild DOCX.

Evolution B: async job orchestration for documents. A document is an async job: store the upload durably, create a job record, fan the segments into the batching service (aggressively batched since latency per segment is not user-critical, throughput is), track progress, reassemble, store the output, and notify. The client polls a status endpoint or receives a push. Failures are retryable at segment granularity.

POST /documents -> {job_id, status: "processing"}
   orchestrator: store file -> extract segments -> enqueue for batch inference
   -> reassemble -> store output -> status "done" with download URL.
GET /documents/{job_id} -> {status, progress: 0.62}
Pin ONE model_version for the whole job so the output is coherent.

Evolution C: streaming for web pages and conversations. Emit each segment’s translation as soon as it completes rather than waiting for the whole batch. For web pages, the visible viewport’s text nodes are prioritized so the user sees the top of the page translate first. For conversations, ASR emits partial transcripts; the translator translates each finalized clause and streams it, optionally to TTS, so dialog feels near-live.

Conversation: audio -> ASR (partial transcripts) -> on clause finalization ->
   translate clause (short, low-latency, cache-friendly) -> stream text ->
   optional TTS in target voice. First partial in <200ms; continuous flow after.
Web page: translate + stream viewport segments first, then off-screen, so the
   user perceives instant translation of what they can see.

Evolution D: coherence and consistency within a job. Pin a single model version for the duration of a document or page so terminology is consistent throughout, and (for quality) allow limited cross-segment context so pronouns/terms stay consistent. Cache is still keyed by that pinned version.

The result: documents and web pages keep their structure via extract-translate-reinject, large documents run as durable async jobs that stream progress and reuse the same batched inference core, conversations feel live via clause-level streaming with ASR/TTS at the edges, and each job stays coherent by pinning one model version - all on top of the exact same batching, caching, and routing core as short text.

API Design & Data Schema

The interactive API is a small REST/gRPC surface; the heavy internal traffic is segment-level inference calls.

Public API (REST/gRPC, shown as REST)

# Short text translation (the core path)
POST /v1/translate
  body: { "text": "hello",
          "source": "auto",            # or "en"; auto -> run language detect
          "target": "es",
          "format": "text|html",       # html preserves inline markup
          "model_hint": "fast|quality" # optional: quantized vs large model
        }
  200:  { "translations": [ { "text": "hola",
                              "detected_source": "en",
                              "confidence": 0.99,
                              "model_version": "nmt-2026.07",
                              "alternatives": ["hola", "buenas"] } ] }

# Batch translate (web pages: many nodes in one call, results stream/array)
POST /v1/translate:batch
  body: { "segments": ["Home","About","Sign in", ...], "source":"auto","target":"fr" }
  200:  { "translations": [ {"text":"Accueil"}, {"text":"À propos"}, ... ] }

# Language detection only
POST /v1/detect
  body: { "text": "bonjour" }
  200:  { "language": "fr", "confidence": 0.98 }

# Document translation (async job)
POST /v1/documents        (multipart: file, source, target)
  202:  { "job_id": "doc_9f3", "status": "processing" }
GET  /v1/documents/{job_id}
  200:  { "status": "processing|done|failed", "progress": 0.62,
          "output_url": "https://.../doc_9f3_es.docx" }   # when done

# Conversation / streaming (WebSocket or gRPC stream)
WS   /v1/conversation
  client -> { "audio_chunk": <bytes> } | { "text": "partial clause", "final": true }
  server -> { "translation": "...", "final": false, "segment_id": 12 }   # streamed

# Feedback (training signal, async, off the hot path)
POST /v1/feedback
  body: { "source":"en","target":"es","original":"hello","model":"nmt-2026.07",
          "suggested":"hola amigo" }
  202:  { "status": "recorded" }

Internal data / config schemas

1. Translation cache (per region) - disposable, in-memory, sharded. Not a database; a distributed KV with TTL eviction, fronted by an in-process LRU. Rebuilt by re-running inference on loss.

Cache entry, keyed by (src_lang, tgt_lang, model_version, hash(normalized_text)):
  translation   : string                    -- the cached output (per segment)
  confidence    : float                     -- model confidence at write time
  ttl_deadline  : absolute timestamp        -- weeks; LRU + TTL eviction
  hit_count     : int                       -- frequency-aware admission (scan-resistant)

Structure: sharded by hash(key) across nodes; in-process LRU in front for the
           hottest phrases; segment-level entries so partial reuse works.
No cross-region consistency needed; each region has its own cache.

2. Model registry - versioned artifacts, strongly consistent, read-mostly. The system of record for what is deployed. A small relational/KV store plus a blob store for the artifacts.

model[model_id, version] :
  artifact_url   : blob path to weights (hundreds of MB - few GB)
  type           : direct | multilingual
  languages      : [ supported language codes ]           -- for multilingual
  pair           : (src, tgt)?                             -- for direct
  quant          : fp16 | int8                             -- latency tier
  status         : canary | active | retired
  rollout_pct    : int                                     -- for canary promotion

3. Routing table - which model/strategy for each pair. Small, read-mostly config pushed to frontends; consulted per miss.

route[(src, tgt)] -> strategy:
  DIRECT   model_id           -- high-traffic tuned pair (e.g. en-es)
  MULTI    model_id           -- multilingual model covers the pair
  PIVOT    via="en"           -- src->en then en->tgt, each resolved recursively

4. Document jobs - durable, transactional-ish. A relational store (job state, progress, ownership) plus blob store (input + output files). Must survive restarts; a user expects their file back.

job[job_id] :
  user_id, source, target, model_version_pinned
  input_blob, output_blob?
  status         : processing | done | failed
  progress       : float
  segment_count, segments_done                -- for progress + retry granularity
  created_at, expires_at                        -- TTL cleanup (e.g. 30 days)

5. Feedback + telemetry - append-heavy, OLAP. Sampled, into a warehouse/columnar store; feeds the offline training pipeline and quality dashboards. Never on the interactive path.

feedback[id] : src, tgt, original, model_version, suggested, user_id?, ts
telemetry    : QPS, cache_hit_ratio, p50/p99 latency, accelerator_utilization,
               per-pair volume, batch_size distribution  -> rolled up by time+region

Why these choices: the cache is huge, hot, and disposable - an in-memory sharded KV with no consistency guarantees, because the model can always recompute; a durable database here would be slow and pointless. The model registry and routing table are tiny and correctness-critical - strongly consistent, globally replicated, because deploying the wrong model version or a broken route is a real incident. Document jobs are durable and stateful - a relational store plus blob storage, because a user’s uploaded file and its translated output must survive failures. Feedback/telemetry is append-heavy and aggregate-read - a columnar/OLAP store fed by sampled events. Each layer gets exactly the consistency, durability, and cost model its access pattern demands. Notably, there is no relational database on the interactive translate path - it is a cache-and-inference problem, not a transactional one; the only durable relational state is document jobs and the small control-plane config.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Accelerator throughput and utilization (the master lever, breaks first). Inference is the expensive, scarce resource; batch-size-1 wastes ~90% of it. Fix: dynamic batching (size-or-timeout) to fill accelerators, length bucketing to kill padding waste, continuous batching so autoregressive decode never stalls on the slowest sequence, per-model replica pools autoscaled on queue depth, and quantized/distilled models for latency-critical short text. This collapses the fleet by roughly an order of magnitude versus naive per-request inference.

2. Cache hit ratio (the cost and latency multiplier). Every point of miss rate is more accelerator time and higher latency. Fix: model-version-keyed multi-tier cache (in-process LRU -> regional distributed cache), segment-level caching so documents/pages reuse shared boilerplate, frequency-aware scan-resistant eviction so unique long sentences do not evict the hot short-phrase set, and skip caching near-zero-hit long unique text. ~70% of interactive traffic never touches an accelerator.

3. The 10,000 language-pair explosion. You cannot ship or keep resident 10,000 models. Fix: multilingual models (one artifact, many languages, transfer learning helps the tail), pivot routing through English for the long tail, and dedicated tuned models only for the highest-traffic pairs. A handful of resident models covers the whole space with healthy batching.

4. Tail latency on cache misses. A miss pays batching wait + queue + forward pass; a badly tuned batcher or an overloaded replica pool blows the sub-second budget. Fix: cap the batch max-wait (a few ms) so batching never adds much latency, autoscale replica pools on batch-wait latency not CPU, route latency-critical short text to fast/quantized models, and stream segment results so the user sees output progressively rather than waiting for the whole input. Hedge slow replicas by re-dispatching a stalled segment.

5. Hot language pair / hot phrase. A viral phrase or a dominant pair (en<->es) could overload one model pool or one cache shard. Fix: hot phrases are almost always cache hits (memory-served, nearly free), so a hot phrase rarely bottlenecks inference. Hot pairs get more replicas (autoscaled independently). The cache is sharded by hash(key) so hot phrases spread across shards; a single blistering phrase is replicated into the in-process LRU on every frontend, so it never even reaches the distributed tier.

6. Document/web-page burst fan-out. One user action can create thousands of segment inferences, spiking internal QPS. Fix: bulk jobs are async and aggressively batched (throughput over per-segment latency), fed through a queue that smooths bursts, segment-level cache absorbs shared boilerplate, and bulk traffic runs at lower priority than interactive text so it never starves live requests. Document jobs are retryable at segment granularity.

7. Model rollout consistency. A new model version must not corrupt caches or make a document mix versions. Fix: model_version is in the cache key, so a rollout is a fresh keyspace and old entries age out; the rollout controller canaries a version to a traffic slice before promotion; a document/page pins one model version for its whole lifetime so its output is coherent. Rollback is flipping the active version pointer.

8. Shard keys, stated plainly. Cache -> sharded by hash(src, tgt, model_version, text) across cache nodes (segment-level, so partial reuse works). Inference -> routed by model_id to that model’s replica pool, then load-balanced within the pool by queue depth. Document jobs -> partitioned by job_id. Telemetry/feedback -> partitioned by time + region. There is no global shared shard on the interactive path - each region is self-sufficient with its own cache and model replicas.

9. Single points of failure. Regions, replica pools, and cache nodes are redundant by construction; the interactive path is stateless behind an L7 LB. Fix: multiple regions each fully capable (own frontends, cache, model servers); the L7 LB routes around dead regions; replica pools have many members and autoscale; the cache is disposable so losing a node just re-runs inference. The model registry and routing config are the centralized brains, so they are globally replicated (consistent store with failover), and the data plane never blocks on them - frontends serve from last-known routing config and resident models even if the control plane is briefly unreachable. Document jobs are durable so a failed worker’s job resumes from its last completed segment.

10. Consistency and correctness. The system is best-effort and eventually consistent by nature: there is no single correct translation, and a model upgrade slightly changing outputs is expected. The tight guarantees are coherence within a job (one pinned model version per document/page) and cache correctness across rollouts (model version in the key). Two regions serving slightly different outputs for the same text (different cache/model-version state mid-rollout) is acceptable and invisible in practice.

Wrap-Up

The trade-offs that define this design:

• Batch aggressively, trade a few ms of latency for an order of magnitude of throughput. Dynamic batching, length bucketing, and continuous batching keep expensive accelerators near full utilization, so the fleet (and the bill) is ~10x smaller than naive per-request inference - trading a few ms of batching wait for enormous cost savings. Batch-size-1 was rejected as wasting ~90% of the hardware.
• The cache is the accelerator fleet’s shield, not just a speedup. A model-version-keyed, segment-level, multi-tier cache absorbs ~70% of interactive traffic in sub-millisecond memory lookups, so most requests never run a model - trading disposable memory and TTL-bounded staleness for a massive cut in accelerator time. No-cache was rejected as paying for the same translation millions of times.
• Collapse 10,000 pairs into a few models. Multilingual models plus pivot-through-English plus dedicated models only for the highest-traffic pairs cover the whole space with a handful of resident artifacts and healthy batching - trading slightly lower quality on the long tail (pivot hops) for tractable coverage. One model per pair was rejected as impossible to store, keep resident, or batch.
• One inference core, many front-ends. Short text, documents, web pages, and conversations all reuse the same batching/caching/routing core, differing only in extraction (DOM/document parsing), orchestration (async jobs), and delivery (streaming) - trading front-end complexity for a single, well-optimized, shared hot path. Separate pipelines per surface were rejected as duplicated cost and inconsistent quality.
• Consistency spent where it matters. Strong consistency only for the tiny control plane (model registry, routing config) and durable document jobs; everything on the interactive path (cache, translations) is eventually consistent and best-effort - which is exactly what buys the cacheability and the scale, with coherence guaranteed only where it is visible (one model version per job).

One-line summary: Google Translate is a stateless, cache-fronted model-serving system where a version-keyed segment-level cache absorbs the repeat-translation power law so most requests never touch an accelerator, a dynamic-batching inference tier keeps GPUs/TPUs near full utilization, a router collapses 100+ languages and 10,000+ pairs into a few resident multilingual models plus pivot routing, and one shared inference core powers text, documents, web pages, and live conversations - all driven by a small strongly-consistent control plane the always-local data plane never waits on, so billions of words a day are translated at sub-second latency and bounded cost.