You say “Alexa, what’s the weather in Bangalore tomorrow?” and about eight hundred milliseconds later a speaker on your kitchen counter says “Tomorrow in Bangalore it’ll be partly cloudy, with a high of 31 degrees.” In that under-a-second window the device woke up on the word “Alexa,” streamed your voice to a data centre, turned the audio into text, figured out that this is a weather question about a specific city and a specific day, called a weather service, turned the answer back into a sentence, synthesized that sentence into speech, and streamed the audio back to the speaker. Any one of those steps done slowly blows the illusion that you are talking to something that understands you.

The core question this design has to answer is: a user speaks a request to a low-power device; the system must detect the wake word on-device, stream and transcribe the audio, understand the intent and its parameters, route to whatever backend fulfills it, generate a natural-language reply, synthesize it to audio, and play it back - end to end in well under a second, in many languages, across 500M devices, without becoming a surveillance machine. That one sentence hides every hard problem: wake-word detection cheap enough to run always-on on a $30 device, streaming speech recognition that starts transcribing before you finish talking, intent parsing that survives accents and misheard words, an open-ended skills ecosystem, sub-second latency across a round trip to the cloud, and the privacy question of a microphone that is always listening. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Wake-word detection. The device continuously listens locally for a wake word (“Alexa”). Only after it fires does anything get sent to the cloud. This runs always-on, on-device, at near-zero power.
  • Speech-to-text (ASR). Once awake, the device streams captured audio to the cloud, which transcribes it to text incrementally, ideally emitting partial transcripts while the user is still speaking.
  • Natural-language understanding (NLU). Turn the transcript into a structured intent: a domain (weather, music, timer, smart-home), an intent name (GetWeather), and slots (city=Bangalore, date=tomorrow).
  • Request fulfillment / skills. Route the intent to the right handler - a first-party capability (timers, alarms) or a third-party “skill” (Uber, Spotify, a smart bulb) - execute it, and get back a result.
  • Response generation + text-to-speech (TTS). Turn the result into a natural sentence and synthesize it into audio streamed back to the device to play.
  • Multi-turn context. “What about Sunday?” after a weather question should reuse the previous city. Maintain short-lived session context.
  • Multi-language. Detect / configure the user’s language and run the whole pipeline (ASR, NLU, TTS) in that language.

Explicitly out of scope (so I own the scope):

  • The audio hardware and DSP. Microphone array, beam-forming, acoustic echo cancellation, and noise suppression are firmware/DSP problems I assume the device already does; I consume clean captured audio.
  • Training the ML models. I design the serving pipeline for ASR/NLU/TTS models. The offline training of those models is its own ML-platform problem; I treat each model as a versioned artifact I deploy and call.
  • The skills marketplace UI / developer console. Building the portal where developers publish skills is a separate product surface; I design the runtime that invokes skills and the interaction model they register.
  • Account, auth, and device provisioning. Sign-in, linking a device to an account, and OAuth to third-party skills are a standard identity service I consume.
  • General open-domain chit-chat / LLM assistant. I focus on the classic intent-and-slot assistant that fulfills concrete tasks fast. An LLM fallback for open questions is noted as an extension, not the core.

The one decision that drives everything: this is a soft-real-time streaming pipeline, not a request/response API. Audio flows as a stream from the moment the wake word fires; ASR emits partials as it goes; NLU can start on a partial and refine; the whole thing is a latency budget of ~1 second split across five stages, and every stage must be pipelined and streamed rather than run to completion before the next begins. Treating it as “POST audio, get JSON back” is the naive framing that guarantees you miss the budget.

Non-Functional Requirements (NFR)

  • Scale: 500M active devices. Assume each device is used ~10 times a day, so ~5B voice requests/day. Wake-word detection runs continuously on every device but never leaves the device unless it fires. The cloud only sees the ~5B post-wake utterances.
  • Latency: the headline number is end-to-end under 1 second from end-of-speech to start-of-audio-reply, p95. Budget roughly: wake-word (on-device, ~0ms cloud), network up ~50ms, ASR finalization ~150-300ms (most of it overlapped with the user speaking), NLU ~50ms, fulfillment ~100-300ms (depends on the skill), response gen + TTS first-byte ~100-150ms, network down ~50ms. TTS and playback stream, so “first audio byte” matters more than “full reply ready.”
  • Availability: 99.9%+ on the interactive path. A failed request degrades to “Sorry, I’m having trouble right now,” which is annoying but safe. On-device fallbacks (local timers, “I didn’t catch that”) keep basic function during a partial outage.
  • Consistency: almost everything is eventual / session-scoped. There is no global ledger. Session context (last city, last intent) must be strongly readable within a session but is ephemeral and can be lost between sessions. Device settings and skill enablement are read-mostly and tolerate seconds of staleness.
  • Durability: device settings, skill enablement, and account data are durable and replicated. Raw audio is deliberately not durable by default - it is transient, processed, and discarded unless the user opts into voice history. Logs and transcripts, where retained, are governed by privacy policy.
  • Privacy: the defining constraint. An always-on microphone in a bedroom demands: wake-word detection on-device so nothing streams before the wake word; a hard visual/audible indicator when streaming; no retention of audio by default; regional data residency; and user controls to review and delete history. Privacy shapes where the wake-word model lives and what we are allowed to store.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they set the architecture.

Request volume (the interactive firehose):

500M devices, ~10 utterances/device/day
= 5,000,000,000 utterances/day
= 5B / 86,400
≈ 58,000 requests/sec average.
Peak (mornings and evenings, overlapping time zones) ~3x
≈ 174,000 requests/sec at peak.

Each request is not one call but a short-lived streaming session holding compute (an ASR decoder) for the ~2-4 seconds the user is speaking plus processing.

Concurrent streaming sessions (what actually sizes the fleet):

Avg utterance holds an ASR stream ~3s (speaking) + ~0.5s tail.
Concurrent sessions ≈ QPS * session_duration
Average: 58,000 * 3.5 ≈ 203,000 concurrent ASR streams.
Peak:   174,000 * 3.5 ≈ 610,000 concurrent ASR streams.

That is the real capacity number: at peak we must hold ~600k live ASR decode sessions simultaneously, each pinned to a GPU/accelerator slice for a few seconds. This dwarfs the “QPS” figure in importance - ASR is stateful and long-lived per request, so we size for concurrency, not throughput.

Audio bandwidth (uplink):

Captured audio: 16 kHz, 16-bit mono = 256 kbps ≈ 32 KB/s raw.
With Opus compression ~16-24 kbps ≈ 2-3 KB/s.
Per utterance ~3s * 3 KB/s ≈ 9 KB uploaded (compressed).
Aggregate uplink at peak: 174,000 * 3 KB/s ≈ 520 MB/s ≈ 4.2 Gbps.

Modest. Audio is small; the cost is compute, not bandwidth.

Compute (the real cost):

ASR is the heavy consumer. Say one accelerator can hold ~40 concurrent
streaming ASR sessions in real time.
Peak 610,000 concurrent / 40 ≈ 15,250 accelerators for ASR alone.
TTS is cheaper and bursty (only at reply time), NLU is cheap CPU.
So the fleet is dominated by streaming ASR accelerators, on the order of
tens of thousands, which is why ASR efficiency = money.

Storage:

Audio: NOT stored by default. Transient in memory during processing.
For users who opt into voice history:
  say 20% opt in, retain 90 days of transcripts + optional audio.
  Transcript per utterance ~200 bytes.
  5B/day * 20% * 200B * 90 days ≈ 18 TB of transcripts. Small.
  Optional audio (9 KB each): 5B * 20% * 9KB * 90 ≈ 810 TB. Large,
  so audio retention is opt-in, tiered to cold storage, and TTL-purged.

Device + account metadata:
  500M devices * ~2 KB (settings, enabled skills, language, account)
  ≈ 1 TB. Trivially served from a sharded KV store.

Session context (ephemeral):
  ~600k live sessions * ~4 KB context ≈ 2.4 GB. Fits in an in-memory
  store (Redis) with a short TTL. Not durable.

The takeaways: ~58k avg / ~174k peak requests/sec, but the sizing number is ~600k concurrent stateful ASR streams pinning tens of thousands of accelerators; audio bandwidth is trivial; storage is dominated by opt-in voice history (so we default to storing nothing). The architecture is a fleet of streaming ASR/TTS accelerators fronted by connection-holding gateways, with cheap CPU NLU and a small metadata layer. Compute, latency, and privacy - not storage or bandwidth - are the constraints.

High-Level Design (HLD)

The system splits into an on-device plane (wake word, capture, playback, local fallbacks), an edge/gateway plane (holds the streaming connection, routes to services, enforces the latency budget), and a cloud processing pipeline (ASR -> NLU -> Dialog/Context -> Fulfillment/Skills -> Response Gen -> TTS). Privacy controls cut across the edge and cloud.

  ┌──────────────────────────────────────────┐
  │  DEVICE (always-on, low power)            │
  │  ┌────────────┐   mic audio ring buffer   │
  │  │ Wake-word   │──fires──┐                 │
  │  │ model (DSP) │         │ open stream     │
  │  └────────────┘         ▼                  │
  │  ┌──────────────────────────────┐  play ▲  │
  │  │ Audio capture + Opus encode   │       │  │
  │  └───────────┬──────────────────┘       │  │
  └──────────────┼──────────────audio reply──┘
                 │ streaming audio (WSS/gRPC)
        ┌────────▼─────────────────────────────────┐
        │  EDGE GATEWAY (regional, stateful conn)   │
        │  - holds bidirectional stream             │
        │  - auth, region, language, orchestration  │
        └───┬───────────────┬───────────────┬───────┘
            │ audio stream   │               │ session ctx
    ┌───────▼────────┐  ┌────▼─────────┐  ┌──▼─────────────┐
    │  ASR Service    │  │  NLU Service │  │ Dialog / Context│
    │ (streaming,     │─▶│ (intent +    │─▶│  Manager        │
    │  GPU decoders)  │  │  slots)      │  │ (session state, │
    └─────────────────┘  └──────┬───────┘  │  Redis)         │
      partial+final text        │ intent   └──────┬──────────┘
                                 ▼                 │ resolved intent
                        ┌────────────────────────────────────┐
                        │  Fulfillment / Skill Router         │
                        │  ┌──────────┐  ┌──────────────────┐ │
                        │  │ 1st-party│  │ 3rd-party Skills  │ │
                        │  │ (timers, │  │ (Uber, Spotify,   │ │
                        │  │  weather)│  │  smart home) HTTPS│ │
                        │  └────┬─────┘  └────────┬─────────┘ │
                        └───────┼────────────────┼───────────┘
                                │ result payload  │
                        ┌───────▼─────────────────▼───────────┐
                        │  Response Generator (NLG template)   │
                        └───────────────┬─────────────────────┘
                                        │ reply text (SSML)
                                ┌───────▼────────┐
                                │  TTS Service    │──stream audio──▶ back
                                │ (neural vocoder)│                 to device
                                └─────────────────┘

  Cross-cutting: Device/Account Store (KV), Skill Registry (interaction
  models), Privacy/History Service, Model Serving control plane.

Request flow, step by step:

  1. Wake. The device’s DSP runs a tiny always-on wake-word model over a rolling audio buffer. Nothing leaves the device until it fires. On a fire, the device lights its indicator (privacy signal) and opens a streaming connection to the nearest Edge Gateway.
  2. Stream up. The device Opus-encodes captured audio and streams it (over a persistent WSS/gRPC bidirectional stream). It also sends the ~0.5s of pre-roll buffered before the wake word so the ASR has full context.
  3. ASR. The Gateway forwards the audio stream to the ASR Service, a streaming decoder that emits partial transcripts as audio arrives and a final transcript when it detects end-of-speech (endpointing). Partials let downstream stages start early.
  4. NLU. The transcript (and refined finals) go to the NLU Service, which classifies domain + intent and extracts slots, producing a structured intent {domain, intent, slots, confidence}.
  5. Context. The Dialog / Context Manager merges the intent with session state (previous city, pending confirmations, device location) to resolve anaphora (“what about Sunday”) and fill defaults, yielding a fully resolved intent.
  6. Fulfillment. The Skill Router dispatches to a first-party handler or a third-party skill endpoint over HTTPS, passing the resolved intent. The skill returns a structured result plus a suggested spoken response.
  7. Response + TTS. The Response Generator turns the result into a natural sentence (SSML with prosody hints); the TTS Service synthesizes it and streams audio back through the Gateway to the device, which plays it as bytes arrive.
  8. Follow-up. If the skill asked a question (“Which Bangalore?”), the Dialog Manager keeps the session open for the next turn.

The key insight: the pipeline is streamed and pipelined, not sequential. ASR emits partials while you speak, NLU can pre-classify on partials, TTS streams first audio before the full sentence is synthesized. We overlap stages to fit five heavy steps into a one-second wall clock.

Component Deep Dive

The hard parts, naive-first then evolved: (1) wake-word detection on a cheap always-on device, (2) streaming ASR that beats the latency budget, (3) intent understanding and multi-turn context, (4) the open-ended skills/fulfillment layer, plus (5) the latency budget and privacy that constrain all of it.

1. Wake-word detection on-device

Naive approach: stream all microphone audio to the cloud continuously, and run wake-word detection there.

Where it breaks:

  • Privacy catastrophe. Streaming every second of household audio to a data centre is exactly the surveillance nightmare users fear, and it is illegal in many jurisdictions without consent for every word spoken near the device.
  • Bandwidth and cost. 500M devices * 32 KB/s continuous = 16 TB/s of audio into the cloud, plus cloud compute to scan silence 24/7. Absurd.
  • Latency. Even the wake decision now includes a network round trip.

First evolution: run wake-word detection on the device’s main CPU. Better - nothing streams until the wake word fires. But a general model on the application CPU drains power and heats a device meant to sit idle for months; it also competes with other device functions.

The answer: a tiny quantized wake-word model running on a low-power DSP over a rolling buffer, with a two-stage confirm.

  • On-device, always-on, tiny. A small neural net (tens of KB, quantized int8) runs on a dedicated low-power DSP / NPU, not the main CPU, scanning a short rolling audio buffer for the wake phrase. It is deliberately tuned to be cheap and slightly trigger-happy: better a few false accepts than a missed “Alexa.”
  • Two-stage detection. Stage 1 on the DSP is a low-power, high-recall detector. On a candidate fire, stage 2 (a slightly larger model, still on-device) re-checks the buffered audio to reject obvious false positives before opening any network stream. Only a stage-2 pass streams audio out.
  • Cloud verification (optional third stage). The cloud can run a full-size verifier on the first ~1s of streamed audio and, if it decides the wake word did not occur, silently abort the session and discard the audio. This catches the rest of the false accepts without ever surfacing them to the user.
  • Pre-roll buffer. The device keeps ~0.5-1s of audio before the wake fire and includes it, so the ASR sees the full utterance context (and the assistant does not clip the start of speech that overlaps the wake word).
  • Privacy indicator. The moment stage 2 passes and streaming begins, a light ring / tone signals it. This is a hard product requirement, not a nicety.
device loop (on DSP, always-on):
  buffer = rolling 2s of audio
  if wakeword_stage1(buffer) > T1:          # cheap, high recall
     if wakeword_stage2(buffer) > T2:       # confirm, still on-device
         light_indicator()
         open_stream(gateway)
         send(pre_roll + live_audio)        # include audio before wake
  # else: discard buffer, never leaves device

The elegant part: the microphone is always listening but nothing ever leaves the device until a two-stage local detector fires, and even then the cloud can veto a false wake and discard the audio - so the privacy boundary is drawn at the device, in hardware, before the network.

2. Streaming ASR under the latency budget

Naive approach: record the full utterance, upload the whole audio file, then run a batch speech-to-text model and return the transcript.

Where it breaks:

  • Serial latency. You wait for the user to finish speaking (say 3s), then upload, then run a model that takes as long as the audio again, then start NLU. That is 5-6 seconds before you even understand the request - the budget is one.
  • No early start. Downstream stages sit idle while the user talks. All the time the user spends speaking is wasted instead of overlapped with transcription.
  • Endpointing is guessed badly. Without streaming you rely on a fixed silence timeout to decide the user stopped, which either cuts people off or adds a long trailing pause.

First evolution: chunked streaming upload with a streaming ASR model. Send audio in small chunks (say 100-200ms) as it is captured; the ASR model consumes chunks and updates a running transcript. Now transcription overlaps with speaking - by the time the user stops, most of the transcript already exists. This is the single biggest latency win. But naive streaming still needs good endpointing and produces jittery partials.

The answer: a streaming ASR service with incremental decoding, model-based endpointing, and partial-transcript fan-out.

  • Streaming decoder pinned per session. Each utterance is assigned an ASR decode session on an accelerator (RNN-T / conformer-transducer or streaming attention model) that maintains internal state and emits tokens incrementally. The session is stateful and lives for the duration of speech - which is why we size for concurrent sessions, not QPS.
  • Partial vs final transcripts. The decoder emits low-confidence partials as audio arrives (used to warm up NLU and for on-screen echo on devices with screens) and a stabilized final at endpoint. NLU can run speculatively on a late partial and re-run on the final if it changed, hiding NLU latency behind ASR.
  • Model-based endpointing. Instead of a fixed silence timeout, an endpointer model predicts when the user has finished a complete utterance (using acoustics and the partial transcript’s syntactic completeness), so we cut over to NLU the instant they are done - not after an arbitrary 700ms pause.
  • Language handling. The device’s configured language selects the ASR model; for multilingual households a lightweight language-ID pass on the first ~1s can route to the right model. Each language is a separately deployed model behind the same service interface.
  • Barge-in support. If the user speaks over the assistant’s reply, the device re-opens the stream and the ASR starts a new session while TTS is cancelled - so you can interrupt.
  • Graceful degradation. If confidence is low or audio is garbled, ASR returns a low-confidence flag; the assistant asks “Sorry, I didn’t catch that” rather than acting on a bad transcript.
ASR session (streaming):
  on audio_chunk:
     state, tokens = decoder.step(state, chunk)
     emit PARTIAL(tokens)                 # downstream may start early
     if endpointer(state, tokens) == DONE:
         emit FINAL(stabilize(tokens))
         close session, free accelerator slot

The point: we transcribe while the user speaks, not after - streaming chunks into a stateful decoder that emits partials, and letting a model (not a timeout) decide when speech ends - so almost all ASR latency is hidden behind the time the user was going to spend talking anyway.

3. Intent understanding and multi-turn context (NLU)

Naive approach: keyword matching. If the transcript contains “weather,” call the weather API with whatever city string appears.

Where it breaks:

  • Brittleness. “Do I need an umbrella tomorrow in Pune” contains no keyword “weather” but is a weather query. Keyword matching misses paraphrases and catches false ones (“I love the weather in this song”).
  • Slot extraction is ad hoc. Pulling “Pune” and “tomorrow” out reliably, and distinguishing city from person from song, needs structure, not substring search.
  • No context. “What about Sunday?” has no city and no verb; keyword matching produces nothing.
  • Ambiguity across domains. “Play Thriller” could be a song, an audiobook, or a movie. Keywords cannot arbitrate.

First evolution: a trained intent classifier + slot tagger per domain. Classify the utterance into (domain, intent) with an ML model, and run a sequence tagger to label slot spans (city, date, artist). Now “do I need an umbrella” maps to Weather.GetForecast and slots come out typed. This handles paraphrase and typed slots but still fails on multi-turn context and cross-domain ambiguity.

The answer: hierarchical NLU (domain -> intent -> slots) plus entity resolution, plus a Dialog/Context Manager for multi-turn.

  • Hierarchical classification. First route to a domain (music, weather, smart-home, timers), then to an intent within that domain, then extract and type slots. Each skill registers an interaction model (its intents and slot types), so the router knows the finite space of intents a given enabled skill set supports - this shrinks the classification problem dramatically versus open-ended.
  • Entity resolution. A raw slot string (“Bangalore”, “the office”, “mom”) is resolved to a canonical entity: Bangalore -> city_id, the office -> a saved location, mom -> a contact. Resolution uses per-user catalogs (contacts, saved places, smart-home device names) plus global gazetteers. “Turn off the kitchen light” resolves kitchen light against that account’s device registry.
  • Dialog / Context Manager holds session state. Keyed by session, it stores the last intent, unfilled slots, pending confirmations, and recently mentioned entities in a short-TTL store (Redis). “What about Sunday?” is resolved by carrying forward the previous turn’s city=Bangalore and swapping date=Sunday. A follow-up “yes” resolves against a pending confirmation. This is what makes the assistant feel like it remembers.
  • Confidence and disambiguation. If two intents are close, or a required slot is missing/ambiguous, the assistant asks a clarifying question (“Did you mean the song or the movie?”) instead of guessing - and keeps the session open for the answer.
  • Speculative execution on partials. For latency, NLU can run on a near-final partial transcript so the intent (and even a skill pre-warm) is ready the instant ASR finalizes; if the final transcript changed the intent, re-run - cheap, since NLU is CPU-light.
understand(transcript, session):
  domain  = classify_domain(transcript, enabled_skills)
  intent  = classify_intent(transcript, domain)
  slots   = tag_slots(transcript, domain, intent)
  slots   = resolve_entities(slots, user_catalogs)      # city_id, device_id
  intent  = merge_context(intent, slots, session)       # carry-forward city
  if missing_required_slot(intent) or low_confidence(intent):
     return CLARIFY(question, keep_session_open)
  return RESOLVED(domain, intent, slots)

The elegant part: NLU is a funnel - domain then intent then typed, resolved slots - scoped by the finite interaction models of the user’s enabled skills, and a session-scoped context store carries entities across turns so short follow-ups resolve against what was just said. We ask rather than guess when confidence is low.

4. Fulfillment and the skills ecosystem

Naive approach: hardcode every capability. A giant switch statement in the assistant: if intent is weather call the weather module, if music call the music module, and so on.

Where it breaks:

  • Does not scale to third parties. Alexa’s value is tens of thousands of third-party skills (order a pizza, control a thermostat, play a meditation). You cannot hardcode capabilities you do not own, and you cannot ship a release every time a developer adds one.
  • Coupling and blast radius. Every capability lives in the core assistant, so one skill’s bug or slow dependency can take down the whole pipeline.
  • No isolation or auth. A third party must reach a user’s data (their Spotify, their thermostat) only with scoped consent, which a monolith cannot cleanly enforce.

First evolution: a plugin interface with a skill registry. Define a contract: a skill registers an interaction model (its intents + slots) and an HTTPS endpoint. The router looks up the skill for a resolved intent and POSTs the intent to its endpoint. Now third parties extend the assistant without touching core. But you still need routing, isolation, latency control, and auth.

The answer: a Skill Router over a registry of interaction models, invoking sandboxed/remote skill endpoints with per-skill auth, timeouts, and fallbacks.

  • Skill Registry. Each skill declares its invocation name, its interaction model (intents, slot types, sample utterances - used to train/scope NLU), its endpoint (an HTTPS webhook or a hosted serverless function), and its required OAuth scopes. NLU is scoped to the intents of the user’s enabled skills plus built-ins.
  • Routing. Given a resolved intent, the router maps it to the owning skill (explicit invocation “Ask Uber to…” routes directly; implicit intents route by the enabled-skill interaction models). It then invokes the skill’s endpoint with the resolved intent, session context, and a scoped access token.
  • Isolation and safety. Third-party skills run outside the core pipeline - as the developer’s own remote service or as sandboxed functions on our serverless platform - so a slow or crashing skill cannot take the assistant down. Each invocation has a hard timeout (say 8s max, but the assistant may speak a “working on it” if it exceeds the interactive budget) and a circuit breaker per skill.
  • Auth and consent. Skills that touch user data use account linking (OAuth); the router passes only a scoped token, never raw credentials. A skill sees the intent and its own linked account, never the household’s other data.
  • First-party fast path. Built-ins (timers, alarms, weather, general knowledge) are internal services on the hot path with tight SLAs, not webhooks - so the common cases are fast and reliable even if the third-party skill fabric is degraded.
  • Directives back to the device. A skill’s response can include directives: speak this SSML, set a timer locally, play this audio stream URL (music skills return a stream the device fetches directly, so audio does not proxy through us). This keeps large media off the assistant’s path.
fulfill(resolved_intent, session):
  skill = registry.route(resolved_intent, session.enabled_skills)
  if skill.is_first_party:
     result = call_internal(skill, resolved_intent)      # tight SLA
  else:
     token  = auth.scoped_token(session.user, skill)
     result = http_post(skill.endpoint, resolved_intent, token,
                        timeout=interactive_budget, breaker=skill)
  # result = { speech_ssml, directives:[...], card?, keep_session? }
  return result or FALLBACK("Sorry, I couldn't reach that skill.")

The point: capabilities are an open ecosystem, not a monolith - skills register an interaction model and an isolated endpoint, NLU is scoped to the user’s enabled skills, and invocation is a sandboxed, timed, per-skill-authed call with first-party built-ins on a guaranteed fast path - so third parties extend the assistant without being able to slow it down or reach data they were not granted.

5. The latency budget, response generation, and TTS

Naive approach: after fulfillment, generate the full reply text, synthesize the entire audio file with a high-quality neural TTS model, then send it to the device to play.

Where it breaks:

  • TTS is slow to fully render. High-quality neural vocoders take real time proportional to the audio length; waiting for a full 6-second reply to render before sending the first byte adds a second of dead air after everything else already finished.
  • Serial stages. Doing response-gen fully, then TTS fully, then transmit fully, stacks latencies that should overlap.

The answer: template-first NLG with streaming TTS, and an aggressive latency budget enforced across the pipeline.

  • Template NLG for the common case. Most replies are structured (“The high tomorrow in {city} is {temp} degrees”). Filling a template with SSML prosody is instant and predictable; we reserve slower generative NLG (or an LLM) for open-ended answers only, as a fallback path.
  • Streaming TTS. The TTS service synthesizes and streams audio in chunks as it renders, so the device starts playing the first words while the tail is still being generated. “First audio byte” latency - not full-render latency - is the metric, and it is a fraction of the total.
  • Prefetch and cache common audio. Frequent fixed responses (“Okay,” “I’m not sure about that,” earcons, and even common weather phrasings) are pre-synthesized and cached, so many replies start playing with zero synthesis latency.
  • Enforce the budget end to end. Each stage has a slice of the ~1s: if fulfillment is slow, the assistant speaks a holding phrase (“One moment”) streamed from cache while the skill finishes, so the user never hears silence. If a stage times out, degrade to a safe spoken error.
  • Overlap everything. ASR partials feed NLU; NLU pre-warms the skill; the first cached earcon can play the instant an intent is recognized; TTS streams. The one-second wall clock is met by making the stages concurrent, not by making each stage faster.
budget (end-of-speech -> first reply audio, target < 1s):
  network up            ~50 ms   (overlapped: streamed during speech)
  ASR finalize          ~200 ms  (most hidden behind speaking)
  NLU + context         ~50 ms   (speculative on partials)
  fulfillment           ~150 ms  (first-party fast; skill has holding phrase)
  NLG (template)        ~5 ms
  TTS first byte        ~100 ms  (streaming; cached for common replies)
  network down          ~50 ms
  => first audio ~ 200-400 ms after end-of-speech in the common case.

The elegant part: we optimize for time-to-first-audio-byte, not time-to-full-reply - templating the common replies, streaming TTS, caching fixed phrases, and speaking a holding phrase rather than silence when a skill is slow - so the assistant starts talking back almost the instant you stop.

API Design & Data Schema

The interactive path is a bidirectional stream; management and registry are ordinary APIs.

Device <-> Gateway (streaming)

# Persistent bidirectional stream (gRPC / WebSocket) opened on wake.
STREAM /v1/converse
  client -> server frames:
    { type:"start", device_id, language:"en-IN", session_id,
      wake_confidence:0.94 }
    { type:"audio",  seq, opus_bytes }        # repeated, ~100ms each
    { type:"end_of_audio" }                   # device VAD hint (optional)
    { type:"barge_in" }                       # user interrupted reply
  server -> client frames:
    { type:"partial_transcript", text }       # optional, for screens
    { type:"directive", action:"speak", audio_stream_url | inline_audio }
    { type:"directive", action:"set_timer", seconds:300 }
    { type:"directive", action:"play_media", stream_url }   # device fetches
    { type:"session_state", keep_open:true }  # expect a follow-up
    { type:"end" }

Management / registry REST

GET  /v1/devices/{device_id}/settings
  -> { language, wake_word, location, enabled_skills:[...], voice_history:false }

POST /v1/skills/register            # developer console
  body: { invocation_name, interaction_model:{ intents:[...], slots:[...] },
          endpoint_url, oauth_scopes:[...] }
  -> { skill_id, status:"in_review" }

POST /v1/account/{id}/skills/{skill_id}/enable   # links OAuth if needed
  -> { enabled:true, account_linked:true }

# Privacy surface
GET    /v1/account/{id}/voice-history          -> [ {utt_id, transcript, ts} ]
DELETE /v1/account/{id}/voice-history          -> { deleted:true }
POST   /v1/account/{id}/settings/voice-history { enabled:false }

Skill invocation contract (Gateway -> skill endpoint)

POST {skill.endpoint_url}
  headers: { Authorization: Bearer <scoped_token> }
  body: { session_id, user_ctx, intent:"OrderRide",
          slots:{ destination:{value:"airport", resolved:"loc_9931"} },
          dialog_state:"IN_PROGRESS" }
  -> { speech_ssml:"<speak>Your ride is 4 minutes away.</speak>",
       directives:[...], should_end_session:true }

Data stores - the right tool per plane

1. Device / Account Store - sharded KV, key device_id / account_id. Read-mostly settings, language, enabled skills, saved locations, smart-home device registry. Point reads on the hot path (every request loads device context), ~1 TB, no cross-entity transactions - a KV / wide-column store (DynamoDB / Cassandra) sharded by device_id. Relational buys nothing at this read rate.

Table: device_settings          (KV, key = device_id)
  device_id     STRING  PK
  account_id    STRING          INDEX (secondary, account -> devices)
  language      STRING          -- e.g. en-IN, de-DE
  location      GEO             -- for local weather/traffic defaults
  wake_word     STRING
  enabled_skills LIST<skill_id>
  voice_history BOOL            -- privacy opt-in, default false
  Shard by: device_id.

2. Session / Context Store - in-memory KV with TTL, key session_id. Ephemeral multi-turn state: last intent, unfilled slots, recently mentioned entities, pending confirmation. Must be fast and strongly readable within a session, but is disposable - Redis with a short TTL (say 60s idle). Not durable by design.

session[session_id] = {
  account_id, device_id, last_domain, last_intent,
  slots_carry:{ city:"Bangalore" }, pending_confirmation?, ttl:60s }

3. Skill Registry - relational (Postgres), moderate size. Interaction models, endpoints, OAuth scopes, review status, developer ownership. Strong consistency, joins to developer accounts and billing, slow-changing, read-heavy at request time (cache the enabled-skill interaction models per account). Relational is right for the entity relationships; a read cache fronts it.

Table: skills                   (Postgres)
  skill_id      UUID   PK
  invocation_name TEXT UNIQUE
  interaction_model JSONB       -- intents, slot types, sample utterances
  endpoint_url  TEXT
  oauth_scopes  TEXT[]
  status        ENUM(in_review, live, disabled)
  developer_id  UUID   FK
  Index: invocation_name; GIN on interaction_model for intent lookup.

4. Voice History Store - opt-in, tiered. Transcripts (small, hot store, 90-day TTL) and optional audio (large, cold object storage, TTL-purged). Off by default; written only when the account enabled history; user-deletable. Governed by regional residency.

Table: voice_history            (KV transcripts + object store for audio)
  utt_id STRING PK, account_id, transcript, intent, ts, audio_ref?, ttl

5. Model Serving control plane - not a data store but a registry of deployed ASR/NLU/TTS model versions per language, with routing (which model version, which accelerator pool) and canarying. The ASR/TTS accelerators themselves are a stateful compute fleet, not a database.

Why KV/in-memory on the hot paths and relational only for the registry: device context and session state are single-key, ultra-hot, non-transactional reads that shard cleanly and demand low latency - a KV/Redis fit. The only place with real relational structure (skills, developers, review, billing) is moderate-size and slow-changing, so it lives in Postgres behind a read cache. Audio is not a database problem at all; it is transient compute plus opt-in cold storage.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Latency - the whole point, breaks first if stages are serial. Running capture -> upload -> ASR -> NLU -> fulfill -> TTS strictly in sequence blows the one-second budget immediately. Fix: stream and pipeline every stage. ASR runs while the user speaks and emits partials; NLU runs speculatively on partials; TTS streams first-byte and caches common phrases; a holding phrase covers slow skills. Optimize time-to-first-audio, not time-to-full-reply.

2. Concurrent ASR compute - the capacity wall. ~600k concurrent stateful decode sessions pin tens of thousands of accelerators; ASR is the dominant cost and the thing that saturates. Fix: autoscale the ASR accelerator pool on concurrent sessions (not QPS), pack multiple streams per accelerator, use efficient streaming architectures (transducers), and admit-control at the gateway (shed to “I’m having trouble” before the pool melts). Right-size per region to follow the daily wake/sleep demand curve.

3. Hot skills and slow third parties. A viral skill or a slow/broken skill endpoint drags latency and can cascade. Fix: per-skill circuit breakers, hard timeouts, and bulkheads. Third-party skills run isolated (their infra or sandboxed functions), never in the core path; a breaker trips a failing skill to a fallback; the interactive budget is enforced with a holding phrase or graceful failure. First-party built-ins have their own guaranteed fast path.

4. Sharding key. The hot stores shard on device_id (settings) and session_id (context). Fix: shard by device_id / account_id / session_id - all naturally high-cardinality, so load spreads evenly and each request’s context is a single-partition read. No natural hot key exists on the device dimension; skills are read-cached per account, not looked up live per request.

5. Regional latency and residency. A device in India hitting a US data centre adds 200ms+ of pure network, and audio may be legally required to stay in-region. Fix: regional edge gateways and regional processing. Route each device to its nearest region for ASR/NLU/TTS; keep audio and history in-region for data residency. The gateway is stateless and regionally replicated so the hot path stays local; only slow-changing global metadata (skill registry) is replicated across regions.

6. Wake-word false accepts / misses. Too sensitive wastes cloud compute and violates privacy expectations; too dull misses the wake word and frustrates users. Fix: two-stage on-device detection + cloud verification. Stage 1 high-recall on the DSP, stage 2 confirm on-device, optional cloud veto that discards audio for a false wake. Tune thresholds per device model and continuously from (privacy-respecting, opted-in) false-accept telemetry.

7. Model rollout risk. A bad ASR/NLU/TTS model version degrades understanding for millions. Fix: versioned models with canary + shadow deploys in the serving control plane. Roll a new model to 1% of traffic, compare accuracy/latency, and roll back instantly on regression. Keep per-language models independent so one language’s bad model does not affect others.

8. Privacy and history growth. Retained audio is huge and is a liability. Fix: store nothing by default. Audio is transient in memory; only opted-in accounts persist transcripts (small, TTL’d) and optionally audio (cold, TTL-purged). Provide review/delete APIs and honour regional deletion law. The privacy boundary is the device, and retention is opt-in and expiring.

9. Single points of failure / graceful degradation. Fix: gateways stateless and replicated; ASR/NLU/TTS are horizontally scaled pools; session store replicated (loss only drops multi-turn context, not correctness). On a partial outage the assistant degrades to on-device fallbacks (local timers, “I’m having trouble”) rather than hanging. The pipeline is best-effort soft-real-time: a failed stage yields a safe spoken error, never a hang.

Wrap-Up

The trade-offs that define this design:

  • Streaming pipeline over request/response. We refuse to treat this as “upload audio, get JSON.” Every stage streams and overlaps - ASR transcribes while you speak, NLU runs on partials, TTS streams first-byte - trading implementation complexity for the sub-second wall clock that makes the assistant feel alive.
  • Size for concurrent stateful ASR sessions, not QPS. The capacity model is ~600k live decode sessions pinning accelerators, so ASR efficiency and concurrency-based autoscaling dominate the cost and the architecture.
  • On-device wake word over cloud listening. The privacy boundary is drawn in hardware: a two-stage local detector means nothing streams until you say the wake word, with a cloud veto to discard false wakes - the opposite of always streaming.
  • Open skills ecosystem over a monolith. Capabilities are isolated, per-skill-authed, timed webhooks scoped by registered interaction models, with first-party built-ins on a guaranteed fast path - so third parties extend the assistant without slowing it or reaching data they were not granted.
  • Session-scoped context and disposable audio over durable storage. Multi-turn context lives in a short-TTL in-memory store; audio is transient and stored only on explicit opt-in, tiered and TTL-purged - because the safe default for a bedroom microphone is to remember nothing.
  • Time-to-first-audio over time-to-full-reply. Template NLG, streaming and cached TTS, and holding phrases for slow skills mean the assistant starts talking back almost the instant you stop, which is what the whole latency budget is really protecting.

One-line summary: a soft-real-time streaming assistant where a two-stage on-device wake word opens a stream to a regional gateway, a fleet of ~600k concurrent streaming-ASR sessions transcribes while you speak, a hierarchical NLU with session-scoped context resolves intent and slots, an isolated per-skill fulfillment layer with first-party fast paths executes the request, and template NLG plus streaming TTS speaks the reply back in under a second - multi-language, horizontally scaled per region, and private by default because the microphone’s audio never leaves the device until you say the word.