Speculative Decoding Is the Inference Trick That Cuts LLM Latency Without Changing the Model

Most self-hosted LLM teams respond to slow generation the same way: buy a bigger GPU. Latency is high, tokens per second is low, so the fix must be more VRAM and more FLOPs. I have watched teams move from an A100 to an H100 to an H200, double their monthly bill, and get a 15 percent improvement in wall-clock latency for a chat endpoint.

The uncomfortable truth is that token generation on a transformer is not compute-bound. It is memory-bound. You are not waiting on the GPU to multiply matrices. You are waiting on the GPU to read the model weights out of HBM, over and over, once per token. Speculative decoding attacks exactly that bottleneck, and it does it without touching your model weights, your fine-tune, or your output distribution. Done right it cuts first-token-adjacent latency and end-to-end generation time by 40 to 60 percent on the workloads where it fits.

This post explains why decoding is slow, how speculative decoding beats that limit, how to pick a draft model that actually helps, and the vLLM configuration to run it in production.

Why single-token decoding wastes your GPU

Autoregressive generation produces one token at a time. To produce token N+1, the model needs token N, which needed token N-1. It is a strict sequential dependency. You cannot parallelize across the time dimension the way you can during the prefill of the prompt.

Here is the part people miss. Generating one token requires a full forward pass, which means reading every weight in the model from HBM into the compute units. For a 70B model in FP16 that is 140GB of weights streamed per token. On an H100 with roughly 3.3TB/s of memory bandwidth, just moving those weights sets a floor of about 42ms per token, before any actual math. The matrix multiplies for a single token are tiny by comparison. The arithmetic units sit mostly idle.

Naive decoding (batch size 1):

  step 1: read 140GB weights -> compute 1 token   [GPU ~5% utilized]
  step 2: read 140GB weights -> compute 1 token   [GPU ~5% utilized]
  step 3: read 140GB weights -> compute 1 token   [GPU ~5% utilized]
  ...

  Bottleneck: HBM bandwidth, not FLOPs.

This is why a single request barely touches GPU utilization. The same forward pass that produces one token for one request could produce one token for 200 requests at almost the same latency, because the expensive part (reading weights) is shared. That is the insight behind continuous batching, and it is great for throughput. It does nothing for the latency of a single request, which is what your user feels.

So the question becomes: can we get more than one token out of a single weight-reading pass, for a single request? That is speculative decoding.

The core idea: guess cheaply, verify in parallel

Speculative decoding splits generation into two roles.

A small, fast draft model proposes a short run of tokens, say the next 4 or 5, one at a time. The draft model is small enough that running it 5 times is still cheap, because reading a 1B model’s weights 5 times is far less memory traffic than reading the 70B model’s weights once.

Then the large target model takes those 5 proposed tokens and verifies all of them in a single forward pass. This is the key move. A transformer can score many positions in parallel in one pass. The large model reads its weights once and, instead of producing just the next token, it checks whether it would have produced each of the draft’s tokens.

The verification uses a rejection-sampling scheme that is provably distribution-preserving. The output you get is sampled from exactly the target model’s distribution. Not an approximation of it. This is the property that makes speculative decoding safe to turn on in production: your model’s behavior does not change, only its speed.

Speculative step (gamma = 4):

  draft model:   proposes  t1 t2 t3 t4      [4 cheap forward passes on a tiny model]
  target model:  verifies [t1 t2 t3 t4] + 1 [ONE forward pass on the big model]

  accepted: t1 t2 t3  -> reject t4, resample from target
  result:   3 tokens emitted from ONE big-model weight read

  Effective speedup = accepted tokens per big-model pass.

If the draft model is a good mimic, the target accepts most of its guesses. Say it accepts 3 of the 4 proposed tokens on average. You just produced roughly 3 tokens for the price of one big-model forward pass plus a handful of tiny draft passes. That is the 2x to 3x throughput and the 40 to 60 percent latency reduction in one move.

The acceptance walk, precisely

For each proposed token, you compare the target model’s probability q(x) to the draft model’s probability p(x) for that token:

• If q(x) >= p(x), accept the token.
• If q(x) < p(x), accept it with probability q(x) / p(x). On rejection, stop and resample the current position from the adjusted distribution norm(max(0, q - p)).

Because verification happens left to right and stops at the first rejection, you keep the longest correct prefix the draft got right, then the target fills in one guaranteed-correct token at the rejection point. Even a fully rejected run still produces one token, so you never go slower than naive decoding in terms of tokens emitted per big-model pass. You only pay the small draft overhead.

What actually determines the speedup

The entire economics come down to one number: the acceptance rate, usually called alpha. It is the probability the target accepts a proposed token. From it you can estimate expected tokens per verification pass. For a proposal length gamma, the expected number of accepted-plus-one tokens is:

E[tokens] = (1 - alpha^(gamma+1)) / (1 - alpha)

Plug in numbers and the behavior is intuitive:

Two lessons jump out. First, acceptance rate dominates everything. A draft model that the target agrees with 90 percent of the time is worth far more than a longer proposal window. Second, pushing gamma higher has sharply diminishing returns once alpha is modest, because a single early rejection throws away the rest of the window and you still paid to draft all of it. In practice gamma between 3 and 5 is the sweet spot for most models.

The catch nobody mentions in the benchmark blog posts: these are expected tokens per big-model pass, not free tokens. The draft model has its own latency, and verification of a longer window is slightly more expensive than verifying one token. The real speedup is:

speedup ~= E[accepted tokens] / (1 + gamma * (draft_cost / target_cost) + verify_overhead)

If your draft model is 1/20th the size of your target, draft_cost / target_cost is small and the denominator stays near 1. If you got greedy and picked a draft model that is 1/3 the size of the target, the draft passes eat most of the savings even at a high acceptance rate. This is the single most common way teams make speculative decoding slower than plain decoding.

Choosing the draft model

The draft model has two jobs that pull in opposite directions. It must be cheap to run, and it must predict the same tokens the target would. Pick for one and you lose the other.

Three practical selection criteria, in order of importance:

1. Same tokenizer and vocabulary as the target. This is non-negotiable for classic speculative decoding. The rejection test compares probabilities token-for-token, so the two models must speak the same token IDs. In practice this means your draft comes from the same model family: Llama 3.1 8B drafting for Llama 3.1 70B, Qwen2.5 1.5B drafting for Qwen2.5 72B. A draft from a different family with a different tokenizer cannot be used directly.

2. Size ratio around 1/10 to 1/30 of the target. Small enough that gamma draft passes are cheap, large enough that it still mimics the target well. For a 70B target, a 1.5B to 8B draft is the usual range. Below roughly 1B the acceptance rate often falls far enough that the extra draft passes stop paying off on hard content like code or math.

3. Distributional alignment with your actual traffic. Acceptance rate is measured on your workload, not on a generic benchmark. A draft that accepts at 85 percent on English chat may collapse to 55 percent on JSON or code, where the target is pickier and the token space is less predictable. Always measure alpha on a sample of your real prompts before committing.

A few alternatives to a separate draft model, which matter because they sidestep the “do I even have a small model in this family” problem:

Prompt-lookup decoding deserves a special mention. For workloads where the output repeats large chunks of the input, like summarization, retrieval-augmented answers, or diff generation, you do not need a draft model at all. You draft by searching the prompt for the current suffix and copying the continuation. It costs almost nothing and the acceptance rate on those workloads is high because the model really does echo its input. If your traffic is input-heavy, try this before you go find a draft model.

Running it in vLLM

vLLM supports speculative decoding and handles the hard parts: the rejection sampler, continuous batching interaction, and KV cache management for the draft and target together. You turn it on through the speculative_config object.

A separate draft model from the same family:

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,
    speculative_config={
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 4,   # this is gamma
    },
    gpu_memory_utilization=0.90,
)

params = SamplingParams(temperature=0.7, max_tokens=512)
out = llm.generate(["Explain speculative decoding in two sentences."], params)
print(out[0].outputs[0].text)

Serving the same thing from the OpenAI-compatible server:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --speculative-config '{"model": "meta-llama/Llama-3.1-8B-Instruct", "num_speculative_tokens": 4}'

The zero-dependency option, prompt-lookup decoding, needs no second model:

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,
    speculative_config={
        "method": "ngram",
        "num_speculative_tokens": 5,
        "prompt_lookup_max": 4,   # longest n-gram to match from the prompt
        "prompt_lookup_min": 2,
    },
)

Three configuration details that decide whether this helps or hurts in production:

Draft tokens interact with batch size. Speculative decoding spends GPU compute to buy latency. At low concurrency the GPU is idle and that spare compute is free, so speedups are large. As your batch fills up, the GPU becomes genuinely compute-bound from batching alone, and the extra verification work competes with real requests. vLLM will scale back speculation under load, and that is correct behavior, but it means your latency win shrinks exactly when traffic is high. Benchmark at your real concurrency, not at batch size 1.

KV cache for two models. The draft model needs its own KV cache alongside the target’s. That is extra VRAM. If you were already near the memory ceiling, enabling speculation can trigger more aggressive preemption or reduce your max batch size. Budget for it. Leave headroom in gpu_memory_utilization.

Temperature and acceptance. Higher sampling temperature flattens the target distribution, which generally raises acceptance because the target is more forgiving about which token comes next. Greedy or very low temperature decoding is pickier and can lower alpha. If you run near-greedy for determinism, measure your actual acceptance before assuming the benchmark numbers apply.

Measuring whether it worked

Do not trust the defaults and do not trust someone else’s benchmark. Turn it on, turn it off, and compare two numbers on your own traffic:

• Inter-token latency (time per output token) at your p50 and p95 concurrency.
• Accepted tokens per step, which vLLM exposes in its metrics. This is your observed alpha in disguise.

If accepted tokens per step is comfortably above 2, speculation is paying off. If it hovers near 1, your draft model is a poor match for your workload and you are better off turning it off, because you are paying draft overhead for almost no acceptances. That single metric tells you more than any blog table, including this one.

A rough decision flow:

Is your workload input-heavy (summarize, RAG, code edit)?
   |-- yes -> try prompt-lookup (ngram) first. Free. Often enough.
   |-- no  -> is there a good small model in the target's family?
               |-- yes -> separate draft model, gamma = 4, measure alpha
               |-- no  -> consider EAGLE/Medusa heads, or skip speculation

Honest assessment

What works: speculative decoding is one of the few optimizations that reduces single-request latency without degrading output quality, without retraining, and without buying hardware. On low-to-medium concurrency chat and code workloads with a well-matched draft model, 2x to 3x faster token generation is realistic and I have seen it hold up in production. Prompt-lookup decoding on input-heavy workloads is close to a free win and criminally underused.

What does not work as advertised: the big speedup numbers come from batch size 1 benchmarks. Under heavy concurrent load the GPU is already busy and the win compresses, sometimes to nearly nothing. A draft model that is too large, from the wrong family, or poorly aligned with your traffic can make you slower than plain decoding while using more VRAM. And it does nothing for prompt processing latency, so if your bottleneck is a 30K-token prompt, this is not your fix.

What to actually do: before you approve the GPU upgrade, measure inter-token latency on your real traffic, then enable speculative decoding with a same-family draft model at gamma 4 and measure again. Watch accepted-tokens-per-step. If it is above 2, you just cut your latency for the cost of a config change. If your workload reuses its input heavily, try prompt-lookup first because it costs you nothing. The upgrade can wait until you have proven you are actually compute-bound, which, for single-request latency, you usually are not.