Latency Optimization › Speculative Decoding

The core latency bottleneck during decode is that each token requires a full forward pass through the model. Speculative decoding breaks this constraint by using a small, fast draft model to generate candidate tokens cheaply, then verifying multiple candidates with the large target model in parallel.

The intuition behind speculative decoding is straightforward. Most of the time, the next few tokens in a generation are highly predictable: function words like "the", "is", "of", high-frequency phrases, and continuations that any reasonable language model would select. A small model can predict these tokens quickly. If the large model would have chosen the same tokens, you get those tokens for free. If the small model was wrong, you fall back to the large model's choice and continue. In expectation, you generate multiple tokens per large-model forward pass rather than exactly one.

Given a target model $M_q$ and a draft model $M_p$:

1. The draft model generates $k$ candidate tokens $\tilde{x}_1, \tilde{x}_2, \ldots, \tilde{x}_k$ autoregressively. This is fast because the draft model is much smaller.

2. The target model performs a single forward pass over the $k$ draft tokens simultaneously (treating them like a prefill), computing the probability distribution $q(\cdot | x_{<t+i})$ for each position.

3. Each draft token $\tilde{x}_i$ is accepted with probability:

$$P(\text{accept} \;\; \tilde{x}_i) = \min\!\left(1, \frac{q(\tilde{x}_i \,|\, x_{<t+i})}{p(\tilde{x}_i \,|\, x_{<t+i})}\right)$$

where:

• $q(\tilde{x}_i \,|\, x_{<t+i})$: probability assigned to the draft token by the target model
• $p(\tilde{x}_i \,|\, x_{<t+i})$: probability assigned by the draft model
• $\min(1, \cdot)$: a rejection sampling correction that guarantees the accepted tokens follow the exact target distribution

1. If a token is rejected, the target model samples a corrected token at that position, and decoding restarts from there. The key guarantee is that the final output distribution is mathematically identical to sampling directly from the target model, token by token.

The acceptance probability formula has a clean probabilistic interpretation. If the target model assigns higher probability to a draft token than the draft model does ($q > p$), the token is accepted with probability 1: the small model was being conservative about a token the large model is confident in, so there is no downside to accepting. If the target model assigns lower probability ($q < p$), the token is accepted only with probability $q/p$: the small model was overconfident, and we accept in proportion to how plausible the target model finds the token. The rejection step, when it occurs, samples from a residual distribution that corrects for the excess probability the draft model placed on incorrect tokens.

The expected number of accepted tokens per target model call, called the acceptance length, determines the speedup. If the acceptance length is $\bar{k}$, then speculative decoding reduces the number of target model calls needed for $N_{\text{out}}$ tokens by a factor of $\bar{k}$, at the cost of running the draft model for the same tokens. The net speedup depends on the ratio of draft model cost to target model cost and the acceptance length achieved.

The heatmap reveals the regime where speculative decoding helps most: high acceptance rates (the draft model frequently matches the target) combined with a cheap draft model (low draft-to-target cost ratio). In practice, acceptance rates of 60-80% are typical when the draft model is a smaller member of the same model family, and draft models are often 10-20x cheaper than the target, placing most real systems well into the beneficial zone.

The draft model must be fast (to minimize the cost of generating $k$ candidates) and accurate (to maximize acceptance rate). The most common approaches are:

• Smaller model from the same family: A 7B model drafting for a 70B model typically achieves higher acceptance rates than an unrelated model because they were trained on the same data and share similar inductive biases.
• Distilled draft model: A model trained specifically to mimic the target model's distribution, which can achieve high acceptance rates at very small model sizes.
• Self-speculative decoding: Drafts are generated using the first few layers of the target model and then verified by the full model. This requires no separate draft model but introduces complexity in the forward pass implementation.
• Lookahead decoding: Generates draft tokens using n-gram patterns from the current context, requiring no model at all for the draft step. Acceptance rates are lower but draft cost approaches zero.

The choice of draft model strategy depends heavily on the deployment context. If you are already running a smaller model for other purposes, using it as a draft model adds nearly zero marginal cost. If the serving system is memory-constrained (a common situation when running very large models on limited GPU counts), self-speculative decoding avoids the memory overhead of loading a separate draft model while still providing latency improvements. Lookahead decoding is most effective for tasks like code generation where n-gram repetition is high, and least effective for open-ended creative generation where context repetition is rare.

The number of draft tokens $k$ is also a tunable parameter. More draft tokens mean larger potential speedups when acceptance rates are high, but the benefit saturates because the probability of an unbroken run of $k$ accepted tokens falls as $\alpha^k$ for a geometric acceptance model. A typical value of $k = 4$ to $8$ captures most of the benefit without excessive overhead.

Speculative decoding is an exact algorithm: the output token distribution is identical to running the target model alone. This follows from the rejection sampling correction. When a draft token is accepted, it is a sample from $q$ (the target) because the acceptance criterion reweights the proposal $p$ to match $q$. When a draft token is rejected, the target model samples directly from $q$. In both cases, the accepted sequence has the same distribution as direct sampling from $M_q$.

This guarantee distinguishes speculative decoding from approximations like quantization or pruning. Speculative decoding never degrades output quality, only output speed. This is a remarkable property: you get a latency reduction for free, with no quality-speed trade-off. The only cost is compute efficiency at the system level: the draft model consumes resources, and rejected draft tokens represent wasted compute. This is why speculative decoding is most attractive when a single request occupies dedicated hardware, and less attractive when the system is already at full capacity.

Beyond the textbook algorithm, production deployments of speculative decoding require handling several practical concerns. The draft model and target model must use compatible tokenization so that draft tokens correspond to positions in the target model's vocabulary. The draft model's KV cache must be managed separately from the target model's, adding memory overhead. And the serving framework must be able to execute the draft generation and target verification steps efficiently, ideally as consecutive GPU kernel launches without returning to the CPU between steps.

Modern serving frameworks like vLLM provide first-class support for speculative decoding, handling these details transparently. For practitioners, the main decision is selecting the draft model and the speculation depth $k$, then measuring whether the acceptance rate on your specific workload is high enough to justify the overhead. If your users are asking questions with predictable, formulaic answers, speculative decoding is likely beneficial. If they are asking for highly creative or unpredictable generations, acceptance rates may be too low for speculative decoding to help.