Speculative Decoding: Fast LLM Inference Without Quality Loss › Speculative Decoding

Autoregressive generation, as we covered in Part XXVIII, produces text one token at a time. Each forward pass through a large language model generates a single token, which then becomes input for the next pass. For a 70-billion parameter model generating a 500-token response, this means 500 sequential forward passes through the entire network. The process is slow, not because the GPU lacks computational power, but because it spends most of its time waiting for model weights to load from memory. This fundamental bottleneck has motivated researchers and engineers to find clever ways to extract more tokens from each expensive forward pass through the large model.

Speculative decoding attacks this problem with a counterintuitive insight: what if we could generate multiple tokens per forward pass of the large model? The approach uses a small, fast draft model to speculatively generate several candidate tokens, then verifies them all at once with the large target model. When the draft model's guesses align with what the target model would have produced, we get multiple tokens for the cost of one large-model forward pass. This technique can deliver 2-3x speedups without any approximation or quality loss. The target model's output distribution remains exactly preserved, meaning the text you generate is statistically indistinguishable from what you would have produced through standard autoregressive generation.

Think of speculative decoding as a proofreading partnership. Imagine a junior editor who reads ahead and suggests how the next few sentences might continue. A senior editor then reviews those suggestions at a glance. When the junior editor's intuition aligns with the senior editor's judgment, you advance through several sentences in the time it would have taken the senior editor to write one. When they disagree, you fall back to the senior editor's version, but you still have not lost time, because the review itself was fast. The junior editor corresponds to the small draft model, and the senior editor to the large target model.

What makes speculative decoding particularly remarkable is its mathematical purity: it is not an approximation. Unlike quantization (which reduces numerical precision) or pruning (which removes model parameters), speculative decoding leaves the target model completely unchanged. The output quality guarantee is exact, not approximate. This is a rare property in the world of inference optimization, where most techniques trade some quality for speed. Speculative decoding achieves both simultaneously.

The technique was developed in 2023 by Leviathan et al. (from Google) and independently by Chen et al. (from DeepMind), which speaks to how naturally the idea emerges from first principles once you understand the memory bandwidth bottleneck. It is now deployed by major LLM serving platforms, including production systems at Google and Meta. Anthropic also uses the technique. Understanding the mathematical guarantee behind it connects the practical engineering of fast inference to the theoretical foundations of distribution-preserving acceleration.