Speculative Decoding: Fast LLM Inference Without Quality Loss › Summary

Speculative decoding accelerates autoregressive generation by parallelizing token verification. A small draft model generates multiple candidate tokens cheaply, which the large target model then verifies in a single forward pass. The technique exploits the memory-bound nature of LLM inference, where processing multiple tokens in one forward pass costs nearly the same as processing one, because the bottleneck is data movement rather than computation.

The key components form an interconnected system. The draft model must be fast and well-aligned with the target, sharing the same vocabulary. The acceptance criterion $\alpha(x) = \min(1, p(x)/q(x))$ determines which draft tokens to accept while preserving the target distribution. The correction distribution $p_{\text{correction}}(x) \propto \max(0, p(x) - q(x))$ handles rejections by sampling only the "missing" probability mass. Together, these components guarantee that the output is statistically identical to what standard autoregressive generation would have produced from the target model.

The acceptance rate $\gamma$ emerges as the critical performance metric. Small improvements in acceptance rate yield disproportionate speedup gains due to the geometric accumulation of consecutive acceptances. Moving from 70% to 80% acceptance roughly doubles the speedup. This nonlinearity motivates significant investment in draft model quality, making draft model alignment the dominant factor in practical deployments.

Draft model selection balances speed against alignment quality. Model family relationships provide the most practical path, with smaller models from the same training pipeline achieving 70-80% acceptance rates on typical text. Distilled models can push this higher at the cost of training infrastructure. The vocabulary compatibility requirement constrains choices to models that share the same tokenizer, effectively tying draft model selection to the target model's provenance.

The verification procedure guarantees lossless acceleration: speculative decoding produces exactly the same output distribution as standard autoregressive generation from the target model. This mathematical guarantee makes speculative decoding uniquely valuable among inference optimization techniques, delivering substantial performance improvements without any sacrifice in generation quality.