Speculative Decoding Math: Algorithms & Speedup Limits › Limitations and Practical Considerations

While the mathematics of speculative decoding provides strong theoretical guarantees, several practical challenges affect real-world performance. Understanding these limitations is important for setting realistic expectations and for designing systems that work well despite them.

The acceptance criterion assumes we can efficiently compute both $p(x)$ and $q(x)$ for all tokens. In practice, this means running the full forward pass of both models, even though we only need the probability of specific draft tokens. Some implementations optimize this by caching intermediate states, but the full softmax computation remains a bottleneck. The residual distribution sampling also requires computing the full target distribution, which makes it difficult to apply speculative decoding with certain memory-efficient generation techniques that avoid materializing full vocabulary distributions. Systems that use top-k or nucleus sampling at inference time must be careful to ensure that the acceptance criterion is applied to the full distribution, not the truncated one, or the mathematical guarantees break down.

The i.i.d. assumption in our speedup analysis rarely holds in practice. Acceptance rates vary significantly based on context: function names and common phrases have high acceptance rates, while creative or technical content sees more rejections. This variance means actual speedups may differ from theoretical predictions, and systems should monitor acceptance rates to adapt draft lengths dynamically. When the current generation context has low acceptance rates (perhaps because the model is generating a rare technical term or exploring a creative tangent), the fixed-$k$ analysis overestimates speedup, and adaptive systems that reduce $k$ in these situations will outperform static configurations. Additionally, the tree-structured extensions to speculative decoding (generating multiple candidate continuations) can improve effective acceptance rates but add significant implementation complexity and memory overhead.

Memory constraints present another practical challenge. In single-GPU deployments, both the draft and target models must fit in GPU memory simultaneously. For a 70B target model already consuming nearly all available VRAM, adding even a 7B draft model may require quantization, CPU offloading, or other compromises that affect performance. Multi-GPU deployments solve this by placing the models on separate GPUs, but then the verification step requires communicating logit tensors between devices, adding latency that reduces the effective cost ratio $c$. Production systems at scale must carefully characterize these communication overheads and include them in the cost model used to select optimal $k$.

Hardware considerations also affect practical speedup. The theoretical model assumes draft and target passes don't interfere with each other's memory or compute. On GPUs with limited memory bandwidth, loading both models' weights can create contention. Some deployments use separate GPUs for draft and target models, while others time-multiplex on a single device. Continuous batching systems, which we will explore in the next chapter, add another layer of complexity to speculative decoding integration because batch composition changes between iterations, and the draft and target models may disagree more when generating for diverse batch members simultaneously than when generating for a single request.

There is also a subtle correctness concern when speculative decoding interacts with stateful generation features. Key-value (KV) caches must be managed carefully to ensure that the cached states reflect the accepted tokens, not all the tokens that were proposed. When tokens are rejected and the sequence diverges from the draft, the KV cache must be rolled back or updated accordingly. Implementation bugs in KV cache management can silently corrupt the output distribution without triggering obvious errors, making them particularly dangerous in production systems.