Speculative Decoding: Fast LLM Inference Without Quality Loss › Variants and Extensions

The basic speculative decoding algorithm has spawned a family of related techniques that extend the core idea in different directions. Understanding these variants helps you choose the right approach for your specific constraints and objectives.

Self-speculative decoding uses early layers of the target model itself as the draft model. Instead of maintaining two separate models, we perform an early exit from the target model after processing some fraction of the layers, use that intermediate output to generate a draft token, and then complete the full forward pass for verification.

This approach eliminates the vocabulary compatibility problem entirely, since the "draft model" and "target model" share the same architecture and tokenizer. Memory requirements are lower than maintaining two full models, because we only need to store the target model once. The tradeoff is that early-exit drafts tend to have lower alignment with the full model's output than a purpose-trained draft model, since the early layers have not yet developed the full model's representational richness.

The key insight in self-speculative decoding is that the later layers of a deep transformer mostly refine rather than fundamentally change the predictions made by the earlier layers. If the first 20 layers of a 40-layer model produce a probability distribution that is 75% aligned with the full model's output, self-speculation can achieve meaningful speedups without any additional model weights in memory.

Medusa, developed at Princeton, takes a different approach by adding multiple lightweight "heads" on top of the main model, each predicting tokens at different offsets from the current position. The base model runs its forward pass normally, but then several small feed-forward networks predict the next-next token, the next-next-next token, and so on simultaneously. This allows drafting multiple positions in a single pass of the base model.

Tree speculation generalizes this further by exploring multiple draft paths simultaneously rather than one linear sequence. Instead of drafting $[A, B, C, D]$ as a single linear sequence, tree speculation might draft multiple candidate paths in parallel. The target model then verifies all paths simultaneously using tree-structured attention masks, accepting the longest consistent prefix from the most likely path. Tree speculation achieves higher acceptance rates than linear speculation because it hedges against uncertainty at each position, covering more of the probability mass of the target distribution with the same computational budget.

Lookahead decoding takes yet another approach, using the target model itself to generate draft tokens through what are called "n-gram speculation windows." The model generates several candidate n-grams in parallel using a Jacobi iteration process, then verifies and selects the best continuation. This approach requires no separate draft model at all, trading some algorithmic complexity for the elimination of the draft model dependency.

The method works by maintaining a cache of recently generated n-grams, using them as draft proposals for future positions. Since the model generated those n-grams itself in earlier rounds, they have a reasonable chance of being accepted when they occur naturally in the continuation. The acceptance rate depends on the repetitiveness of the generated text: more repetitive text has more applicable cached n-grams and higher acceptance rates.