Speculative Decoding: Fast LLM Inference Without Quality Loss › Draft Model Selection

The choice of draft model critically affects speculative decoding performance. A good draft model balances three competing requirements: speed, alignment with the target model, and vocabulary compatibility. Getting this balance right is often the most challenging aspect of deploying speculative decoding in practice, and the decisions made here determine whether you achieve 3x speedup or barely exceed 1x.

Think of selecting a draft model as hiring the junior editor from our earlier analogy. You want someone fast enough that their suggestions do not slow down the review process, smart enough that their suggestions are usually right, and trained in the same editorial style as the senior editor. A brilliant but slow junior editor defeats the purpose. A fast but incompetent one generates constant corrections that add overhead without benefit.

The draft model must be fast enough that generating $K$ tokens takes less time than one target model forward pass. If the draft model is too slow, the combined draft-plus-verify time exceeds standard autoregressive generation, producing a slowdown rather than speedup. This constraint places an upper bound on draft model size, typically limiting it to 10-15% of the target model's parameters.

Consider a target model requiring 100ms per forward pass. If we draft $K=5$ tokens and each draft forward pass takes 15ms, drafting costs 75ms. The total round time is 175ms. For this to beat standard generation (100ms per token), we need to accept more than 1.75 tokens per round on average. If our acceptance rate is only 60%, expected tokens per round is roughly 2.0, giving a speedup of approximately 1.14x, which is marginal. At 80% acceptance rate, expected tokens rises to about 3.4, giving roughly 1.94x, a meaningful benefit. The threshold at which speculative decoding becomes worthwhile depends on both the draft-to-target speed ratio and the acceptance rate.

The table shows how strongly acceptance rate affects speedup. At 90% acceptance, we achieve 2.6x speedup. At 50% acceptance, speedup drops to 1.2x. This makes draft model alignment the dominant factor in speculative decoding performance. The time invested in finding or training a well-aligned draft model pays dividends throughout the system's entire deployment lifetime.

A draft model is well-aligned when its probability distribution closely matches the target model's. If the draft model assigns high probability to the same tokens the target model prefers, acceptance rates will be high. If they diverge significantly, most draft tokens will be rejected. Alignment depends on more than getting the top-1 token right: the probability mass needs to be similar across the entire distribution, which determines how often tokens are accepted outright and how well the correction distribution compensates for rejections.

Alignment arises naturally when the draft and target models share training data, architecture style, and training procedure. Models trained on the same corpus with similar objectives will develop similar "linguistic intuitions," agreeing on which continuations are likely even when they differ in their confidence levels. This is why model family relationships are such a practical shortcut.

Common approaches to obtaining aligned draft models include:

• Distillation: Train a small model specifically to match the target model's output distribution. Distillation creates the best alignment but requires training infrastructure and access to the target model's outputs. Acceptance rates above 85% are achievable with careful distillation.
• Model families: Use a smaller model from the same family (for example, LLaMA-7B drafting for LLaMA-70B). Shared training data and similar architectures lead to naturally aligned predictions. This is the most practical approach for most deployments.
• Same-model early exit: Use the first $N$ layers of the target model itself as the draft model, treating intermediate representations as a smaller model's output. This guarantees vocabulary compatibility and achieves reasonable alignment, but requires architectural modifications.

The model family approach is the most practical for deployment. LLaMA-2-7B achieves approximately 70-80% acceptance rates when drafting for LLaMA-2-70B on typical text. Distilled models can achieve 85%+ acceptance rates. The choice between these approaches depends on available resources and the importance of maximizing speedup versus minimizing deployment complexity.

The draft and target models must share exactly the same vocabulary and tokenizer. If they tokenize text differently, the draft tokens cannot be verified by the target model, because the token IDs would refer to different subwords. A token ID of 7342 might mean "Paris" in one tokenizer and "river" in another, so the probabilities are not comparable.

This requirement constrains draft model selection significantly. You cannot use a GPT-2 draft model with a LLaMA target model, even if both might produce similar text. The tokens simply do not correspond. This constraint means that speculative decoding works best within model families that share tokenizers. Organizations with proprietary target models using unique tokenization schemes often need to train custom draft models specifically for that purpose, adding to deployment complexity. This is one reason why large model providers such as Meta have invested in releasing small models alongside their large ones, knowing that the small models serve as both standalone deployments and draft models for the large ones.