Speculative Decoding

Autoregressive generation, as we covered in Part XXVIII, produces text one token at a time. Each forward pass through a large language model generates a single token, which then becomes input for the next pass. For a 70-billion parameter model generating a 500-token response, this means 500 sequential forward passes through the entire network. The process is slow, not because the GPU lacks computational power, but because it spends most of its time waiting for model weights to load from memory. This fundamental bottleneck has motivated researchers and engineers to find clever ways to extract more tokens from each expensive forward pass through the large model.

Speculative decoding attacks this problem with a counterintuitive insight: what if we could generate multiple tokens per forward pass of the large model? The approach uses a small, fast draft model to speculatively generate several candidate tokens, then verifies them all at once with the large target model. When the draft model's guesses align with what the target model would have produced, we get multiple tokens for the cost of one large-model forward pass. This technique can deliver 2-3x speedups without any approximation or quality loss. The target model's output distribution remains exactly preserved, meaning the text you generate is statistically indistinguishable from what you would have produced through standard autoregressive generation.

Think of speculative decoding as a proofreading partnership. Imagine a junior editor who reads ahead and suggests how the next few sentences might continue. A senior editor then reviews those suggestions at a glance. When the junior editor's intuition aligns with the senior editor's judgment, you advance through several sentences in the time it would have taken the senior editor to write one. When they disagree, you fall back to the senior editor's version, but you still have not lost time, because the review itself was fast. The junior editor corresponds to the small draft model, and the senior editor to the large target model.

What makes speculative decoding particularly remarkable is its mathematical purity: it is not an approximation. Unlike quantization (which reduces numerical precision) or pruning (which removes model parameters), speculative decoding leaves the target model completely unchanged. The output quality guarantee is exact, not approximate. This is a rare property in the world of inference optimization, where most techniques trade some quality for speed. Speculative decoding achieves both simultaneously.

The technique was developed in 2023 by Leviathan et al. (from Google) and independently by Chen et al. (from DeepMind), which speaks to how naturally the idea emerges from first principles once you understand the memory bandwidth bottleneck. It is now deployed by major LLM serving platforms, including production systems at Google and Meta. Anthropic also uses the technique. Understanding the mathematical guarantee behind it connects the practical engineering of fast inference to the theoretical foundations of distribution-preserving acceleration.

To understand why speculative decoding works, we need to understand why autoregressive generation is slow in the first place. On the surface, it might seem that faster hardware is the solution: buy a better GPU, and the problem goes away. But the bottleneck is more fundamental than that. Modern GPUs have enormous computational throughput, yet that capacity sits idle during token generation because the work is not compute-limited. This distinction reveals that the solution to slow inference is not more computational power, but smarter use of the data movement that dominates generation time.

During inference, the GPU must load every model weight from memory for each forward pass. A 70B parameter model in 16-bit precision requires loading 140 GB of data per token generated. The NVIDIA A100's memory bandwidth of 2 TB/s means this takes roughly 70 milliseconds, during which the actual matrix multiplications complete almost instantly. The GPU computes at 312 trillion floating-point operations per second, yet token generation takes 70ms, not because the computations are demanding, but because the data transfer dominates. The GPU spends over 95% of its time waiting for data to arrive from memory, not computing.

This phenomenon is captured by the arithmetic intensity metric, which measures the ratio of floating-point operations to bytes transferred. Training has high arithmetic intensity because gradients and activations reuse the same weights many times across large batch sizes. A single training step processes thousands of examples simultaneously, amortizing the weight loading cost across all of them. Inference has low arithmetic intensity because each token requires loading all weights for minimal computation: one token, one pass, all weights. We say inference is memory-bound rather than compute-bound. This distinction is critical because memory-bound workloads cannot be sped up by adding more computational units. They can only be accelerated by reducing memory transfers or amortizing them across more useful work.

Think of the GPU's relationship to its memory like a factory floor that receives raw materials via a narrow conveyor belt. No matter how many workers you add to the factory floor, production is limited by how fast materials arrive on the belt. The arithmetic intensity measures how efficiently the workers use each load of materials. Training processes materials intensively and reuses them many times. Inference grabs one widget from the belt, performs a trivial operation, and needs the next load immediately.

The arithmetic intensity of 1 FLOP per byte is far below the GPU's operational intensity (ratio of compute to bandwidth), which exceeds 150 FLOPs per byte for the A100. This confirms that inference is deeply memory-bound. We load data constantly but barely compute anything with it. The GPU's thousands of CUDA cores sit idle most of the time, waiting for the next batch of weights to arrive from memory. No amount of additional compute cores changes this situation.

The key insight for speculative decoding emerges from this analysis: if we process multiple tokens simultaneously, we can amortize the memory loading cost. Loading the model weights once and processing 5 tokens takes nearly the same time as processing 1 token, but produces 5x the output. The challenge is that autoregressive generation seems inherently sequential: each token depends on all previous tokens, so we cannot generate token 4 without first knowing tokens 1, 2, and 3. Speculative decoding overcomes this apparent limitation through a clever draft-and-verify strategy that preserves sequential correctness while enabling parallel verification.

Speculative decoding exploits the memory-bandwidth insight through a two-model architecture. A small, fast draft model generates multiple candidate tokens. The large target model then verifies all candidates in a single forward pass. If the draft model's predictions match what the target would have produced, we keep them. If they diverge, we reject the mismatched tokens and use the target model's correction. This approach transforms the problem from "how do we make the large model faster" to "how do we predict what the large model will say, so we can verify those predictions efficiently."

The efficiency argument rests on a simple asymmetry. Generating one draft token with a small model takes roughly 5% of the time that verifying one token with the large model would take. But verifying 5 draft tokens with the large model takes roughly the same time as verifying 1, since verification is a single forward pass through the large model that processes all positions in parallel. So if we generate 5 draft tokens cheaply and then verify them all at once, the combined cost is approximately 1.25x the cost of a single large-model token. If 4 of those 5 tokens are accepted, we generated 4 tokens for the cost of 1.25, which is a 3.2x speedup.

The process works in rounds. Each round proceeds through four phases:

1. Draft phase: The draft model autoregressively generates $K$ candidate tokens (typically $K = 4$ to $8$).
2. Verify phase: The target model processes all $K$ candidates in one forward pass, computing the probability of each candidate given the preceding context.
3. Accept/reject phase: Compare draft and target probabilities to decide which candidates to keep, using a mathematically principled criterion that preserves the target distribution.
4. Correction phase: If a candidate is rejected, sample a correction token from an adjusted distribution that accounts for the tokens already accepted or rejected.

After each round, the process repeats from the new position in the sequence, incorporating any accepted tokens and the correction token into the running context. Generation continues until the desired length is reached or an end-of-sequence token appears.

The approach succeeds because good draft models predict the same tokens the target model would have chosen most of the time. Languages are highly predictable: when you read "The capital of France is", the next word is overwhelmingly "Paris". Both the small draft model and the large target model will assign high probability to "Paris" here, so speculative decoding accepts the draft token with near certainty. Predictable text produces high acceptance rates and large speedups. Ambiguous or creative text produces lower acceptance rates and more modest speedups, but the algorithm remains correct in all cases.

The key efficiency gain comes from the verification phase. When we pass $K$ draft tokens to the target model, we use the parallelism of the transformer architecture. Computing attention over a sequence of $K$ additional tokens requires nearly the same memory bandwidth as computing attention for 1 token, since we load the model weights just once. The computational cost increases linearly with $K$, but as we established, the computational cost is negligible compared to memory transfer. This means that verifying 5 draft tokens costs almost the same wall-clock time as verifying 1 token, creating the opportunity for substantial speedups whenever draft tokens are accepted.

How does the target model verify multiple tokens in one forward pass? The answer lies in the causal attention mask we covered in Part XIII, Chapter 4. This mechanism, originally designed to enable efficient training on entire sequences, turns out to be exactly what we need for parallel verification during inference.

When we feed the sequence $[\text{prompt}, \text{draft}_1, \text{draft}_2, \text{draft}_3, \text{draft}_4]$ to the target model, the causal mask ensures each position only attends to preceding tokens. This creates a natural structure where each position independently computes the probability of its token given everything that came before it, even though all positions are processed simultaneously in one forward pass.

At position $\text{draft}_1$, the model computes $p(\text{draft}_1 \mid \text{prompt})$. At position $\text{draft}_2$, it computes $p(\text{draft}_2 \mid \text{prompt}, \text{draft}_1)$. Each position's output gives us the probability the target model assigns to the draft token at that position, conditioned on everything before it. These computations happen simultaneously in a single forward pass, not sequentially as they would in standard autoregressive generation. The causal mask handles the dependency structure automatically.

The key insight here is that the causal mask is doing double duty. During training, it prevents the model from "cheating" by looking at future tokens. During speculative decoding verification, it enables something remarkable: the model independently verifies each draft token in its correct conditional context without needing to run $K$ separate forward passes. The same architectural decision that enables efficient training also enables efficient verification. This is not a coincidence: both applications require computing $p(x_t \mid x_{<t})$ for all positions $t$ simultaneously.

This parallel verification is what makes speculative decoding efficient. Without it, verifying $K$ tokens would require $K$ forward passes through the target model, eliminating any speedup. With it, we amortize the memory bandwidth cost across $K$ potential tokens. The causal mask ensures that even though we process all positions simultaneously, each position's output depends only on previous positions, maintaining the autoregressive property that makes language model outputs coherent and consistent.

The target model also produces a prediction at the position just after the last draft token. This is a "bonus" token that comes for free from the forward pass. When all $K$ draft tokens are accepted, we can also accept this bonus token, giving us $K+1$ tokens from a single round. This bonus token has zero marginal cost and is always distributed exactly according to the target model, since it is sampled directly from the target's output distribution without any draft or acceptance logic.

The choice of draft model critically affects speculative decoding performance. A good draft model balances three competing requirements: speed, alignment with the target model, and vocabulary compatibility. Getting this balance right is often the most challenging aspect of deploying speculative decoding in practice, and the decisions made here determine whether you achieve 3x speedup or barely exceed 1x.

Think of selecting a draft model as hiring the junior editor from our earlier analogy. You want someone fast enough that their suggestions do not slow down the review process, smart enough that their suggestions are usually right, and trained in the same editorial style as the senior editor. A brilliant but slow junior editor defeats the purpose. A fast but incompetent one generates constant corrections that add overhead without benefit.

The draft model must be fast enough that generating $K$ tokens takes less time than one target model forward pass. If the draft model is too slow, the combined draft-plus-verify time exceeds standard autoregressive generation, producing a slowdown rather than speedup. This constraint places an upper bound on draft model size, typically limiting it to 10-15% of the target model's parameters.

Consider a target model requiring 100ms per forward pass. If we draft $K=5$ tokens and each draft forward pass takes 15ms, drafting costs 75ms. The total round time is 175ms. For this to beat standard generation (100ms per token), we need to accept more than 1.75 tokens per round on average. If our acceptance rate is only 60%, expected tokens per round is roughly 2.0, giving a speedup of approximately 1.14x, which is marginal. At 80% acceptance rate, expected tokens rises to about 3.4, giving roughly 1.94x, a meaningful benefit. The threshold at which speculative decoding becomes worthwhile depends on both the draft-to-target speed ratio and the acceptance rate.

The table shows how strongly acceptance rate affects speedup. At 90% acceptance, we achieve 2.6x speedup. At 50% acceptance, speedup drops to 1.2x. This makes draft model alignment the dominant factor in speculative decoding performance. The time invested in finding or training a well-aligned draft model pays dividends throughout the system's entire deployment lifetime.

A draft model is well-aligned when its probability distribution closely matches the target model's. If the draft model assigns high probability to the same tokens the target model prefers, acceptance rates will be high. If they diverge significantly, most draft tokens will be rejected. Alignment depends on more than getting the top-1 token right: the probability mass needs to be similar across the entire distribution, which determines how often tokens are accepted outright and how well the correction distribution compensates for rejections.

Alignment arises naturally when the draft and target models share training data, architecture style, and training procedure. Models trained on the same corpus with similar objectives will develop similar "linguistic intuitions," agreeing on which continuations are likely even when they differ in their confidence levels. This is why model family relationships are such a practical shortcut.

Common approaches to obtaining aligned draft models include:

• Distillation: Train a small model specifically to match the target model's output distribution. Distillation creates the best alignment but requires training infrastructure and access to the target model's outputs. Acceptance rates above 85% are achievable with careful distillation.
• Model families: Use a smaller model from the same family (for example, LLaMA-7B drafting for LLaMA-70B). Shared training data and similar architectures lead to naturally aligned predictions. This is the most practical approach for most deployments.
• Same-model early exit: Use the first $N$ layers of the target model itself as the draft model, treating intermediate representations as a smaller model's output. This guarantees vocabulary compatibility and achieves reasonable alignment, but requires architectural modifications.

The model family approach is the most practical for deployment. LLaMA-2-7B achieves approximately 70-80% acceptance rates when drafting for LLaMA-2-70B on typical text. Distilled models can achieve 85%+ acceptance rates. The choice between these approaches depends on available resources and the importance of maximizing speedup versus minimizing deployment complexity.

The draft and target models must share exactly the same vocabulary and tokenizer. If they tokenize text differently, the draft tokens cannot be verified by the target model, because the token IDs would refer to different subwords. A token ID of 7342 might mean "Paris" in one tokenizer and "river" in another, so the probabilities are not comparable.

This requirement constrains draft model selection significantly. You cannot use a GPT-2 draft model with a LLaMA target model, even if both might produce similar text. The tokens simply do not correspond. This constraint means that speculative decoding works best within model families that share tokenizers. Organizations with proprietary target models using unique tokenization schemes often need to train custom draft models specifically for that purpose, adding to deployment complexity. This is one reason why large model providers such as Meta have invested in releasing small models alongside their large ones, knowing that the small models serve as both standalone deployments and draft models for the large ones.

The verification procedure is the mathematical heart of speculative decoding. It determines which draft tokens to accept and how to generate corrections when tokens are rejected. We can construct an acceptance criterion that exactly preserves the target model's output distribution while maximizing the number of accepted tokens. This is not an approximation or heuristic: it is a mathematically guaranteed property that makes speculative decoding a lossless acceleration technique.

The mathematical result that makes speculative decoding work is called the acceptance-rejection sampling theorem. In classical statistics, acceptance-rejection sampling is a technique for drawing samples from a target distribution $p$ when you can only sample easily from a proposal distribution $q$. You draw a sample from $q$, then accept it with probability proportional to $p/q$. The accepted samples are guaranteed to be distributed according to $p$. Speculative decoding applies this classical idea to language model token generation, with the draft model as the proposal distribution and the target model as the target distribution.

For each draft token, we compare the draft model's probability $q(x)$ with the target model's probability $p(x)$. We want to accept as many draft tokens as possible while ensuring the accepted tokens are distributed according to $p$, not $q$. The acceptance probability that achieves this is:

where:

• $\alpha(x)$: the probability of accepting candidate token $x$ as a valid sample from the target distribution
• $p(x)$: the probability assigned to token $x$ by the target model given the current context
• $q(x)$: the probability assigned to token $x$ by the draft model given the same context
• The minimum ensures the acceptance probability never exceeds 1, since probabilities must be at most certain

This formula has an elegant interpretation. When $p(x) \geq q(x)$ (the target model likes this token at least as much as the draft model), we always accept. This makes sense: the target model considers this token at least as likely as the draft model proposed, so accepting it only makes our output distribution more aligned with the target. When $p(x) < q(x)$ (the draft model overestimated this token's likelihood), we accept with probability equal to how much it overestimated. The more the draft model overestimated a token, the lower our acceptance probability, preventing us from biasing our output toward tokens the draft model unfairly favored.

The results show that we accept with certainty when the target model prefers a token more than the draft model does (probability 1.00), but only probabilistically when the draft model is overconfident. The "Draft overestimates" scenario, where the draft gives 20% probability but the target gives only 5%, yields a 25% acceptance probability. This means the draft model proposed this token 4x more often than the target would have wanted, so we discard it 75% of the time to prevent it from appearing 4x too frequently in our output.

When a draft token is rejected, we need to sample a correction token. Simply sampling from the target model's distribution at that position would bias the output: we would undersample tokens that were already accepted under the acceptance criterion and oversample tokens that were never drafted. The correction distribution must account precisely for the tokens that have already been "covered" by the acceptance procedure.

To understand why, think about what the acceptance step has already done. Imagine the target distribution assigns 30% probability to token A and 20% to token B. The draft distribution assigns 10% to A and 25% to B. In the acceptance step, token A (if drafted) would be accepted with probability 1.0 (since $p/q = 3.0 > 1$), and token B would be accepted with probability $0.20/0.25 = 0.80$. So token A is well-covered, but token B is slightly over-proposed. If we reject and then sample from $p$ directly, we would add another 30% chance of token A on top of the already-accepted A, distorting the distribution.

The correction distribution adjusts for the tokens that would have been accepted through drafting, sampling only the "residual" probability mass:

where:

• $p_{\text{correction}}(x)$: the adjusted probability of sampling token $x$ as a correction after a rejection
• $p(x) - q(x)$: the difference between target and draft probabilities for token $x$
• $\max(0, \dots)$: selects only tokens where the target assigns more probability than the draft (the "under-sampled" tokens)
• $\sum_{x'} \max(0,\, p(x') - q(x'))$: the normalization constant ensuring the correction distribution sums to 1

This distribution has a clear interpretation. It samples only from tokens where the target model assigns more probability than the draft model proposed. These are precisely the tokens that the draft model under-represented. By sampling from this residual distribution, we fill in the probability mass that was missed by the draft model's proposal. This keeps the combined output of accepted drafts and corrections matches the target distribution exactly. The formal proof of this property shows that the marginal distribution of each generated token equals $p(x)$, even after the acceptance-rejection process.

Notice how the correction distribution emphasizes token 0 and token 2, where the target model assigns more probability than the draft. Token 1 and token 3 receive zero correction probability because the draft model already over-sampled them. By combining accepted draft tokens with correction samples, we ensure the overall distribution perfectly matches what the target model would have produced through standard autoregressive generation. This is not approximately right: it is mathematically exact.

During verification, we process draft tokens sequentially and stop at the first rejection. If draft token 3 is rejected, we do not verify tokens 4 and beyond. This is because the correction at position 3 changes the context, invalidating the draft model's predictions for subsequent positions. The draft model generated tokens 4 and beyond assuming token 3 would be accepted as-is. Once we substitute a correction token, those subsequent predictions become irrelevant because they were conditioned on a different context.

This sequential processing means the number of accepted tokens follows a geometric-like distribution. With acceptance rate $\gamma$ per token, the expected total number of tokens generated per round combines the always-present correction token with the expected accepted drafts:

where:

• $\mathbb{E}[\text{tokens}]$: the expected number of tokens produced per round (drafts plus correction)
• $1$: the correction token, which is always generated regardless of how many drafts are accepted
• $\sum_{k=1}^{K} \gamma^k$: the expected number of accepted draft tokens, summed over each position
• $\gamma$: the probability of accepting a single draft token (the acceptance rate)
• $K$: the number of draft tokens attempted per round

This formula sums the always-present correction token with the geometric series of accepted drafts. The geometric series arises because each subsequent draft token can only be accepted if all previous draft tokens were also accepted. The probability of accepting the first two drafts is $\gamma^2$, the first three is $\gamma^3$, and so on. This cumulative structure explains why acceptance rate has such a dramatic effect on speedup: small improvements in $\gamma$ compound across multiple positions. Moving from $\gamma = 0.7$ to $\gamma = 0.8$ with $K = 5$ increases expected tokens from 3.08 to 3.93, a 28% improvement in throughput from a 14% improvement in acceptance rate.

Let us trace through a complete speculative decoding round numerically to make the abstract procedure concrete. We will use a vocabulary of 5 tokens for simplicity and demonstrate both the acceptance and rejection cases.

Setup. Our vocabulary has tokens $\{A, B, C, D, E\}$ with IDs $\{0, 1, 2, 3, 4\}$. We generate $K = 3$ draft tokens. The target model's probability distribution over the next token given the current context is:

The draft model's distribution is:

Step 1: Draft generation. The draft model samples from $q$ and produces the sequence $[A, B, C]$ with the probabilities it assigned at each step: $q(A) = 0.30$, $q(B) = 0.30$ (given A was sampled), $q(C) = 0.15$ (given A, B were sampled). For simplicity we treat each step as independent draws.

Step 2: Parallel verification. The target model processes $[\text{context}, A, B, C]$ in one forward pass. At each position it computes:

• Position of A: $p(A) = 0.40$
• Position of B: $p(B) = 0.25$
• Position of C: $p(C) = 0.20$

Step 3: Accept/reject for token A. We compute $\alpha(A) = \min(1, p(A) / q(A)) = \min(1, 0.40 / 0.30) = \min(1, 1.33) = 1.0$. We accept token A with certainty because the target model assigned it higher probability than the draft model. The key insight is that when the target agrees or prefers the draft's choice, acceptance is guaranteed.

Step 4: Accept/reject for token B. We compute $\alpha(B) = \min(1, p(B) / q(B)) = \min(1, 0.25 / 0.30) = \min(1, 0.833) = 0.833$. We draw a uniform random number; suppose it comes up 0.72, which is less than 0.833, so we accept token B. Both A and B are now accepted.

Step 5: Accept/reject for token C. We compute $\alpha(C) = \min(1, p(C) / q(C)) = \min(1, 0.20 / 0.15) = \min(1, 1.33) = 1.0$. We accept token C with certainty.

Step 6: Bonus token. All three drafts were accepted. The target model also produced a logit at position after C during the forward pass, so we sample a bonus token directly from that distribution, obtaining token D (say, $p(D) = 0.10$). This token costs nothing extra.

Result of the round. We produced 4 tokens $[A, B, C, D]$ from a single target model forward pass (plus the cheap draft passes). Standard generation would have required 4 target model forward passes.

Rejection scenario. Now suppose the draft produced token B at position 1 with $q(B) = 0.30$, but the target's probability at that position is $p(B) = 0.10$. Then $\alpha(B) = \min(1, 0.10/0.30) = 0.333$. If our random draw is 0.65 (above 0.333), we reject B. We compute the correction distribution:

normalized to give $p_{\text{correction}} = [0.67, 0, 0.33, 0, 0]$. We sample token A with probability 0.67 or token C with probability 0.33. Say we draw A. The round ends with only the accepted tokens from before position 1 plus the correction A. Future drafts will be generated from the corrected context.

This worked example shows both the happy path (high acceptance, multiple tokens per round) and the rejection path (context correction, minimal waste). Even in the rejection case, we still generate one correct token from the target distribution, just as standard generation would have.

The acceptance rate $\gamma$ is the probability that a given draft token is accepted. It is the most important single metric for predicting speculative decoding performance. This metric captures draft model quality and determines whether speculative decoding will provide meaningful speedups in your specific deployment.

Empirical acceptance rate is measured by running speculative decoding on representative text and tracking the fraction of draft tokens that pass the acceptance test. The expected acceptance for each token is $\alpha(x) = \min(1, p(x)/q(x))$, so the overall rate is the average of these values across all drafted tokens.

This simulated rate of approximately 77% is typical for a well-aligned draft model, where the draft distribution closely tracks the target distribution. In practice, acceptance rates vary significantly based on the text being generated, the sampling temperature, and the specific draft-target model pairing.

Several factors influence acceptance rate in practice, and understanding these factors helps you optimize your speculative decoding deployments.

Model family similarity. Draft models from the same family as the target (for example, LLaMA-7B for LLaMA-70B) typically achieve 70-85% acceptance rates. Models from different families may drop to 40-60%. This improvement comes from shared training data, similar architectures, and aligned tokenization schemes that cause both models to "think" in similar ways about the same inputs.

Temperature. Higher sampling temperatures increase randomness, generally reducing acceptance rates. At temperature 0 (greedy decoding), acceptance is deterministic based on whether draft and target agree on the argmax token. As temperature increases, both models become less confident, their probability distributions flatten, and disagreements become more likely even when they agree on the most probable tokens.

Context type. Acceptance rates vary significantly by content. Factual text with predictable continuations (encyclopedic content, structured data, code with common patterns) achieves higher acceptance than creative writing or code with many valid alternatives. When there is a clear "right answer" that both models recognize, acceptance is high. When multiple reasonable continuations exist, the models may prefer different alternatives, lowering acceptance.

Sequence position. Early tokens in a response may have lower acceptance as the models "warm up" to the context. Later tokens often show higher acceptance once both models have established similar interpretations. This pattern suggests that the models' internal representations converge as they process more shared context together.

As temperature increases, the distributions flatten and diverge, causing the acceptance rate to drop significantly. This highlights why speculative decoding is most effective for lower-temperature, more deterministic generation tasks. For applications like creative writing that benefit from higher temperatures, the speedup from speculative decoding will be more modest, though still worthwhile.

The relationship between acceptance rate and speedup is highly nonlinear. Small improvements in acceptance rate yield disproportionate speedup gains because more consecutive tokens pass verification. This nonlinearity arises from the geometric nature of sequential acceptance: improving acceptance from 70% to 80% does not merely improve each token's acceptance by 10%, it dramatically increases the probability of long acceptance runs.

Consider the probability of accepting 5 consecutive draft tokens. At $\gamma = 0.70$, this is $0.70^5 = 0.168$, meaning long runs occur 17% of the time. At $\gamma = 0.80$, it is $0.80^5 = 0.328$, nearly doubling. The entire tail of the distribution shifts significantly with what looks like a modest change in the per-token rate.

The plot demonstrates the importance of draft model alignment. Moving from 70% to 85% acceptance rate approximately doubles the speedup. This nonlinear relationship motivates significant investment in draft model quality. Even modest improvements in alignment translate to substantial real-world performance gains, making draft model optimization valuable for production deployments.

The basic speculative decoding algorithm has spawned a family of related techniques that extend the core idea in different directions. Understanding these variants helps you choose the right approach for your specific constraints and objectives.

Self-speculative decoding uses early layers of the target model itself as the draft model. Instead of maintaining two separate models, we perform an early exit from the target model after processing some fraction of the layers, use that intermediate output to generate a draft token, and then complete the full forward pass for verification.

This approach eliminates the vocabulary compatibility problem entirely, since the "draft model" and "target model" share the same architecture and tokenizer. Memory requirements are lower than maintaining two full models, because we only need to store the target model once. The tradeoff is that early-exit drafts tend to have lower alignment with the full model's output than a purpose-trained draft model, since the early layers have not yet developed the full model's representational richness.

The key insight in self-speculative decoding is that the later layers of a deep transformer mostly refine rather than fundamentally change the predictions made by the earlier layers. If the first 20 layers of a 40-layer model produce a probability distribution that is 75% aligned with the full model's output, self-speculation can achieve meaningful speedups without any additional model weights in memory.

Medusa, developed at Princeton, takes a different approach by adding multiple lightweight "heads" on top of the main model, each predicting tokens at different offsets from the current position. The base model runs its forward pass normally, but then several small feed-forward networks predict the next-next token, the next-next-next token, and so on simultaneously. This allows drafting multiple positions in a single pass of the base model.

Tree speculation generalizes this further by exploring multiple draft paths simultaneously rather than one linear sequence. Instead of drafting $[A, B, C, D]$ as a single linear sequence, tree speculation might draft multiple candidate paths in parallel. The target model then verifies all paths simultaneously using tree-structured attention masks, accepting the longest consistent prefix from the most likely path. Tree speculation achieves higher acceptance rates than linear speculation because it hedges against uncertainty at each position, covering more of the probability mass of the target distribution with the same computational budget.

Lookahead decoding takes yet another approach, using the target model itself to generate draft tokens through what are called "n-gram speculation windows." The model generates several candidate n-grams in parallel using a Jacobi iteration process, then verifies and selects the best continuation. This approach requires no separate draft model at all, trading some algorithmic complexity for the elimination of the draft model dependency.

The method works by maintaining a cache of recently generated n-grams, using them as draft proposals for future positions. Since the model generated those n-grams itself in earlier rounds, they have a reasonable chance of being accepted when they occur naturally in the continuation. The acceptance rate depends on the repetitiveness of the generated text: more repetitive text has more applicable cached n-grams and higher acceptance rates.

This section presents a complete speculative decoding implementation using the Hugging Face transformers library. We will use actual models to demonstrate the concept with measurable results. This implementation captures all the key components: draft generation, parallel verification, and the acceptance-correction procedure.

For demonstration, we use small models that can run on limited hardware. The concepts apply identically to larger production models. The key requirement is that both models share the same vocabulary, which GPT-2 and GPT-2-medium naturally satisfy since they use the same tokenizer.

The core implementation encapsulates the three main operations: drafting candidate tokens, verifying them against the target model, and applying the acceptance/rejection logic. Each method is designed to be modular and clearly reflect the algorithm's structure. Notice how verify_tokens captures the key efficiency property: a single forward pass computes verification probabilities for all $K$ draft tokens simultaneously.

This function orchestrates the complete generation process, repeatedly running speculative decoding rounds until the desired number of tokens is reached. It also tracks statistics that help us evaluate the system's performance in practice.

To measure the benefit of speculative decoding, we need a baseline. This standard autoregressive generation function provides that baseline, using the same target model but generating one token at a time. The comparison between the two approaches reveals the practical speedup factor for your specific model pair and hardware.

Since loading actual models requires significant resources, we demonstrate the workflow with a simulation that captures the key dynamics. This simulation explores how different acceptance rates affect performance without requiring large language model weights.

The simulation confirms our theoretical analysis. At 80% acceptance rate, we achieve approximately 2x speedup, generating over 3 tokens per round on average. This matches well with reported results from production speculative decoding systems, validating that our theoretical framework accurately predicts real-world performance. The standard deviation in speedup also narrows at higher acceptance rates, indicating more consistent performance as draft model quality improves.

Speculative decoding exposes several parameters that you need to tune for your deployment. Getting these right makes the difference between a large speedup and a marginal improvement.

• draft_model: The smaller, faster model used to generate candidate tokens. Ideally from the same model family as the target, with the same tokenizer, and roughly 7-10% the parameter count.
• target_model: The large model used to verify candidates and guarantee output distribution quality. Its output distribution is never modified.
• acceptance_rate ($\gamma$): The probability that a draft token matches the target model's preference. Measured empirically on representative inputs and determines the achievable speedup.
• num_draft_tokens ($K$): The number of candidate tokens generated per round. Typically 4-8. Larger $K$ yields higher expected tokens per round but risks more wasted computation when acceptance rates are low.
• temperature: Controls randomness in sampling. Higher temperatures generally reduce acceptance rates, as discussed above.

Choosing $K$ requires balancing the expected tokens per round against the overhead of drafting. For high acceptance rates (above 80%), $K = 6$ to $8$ is generally optimal. For moderate acceptance rates (60-75%), $K = 4$ to $5$ tends to be better because the probability of long acceptance runs is lower and the overhead of generating many draft tokens is not recouped.

Speculative decoding can reduce latency but comes with limitations that affect deployment decisions. Understanding these tradeoffs helps you decide when and how to apply the technique, and whether the engineering investment is justified for your use case.

The most significant constraint is the requirement for a well-aligned draft model. Finding or training a draft model that achieves 70%+ acceptance rate while being fast enough to provide speedups is non-trivial. For proprietary models without available smaller variants, this can be a blocking issue. Some organizations train dedicated draft models using distillation, but this adds significant infrastructure overhead and requires access to the target model's output distribution during training. The alternative of using early layers of the target model (self-speculative decoding) avoids this problem but requires architectural modifications that may not be available in all frameworks. In practice, many organizations find that the most accessible approach is to wait for the model provider to release a small companion model, which has become standard practice for major open-weight model families but remains rare for proprietary closed models.

Memory requirements increase because both models must be loaded simultaneously. For a 70B target model with a 7B draft model, total GPU memory increases by roughly 10%. On memory-constrained deployments, this overhead may prevent using speculative decoding entirely, or force quantization of one or both models. The interaction between quantization and speculative decoding is an important practical concern: quantizing the draft model more aggressively than the target can preserve generation quality while minimizing the memory overhead of maintaining two models simultaneously. However, aggressive quantization of the draft model may reduce its alignment with the full-precision target model, lowering acceptance rates and partly negating the speedup. Finding the right balance requires empirical tuning.

Batched inference presents complications that can negate speculative decoding benefits. When serving multiple concurrent requests, the sequences in a batch may have different acceptance patterns. One sequence might accept all 5 draft tokens while another rejects after 2. Handling this efficiently requires sophisticated orchestration: either padding shorter accepted sequences (wasting compute on padding), or dynamically adjusting batch composition (adding complexity to the serving infrastructure). The simpler approach of running speculative decoding independently per sequence underutilizes batch parallelism. This is a particularly acute problem for high-throughput serving scenarios, where continuous batching dominates. Speculative decoding is most beneficial for latency-sensitive applications serving individual requests, rather than throughput-optimized batch serving.

The technique is also less effective for certain content types. When generating highly creative text, code with many valid alternatives, or responses in low-resource languages where the draft model has weaker coverage, acceptance rates may fall below the break-even threshold. Production systems often implement adaptive strategies that monitor acceptance rates in real time and fall back to standard generation when rates drop too low. This keeps speculative decoding never becomes a net negative.

Despite these limitations, speculative decoding has become a standard technique in production LLM serving. The 2-3x speedups it provides translate directly to reduced latency for users and reduced cost for providers. For conversational applications where response time critically affects user experience, shaving 50-70% off generation time is substantial. The technique also demonstrates a broader principle: the memory-bound nature of LLM inference creates opportunities for algorithmic improvements that do not require hardware upgrades or model architecture changes. Speculative decoding preserves the exact output distribution of the target model while improving efficiency, whereas most optimization techniques trade quality for performance.

Speculative decoding accelerates autoregressive generation by parallelizing token verification. A small draft model generates multiple candidate tokens cheaply, which the large target model then verifies in a single forward pass. The technique exploits the memory-bound nature of LLM inference, where processing multiple tokens in one forward pass costs nearly the same as processing one, because the bottleneck is data movement rather than computation.

The key components form an interconnected system. The draft model must be fast and well-aligned with the target, sharing the same vocabulary. The acceptance criterion $\alpha(x) = \min(1, p(x)/q(x))$ determines which draft tokens to accept while preserving the target distribution. The correction distribution $p_{\text{correction}}(x) \propto \max(0, p(x) - q(x))$ handles rejections by sampling only the "missing" probability mass. Together, these components guarantee that the output is statistically identical to what standard autoregressive generation would have produced from the target model.

The acceptance rate $\gamma$ emerges as the critical performance metric. Small improvements in acceptance rate yield disproportionate speedup gains due to the geometric accumulation of consecutive acceptances. Moving from 70% to 80% acceptance roughly doubles the speedup. This nonlinearity motivates significant investment in draft model quality, making draft model alignment the dominant factor in practical deployments.

Draft model selection balances speed against alignment quality. Model family relationships provide the most practical path, with smaller models from the same training pipeline achieving 70-80% acceptance rates on typical text. Distilled models can push this higher at the cost of training infrastructure. The vocabulary compatibility requirement constrains choices to models that share the same tokenizer, effectively tying draft model selection to the target model's provenance.

The verification procedure guarantees lossless acceleration: speculative decoding produces exactly the same output distribution as standard autoregressive generation from the target model. This mathematical guarantee makes speculative decoding uniquely valuable among inference optimization techniques, delivering substantial performance improvements without any sacrifice in generation quality.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about speculative decoding.