Speculative Decoding Math: Algorithms & Speedup Limits › Draft Quality Effects

The acceptance probability $\alpha$ directly depends on how well the draft model approximates the target model. Understanding this relationship helps us choose appropriate draft models and predict performance. The choice of draft model is a necessary practical decision in deploying speculative decoding, and the mathematical connection between distributional similarity and acceptance rate gives us a principled framework for that choice.

Think of the draft model as a stand-in actor rehearsing the lines of the lead performer. The more similar their performance style, the less the director (target model) needs to intervene with corrections. If the stand-in naturally speaks in the same rhythm and vocabulary as the lead, most rehearsal takes place without correction. If their styles differ substantially, constant intervention is needed and the rehearsal provides little benefit over having the lead perform every scene directly.

The mathematical connection between draft quality and speedup is direct and quantifiable. We do not need to guess how much a better draft model will improve throughput. The formula $S = \mathbb{E}[N] / (1 + k/c)$ with $\mathbb{E}[N] = (1 - \alpha^{k+1})/(1-\alpha)$ tells us exactly how changes in $\alpha$ translate to changes in speedup. A draft model that improves $\alpha$ from 0.7 to 0.8 with $k=5$ and $c=10$ increases $\mathbb{E}[N]$ from about 2.84 to 3.28 tokens, translating to a speedup increase from roughly $1.89\times$ to $2.19\times$, a 16% improvement from a single quality metric increase.

Draft quality can be quantified by the expected acceptance probability:

$$\alpha = \sum_x \min(q(x), p(x))$$

where:

• $\alpha$: measure of draft quality (higher is better)
• $q(x)$: draft distribution
• $p(x)$: target distribution

This quantity has a clear interpretation: it is the probability mass that the two distributions share. A high-quality draft model assigns similar probabilities to most tokens as the target model, resulting in large overlap. A poor draft model might assign high probability to tokens the target model dislikes, or vice versa, resulting in small overlap.

This is related to the total variation distance between distributions, one of the most basic measures of dissimilarity between probability distributions. Total variation distance measures the maximum probability that an observer could assign to an event using one distribution minus the probability they would assign using the other. It is a tight bound on how distinguishable two distributions are:

$$\begin{aligned} TV(p, q) &= \frac{1}{2}\sum_x |p(x) - q(x)| && \text{(definition of TV distance)} \\ &= \frac{1}{2}\sum_x (p(x) + q(x) - 2\min(p(x), q(x))) && \text{(algebraic identity)} \\ &= \frac{1}{2}\left(\sum_x p(x) + \sum_x q(x) - 2\sum_x \min(p(x), q(x))\right) && \text{(split sums)} \\ &= \frac{1}{2}(1 + 1 - 2\alpha) && \text{(probabilities sum to 1)} \\ &= 1 - \alpha && \text{(simplify)} \end{aligned}$$

where:

• $TV(p, q)$: total variation distance between target and draft distributions
• $p(x)$: target distribution probability for token $x$
• $q(x)$: draft distribution probability for token $x$
• $|p(x) - q(x)|$: absolute difference in probability for token $x$
• $\alpha$: acceptance probability (equals $1 - TV(p,q)$)

The relationship $\alpha = 1 - TV(p, q)$ is remarkably clean. It says that the acceptance probability is exactly one minus the total variation distance between the draft and target distributions. Higher draft quality means smaller total variation distance, which means higher acceptance probability. This connection to total variation distance indicates that the acceptance probability captures the most basic notion of distributional similarity: the maximum probability that an adversary could distinguish between samples from the two distributions.

In practical terms, this connection is useful for intuition. Total variation distance of 0.2 means acceptance probability of 0.8. It also means that if you drew a sample from either distribution at random, an optimal observer could identify which distribution it came from with probability at most 0.2 better than chance. This is the sense in which the draft and target models are "close" when acceptance rates are high.

Larger draft models tend to produce distributions closer to the target, increasing $\alpha$. However, larger models are slower, decreasing $c$. The optimal draft model balances these factors, and finding this balance requires understanding the tradeoffs involved. This is one of the key engineering decisions in deploying speculative decoding at scale.

Consider three scenarios that illustrate how the quality-speed tradeoff plays out in practice:

• Very small draft model: Fast inference ($c$ large) but poor distribution match ($\alpha$ small). We accept few tokens per iteration, limiting speedup. As a concrete example, imagine using a tiny 125M parameter model as the draft for a 70B parameter target. The draft model runs 100 times faster than the target, but it might only match the target's distribution 40% of the time. Most iterations produce just one or two tokens before rejection. The high $c$ cannot compensate for the poor $\alpha$.

• Medium draft model: Moderate speed and moderate acceptance rate. Often the sweet spot for practical applications. A 7B draft model for a 70B target might run 10 times faster while achieving 75% acceptance rates. With $k=5$ and $c=10$, this yields a speedup of roughly $(1-0.75^6)/(1-0.75) \div (1+5/10) \approx 2.7\times$. This combination often yields the highest speedups in practice precisely because neither dimension of the tradeoff dominates.

• Large draft model: Good distribution match ($\alpha$ close to 1) but slow ($c$ small). The overhead of running the draft model eats into speedup gains. A 30B draft for a 70B target might achieve 90% acceptance rates, but if it only runs 2-3 times faster than the target, the iteration cost is too high to achieve meaningful speedup. With $c=2.5$ and $k=5$, the speedup is $(1-0.9^6)/(1-0.9) \div (1+5/2.5) \approx 1.8\times$, lower than the medium model despite better acceptance rates.

The counterintuitive conclusion is that you do not always want the best possible draft model. A draft model that is somewhat worse but much faster can outperform a nearly perfect but slower draft model. The mathematics gives you the tools to find the optimal point for your specific hardware configuration.

In practice, acceptance rates typically range from 0.6 to 0.9 depending on several factors that interact in fine-grained ways. Using a smaller model from the same training family (for example, LLaMA-7B as a draft for LLaMA-70B) yields higher acceptance rates than using a distantly related model, because models trained on the same data with similar objectives develop similar probability distributions. Task difficulty matters too: code completion and factual text have more predictable continuations, leading to high acceptance rates, while creative writing or open-ended reasoning sees lower rates as both models explore different possibilities.

Temperature settings also have a significant effect on acceptance rates. At very low temperatures where sampling becomes nearly deterministic, both draft and target models tend to select the same highest-probability token, resulting in acceptance rates approaching 1. At higher temperatures where sampling explores more of the distribution, the models are more likely to disagree, reducing acceptance rates. This creates an interesting interaction: the temperatures that make output most diverse and creative are precisely the temperatures where speculative decoding helps least, while the temperatures that produce most predictable output are where speculative decoding provides maximum benefit.

These factors interact in interesting ways. For instance, using a very low temperature on a strongly matched draft-target pair can push acceptance rates above 0.95, making even aggressive draft lengths ($k=10$ or more) productive. Monitoring acceptance rates per task type and adjusting draft lengths accordingly can significantly improve practical throughput.