Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Speculative Decoding: Fast LLM Inference Without Quality Loss › Acceptance Rate

Michael BrenndoerferJanuary 16, 20267 min read

Language AI HandbookAll chapters

Part 42 of 67Inference Optimization

Chapter 323Speculative Decoding: Fast LLM Inference Without Quality Loss › Acceptance Rate

Reading section 8 of 14 · Acceptance Rate · 7 minSection 8 of 14 · 7 min

Section 8 of 14 · 7 min read

Speculative Decoding: Fast LLM Inference Without Quality Loss › Acceptance RateOne section · Sections in this chapter8 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The acceptance rate $\gamma$ is the probability that a given draft token is accepted. It is the most important single metric for predicting speculative decoding performance. This metric captures draft model quality and determines whether speculative decoding will provide meaningful speedups in your specific deployment.

Measuring Acceptance RateLink Copied

Empirical acceptance rate is measured by running speculative decoding on representative text and tracking the fraction of draft tokens that pass the acceptance test. The expected acceptance for each token is $\alpha(x) = \min(1, p(x)/q(x))$ , so the overall rate is the average of these values across all drafted tokens.

In[17]:

Code

def measure_acceptance_rate(draft_probs_list, target_probs_list):
    """
    Measure empirical acceptance rate from recorded probabilities.

    draft_probs_list: list of (token_id, draft_probability) for drafted tokens
    target_probs_list: list of target_probability for same tokens
    """
    total_draft = 0
    total_accepted = 0

    for (token_id, q_draft), p_target in zip(
        draft_probs_list, target_probs_list
    ):
        total_draft += 1

        # Acceptance probability
        accept_prob = min(1.0, p_target / q_draft)

        # Expected acceptance (for aggregate statistics)
        total_accepted += accept_prob

    return total_accepted / total_draft if total_draft > 0 else 0.0


# Simulate acceptance measurement
n_tokens = 1000

# Simulate well-aligned models (high correlation)
base_probs = np.random.exponential(0.1, n_tokens)
base_probs = base_probs / base_probs.sum() * n_tokens  # Scale

draft_probs = base_probs + np.random.normal(0, 0.02, n_tokens)
draft_probs = np.maximum(draft_probs, 0.001)  # Ensure positive

target_probs = base_probs + np.random.normal(0, 0.015, n_tokens)
target_probs = np.maximum(target_probs, 0.001)

# Create list format
measurements = list(
    zip([(i, d) for i, d in enumerate(draft_probs)], target_probs)
)

measured_rate = measure_acceptance_rate(
    [(i, d) for i, d in enumerate(draft_probs)], list(target_probs)
)

Out[18]:

Console

Simulated acceptance rate: 96.0%
Based on 1000 draft tokens

Out[19]:

Visualization

Scatter plot of target probability versus draft probability for 1000 tokens, colored by acceptance probability, with a diagonal reference line. — Scatter plot showing target versus draft probabilities for 1000 simulated tokens, colored by acceptance probability. Tokens above the diagonal (where target probability exceeds draft probability) are always accepted with certainty. Tokens below the diagonal are accepted probabilistically in proportion to the ratio p(x)/q(x), with lighter yellow-green points indicating lower acceptance probability.

This simulated rate of approximately 77% is typical for a well-aligned draft model, where the draft distribution closely tracks the target distribution. In practice, acceptance rates vary significantly based on the text being generated, the sampling temperature, and the specific draft-target model pairing.

Factors Affecting Acceptance RateLink Copied

Several factors influence acceptance rate in practice, and understanding these factors helps you optimize your speculative decoding deployments.

Model family similarity. Draft models from the same family as the target (for example, LLaMA-7B for LLaMA-70B) typically achieve 70-85% acceptance rates. Models from different families may drop to 40-60%. This improvement comes from shared training data, similar architectures, and aligned tokenization schemes that cause both models to "think" in similar ways about the same inputs.

Temperature. Higher sampling temperatures increase randomness, generally reducing acceptance rates. At temperature 0 (greedy decoding), acceptance is deterministic based on whether draft and target agree on the argmax token. As temperature increases, both models become less confident, their probability distributions flatten, and disagreements become more likely even when they agree on the most probable tokens.

Context type. Acceptance rates vary significantly by content. Factual text with predictable continuations (encyclopedic content, structured data, code with common patterns) achieves higher acceptance than creative writing or code with many valid alternatives. When there is a clear "right answer" that both models recognize, acceptance is high. When multiple reasonable continuations exist, the models may prefer different alternatives, lowering acceptance.

Sequence position. Early tokens in a response may have lower acceptance as the models "warm up" to the context. Later tokens often show higher acceptance once both models have established similar interpretations. This pattern suggests that the models' internal representations converge as they process more shared context together.

In[20]:

Code

def simulate_acceptance_by_temperature(temperatures, base_alignment=0.8):
    """
    Simulate how acceptance rate changes with temperature.
    Higher temperature = more randomness = harder to predict = lower acceptance
    """
    results = []
    for temp in temperatures:
        # Model alignment decreases with temperature (simplified model)
        effective_alignment = base_alignment * (
            1.0 / (1.0 + 0.5 * (temp - 1.0))
        )
        effective_alignment = max(0.3, min(0.95, effective_alignment))
        results.append(effective_alignment)
    return results


temperatures = [0.1, 0.5, 0.7, 1.0, 1.2, 1.5, 2.0]
acceptance_rates = simulate_acceptance_by_temperature(temperatures)

Out[21]:

Console

Acceptance rate vs temperature (simulated):

Temperature     Acceptance Rate
------------------------------
0.1             95.0%          
0.5             95.0%          
0.7             94.1%          
1.0             80.0%          
1.2             72.7%          
1.5             64.0%          
2.0             53.3%

As temperature increases, the distributions flatten and diverge, causing the acceptance rate to drop significantly. This highlights why speculative decoding is most effective for lower-temperature, more deterministic generation tasks. For applications like creative writing that benefit from higher temperatures, the speedup from speculative decoding will be more modest, though still worthwhile.

Acceptance Rate and Speedup RelationshipLink Copied

The relationship between acceptance rate and speedup is highly nonlinear. Small improvements in acceptance rate yield disproportionate speedup gains because more consecutive tokens pass verification. This nonlinearity arises from the geometric nature of sequential acceptance: improving acceptance from 70% to 80% does not merely improve each token's acceptance by 10%, it dramatically increases the probability of long acceptance runs.

Consider the probability of accepting 5 consecutive draft tokens. At $\gamma = 0.70$ , this is $0.70^5 = 0.168$ , meaning long runs occur 17% of the time. At $\gamma = 0.80$ , it is $0.80^5 = 0.328$ , nearly doubling. The entire tail of the distribution shifts significantly with what looks like a modest change in the per-token rate.

In[22]:

Code

import numpy as np


def theoretical_speedup(gamma, K, draft_ratio=0.1):
    """
    Calculate theoretical speedup for given acceptance rate.

    gamma: acceptance rate per token
    K: number of draft tokens
    draft_ratio: ratio of draft model time to target model time
    """
    # Expected accepted tokens
    expected_accepted = sum(gamma**i for i in range(1, K + 1)) + 1

    # Time ratio: (K * draft_time + target_time) / (expected * target_time)
    round_time_ratio = K * draft_ratio + 1
    standard_time_ratio = expected_accepted

    return standard_time_ratio / round_time_ratio


# Calculate speedups for range of acceptance rates
gammas = np.linspace(0.3, 0.95, 50)
K_values = [4, 6, 8]
draft_ratio = 0.15  # Draft model is 15% of target model time

Out[23]:

Visualization

Line plot of speedup factor versus acceptance rate for three values of K, showing steep increases at higher acceptance rates. — Theoretical speedup factor as a function of acceptance rate for K = 4, 6, and 8 draft tokens per round. The curves rise steeply at higher acceptance rates, showing that the relationship between draft model quality and inference speedup is strongly nonlinear. The shaded green region marks the typical acceptance rate range for well-aligned model families.

The plot demonstrates the importance of draft model alignment. Moving from 70% to 85% acceptance rate approximately doubles the speedup. This nonlinear relationship motivates significant investment in draft model quality. Even modest improvements in alignment translate to substantial real-world performance gains, making draft model optimization valuable for production deployments.