Speculative Decoding: Fast LLM Inference Without Quality Loss › The Speculative Decoding Paradigm
Language AI HandbookAll chapters Speculative Decoding: Fast LLM Inference Without Quality Loss › The Speculative Decoding ParadigmOne section · Sections in this chapter3 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Speculative decoding exploits the memory-bandwidth insight through a two-model architecture. A small, fast draft model generates multiple candidate tokens. The large target model then verifies all candidates in a single forward pass. If the draft model's predictions match what the target would have produced, we keep them. If they diverge, we reject the mismatched tokens and use the target model's correction. This approach transforms the problem from "how do we make the large model faster" to "how do we predict what the large model will say, so we can verify those predictions efficiently."
The efficiency argument rests on a simple asymmetry. Generating one draft token with a small model takes roughly 5% of the time that verifying one token with the large model would take. But verifying 5 draft tokens with the large model takes roughly the same time as verifying 1, since verification is a single forward pass through the large model that processes all positions in parallel. So if we generate 5 draft tokens cheaply and then verify them all at once, the combined cost is approximately 1.25x the cost of a single large-model token. If 4 of those 5 tokens are accepted, we generated 4 tokens for the cost of 1.25, which is a 3.2x speedup.
A smaller language model used to quickly generate candidate tokens for verification. The draft model should share the same vocabulary as the target model and ideally produce similar probability distributions. The quality of the draft model's alignment with the target determines how many tokens we accept per round, which directly controls the speedup factor.
The large language model whose output distribution we want to preserve exactly. The target model verifies draft tokens and provides corrections when drafts are rejected. Critically, the target model's output distribution is never approximated. Speculative decoding is a lossless acceleration technique that preserves the exact statistical behavior of the large model.
The process works in rounds. Each round proceeds through four phases:
- Draft phase: The draft model autoregressively generates candidate tokens (typically to ).
- Verify phase: The target model processes all candidates in one forward pass, computing the probability of each candidate given the preceding context.
- Accept/reject phase: Compare draft and target probabilities to decide which candidates to keep, using a mathematically principled criterion that preserves the target distribution.
- Correction phase: If a candidate is rejected, sample a correction token from an adjusted distribution that accounts for the tokens already accepted or rejected.
After each round, the process repeats from the new position in the sequence, incorporating any accepted tokens and the correction token into the running context. Generation continues until the desired length is reached or an end-of-sequence token appears.
The approach succeeds because good draft models predict the same tokens the target model would have chosen most of the time. Languages are highly predictable: when you read "The capital of France is", the next word is overwhelmingly "Paris". Both the small draft model and the large target model will assign high probability to "Paris" here, so speculative decoding accepts the draft token with near certainty. Predictable text produces high acceptance rates and large speedups. Ambiguous or creative text produces lower acceptance rates and more modest speedups, but the algorithm remains correct in all cases.
import random
def speculative_decoding_round(
draft_model, target_model, input_ids, num_draft_tokens=4
):
"""
One round of speculative decoding.
Returns accepted tokens and the number accepted.
"""
# Phase 1: Draft K candidate tokens
draft_tokens = []
draft_probs = []
current_ids = input_ids.copy()
for _ in range(num_draft_tokens):
# Get draft model's probability distribution
p_draft = draft_model.get_next_token_probs(current_ids)
# Sample from draft distribution
token = sample_from_distribution(p_draft)
draft_tokens.append(token)
draft_probs.append(p_draft[token])
current_ids = current_ids + [token]
# Phase 2: Verify all K tokens with target model in one pass
# Target model computes p(token_i | input + tokens_0..i-1) for all i
target_probs = target_model.get_all_next_token_probs(
input_ids, draft_tokens
)
# Phase 3 & 4: Accept/reject and correct
accepted_tokens = []
for i, token in enumerate(draft_tokens):
p_target = target_probs[i][token]
p_draft_val = draft_probs[i]
# Accept with probability min(1, p_target / p_draft)
if random.random() < min(1.0, p_target / p_draft_val):
accepted_tokens.append(token)
else:
# Reject: sample correction from adjusted distribution
correction = sample_adjusted_distribution(
target_probs[i],
draft_model.get_next_token_probs(input_ids + accepted_tokens),
)
accepted_tokens.append(correction)
break # Stop after first rejection
return accepted_tokensThe key efficiency gain comes from the verification phase. When we pass draft tokens to the target model, we use the parallelism of the transformer architecture. Computing attention over a sequence of additional tokens requires nearly the same memory bandwidth as computing attention for 1 token, since we load the model weights just once. The computational cost increases linearly with , but as we established, the computational cost is negligible compared to memory transfer. This means that verifying 5 draft tokens costs almost the same wall-clock time as verifying 1 token, creating the opportunity for substantial speedups whenever draft tokens are accepted.