Speculative Decoding: Fast LLM Inference Without Quality Loss › Key Parameters

Speculative decoding exposes several parameters that you need to tune for your deployment. Getting these right makes the difference between a large speedup and a marginal improvement.

• draft_model: The smaller, faster model used to generate candidate tokens. Ideally from the same model family as the target, with the same tokenizer, and roughly 7-10% the parameter count.
• target_model: The large model used to verify candidates and guarantee output distribution quality. Its output distribution is never modified.
• acceptance_rate ($\gamma$): The probability that a draft token matches the target model's preference. Measured empirically on representative inputs and determines the achievable speedup.
• num_draft_tokens ($K$): The number of candidate tokens generated per round. Typically 4-8. Larger $K$ yields higher expected tokens per round but risks more wasted computation when acceptance rates are low.
• temperature: Controls randomness in sampling. Higher temperatures generally reduce acceptance rates, as discussed above.

Choosing $K$ requires balancing the expected tokens per round against the overhead of drafting. For high acceptance rates (above 80%), $K = 6$ to $8$ is generally optimal. For moderate acceptance rates (60-75%), $K = 4$ to $5$ tends to be better because the probability of long acceptance runs is lower and the overhead of generating many draft tokens is not recouped.