Speculative Decoding Math: Algorithms & Speedup Limits › Key Parameters

The key parameters governing speculative decoding performance are:

• k (Draft length): The number of tokens proposed by the draft model in each step. Longer drafts amortize the target model pass cost over more tokens but face diminishing returns as acceptance probability decays geometrically.
• c (Cost ratio): The ratio $T_{\text{target}} / T_{\text{draft}}$, representing how much faster the draft model is compared to the target model. Larger $c$ allows longer drafts and greater theoretical speedup.
• alpha (Acceptance probability): The probability $\alpha$ that a draft token is accepted by the target model. Equal to the probability mass overlap between draft and target distributions, and equal to $1 - TV(p, q)$.

Understanding how these three parameters interact is the key to successfully configuring and optimizing speculative decoding deployments. The speedup formula $S = [(1-\alpha^{k+1})/(1-\alpha)] / (1 + k/c)$ ties them together into a single performance prediction that you can optimize.