Speculative Decoding: Fast LLM Inference Without Quality Loss › Worked Example: Step-by-Step Round

Let us trace through a complete speculative decoding round numerically to make the abstract procedure concrete. We will use a vocabulary of 5 tokens for simplicity and demonstrate both the acceptance and rejection cases.

Setup. Our vocabulary has tokens $\{A, B, C, D, E\}$ with IDs $\{0, 1, 2, 3, 4\}$. We generate $K = 3$ draft tokens. The target model's probability distribution over the next token given the current context is:

$$p = [0.40,\ 0.25,\ 0.20,\ 0.10,\ 0.05]$$

The draft model's distribution is:

$$q = [0.30,\ 0.30,\ 0.15,\ 0.15,\ 0.10]$$

Step 1: Draft generation. The draft model samples from $q$ and produces the sequence $[A, B, C]$ with the probabilities it assigned at each step: $q(A) = 0.30$, $q(B) = 0.30$ (given A was sampled), $q(C) = 0.15$ (given A, B were sampled). For simplicity we treat each step as independent draws.

Step 2: Parallel verification. The target model processes $[\text{context}, A, B, C]$ in one forward pass. At each position it computes:

• Position of A: $p(A) = 0.40$
• Position of B: $p(B) = 0.25$
• Position of C: $p(C) = 0.20$

Step 3: Accept/reject for token A. We compute $\alpha(A) = \min(1, p(A) / q(A)) = \min(1, 0.40 / 0.30) = \min(1, 1.33) = 1.0$. We accept token A with certainty because the target model assigned it higher probability than the draft model. The key insight is that when the target agrees or prefers the draft's choice, acceptance is guaranteed.

Step 4: Accept/reject for token B. We compute $\alpha(B) = \min(1, p(B) / q(B)) = \min(1, 0.25 / 0.30) = \min(1, 0.833) = 0.833$. We draw a uniform random number; suppose it comes up 0.72, which is less than 0.833, so we accept token B. Both A and B are now accepted.

Step 5: Accept/reject for token C. We compute $\alpha(C) = \min(1, p(C) / q(C)) = \min(1, 0.20 / 0.15) = \min(1, 1.33) = 1.0$. We accept token C with certainty.

Step 6: Bonus token. All three drafts were accepted. The target model also produced a logit at position after C during the forward pass, so we sample a bonus token directly from that distribution, obtaining token D (say, $p(D) = 0.10$). This token costs nothing extra.

Result of the round. We produced 4 tokens $[A, B, C, D]$ from a single target model forward pass (plus the cheap draft passes). Standard generation would have required 4 target model forward passes.

Rejection scenario. Now suppose the draft produced token B at position 1 with $q(B) = 0.30$, but the target's probability at that position is $p(B) = 0.10$. Then $\alpha(B) = \min(1, 0.10/0.30) = 0.333$. If our random draw is 0.65 (above 0.333), we reject B. We compute the correction distribution:

$$\text{residual}(x) = \max(0, p(x) - q(x)) = [0.10, 0, 0.05, 0, 0]$$

normalized to give $p_{\text{correction}} = [0.67, 0, 0.33, 0, 0]$. We sample token A with probability 0.67 or token C with probability 0.33. Say we draw A. The round ends with only the accepted tokens from before position 1 plus the correction A. Future drafts will be generated from the corrected context.

This worked example shows both the happy path (high acceptance, multiple tokens per round) and the rejection path (context correction, minimal waste). Even in the rejection case, we still generate one correct token from the target distribution, just as standard generation would have.