Speculative Decoding: Fast LLM Inference Without Quality Loss › The Memory Bandwidth Bottleneck

To understand why speculative decoding works, we need to understand why autoregressive generation is slow in the first place. On the surface, it might seem that faster hardware is the solution: buy a better GPU, and the problem goes away. But the bottleneck is more fundamental than that. Modern GPUs have enormous computational throughput, yet that capacity sits idle during token generation because the work is not compute-limited. This distinction reveals that the solution to slow inference is not more computational power, but smarter use of the data movement that dominates generation time.

During inference, the GPU must load every model weight from memory for each forward pass. A 70B parameter model in 16-bit precision requires loading 140 GB of data per token generated. The NVIDIA A100's memory bandwidth of 2 TB/s means this takes roughly 70 milliseconds, during which the actual matrix multiplications complete almost instantly. The GPU computes at 312 trillion floating-point operations per second, yet token generation takes 70ms, not because the computations are demanding, but because the data transfer dominates. The GPU spends over 95% of its time waiting for data to arrive from memory, not computing.

This phenomenon is captured by the arithmetic intensity metric, which measures the ratio of floating-point operations to bytes transferred. Training has high arithmetic intensity because gradients and activations reuse the same weights many times across large batch sizes. A single training step processes thousands of examples simultaneously, amortizing the weight loading cost across all of them. Inference has low arithmetic intensity because each token requires loading all weights for minimal computation: one token, one pass, all weights. We say inference is memory-bound rather than compute-bound. This distinction is critical because memory-bound workloads cannot be sped up by adding more computational units. They can only be accelerated by reducing memory transfers or amortizing them across more useful work.

Think of the GPU's relationship to its memory like a factory floor that receives raw materials via a narrow conveyor belt. No matter how many workers you add to the factory floor, production is limited by how fast materials arrive on the belt. The arithmetic intensity measures how efficiently the workers use each load of materials. Training processes materials intensively and reuses them many times. Inference grabs one widget from the belt, performs a trivial operation, and needs the next load immediately.

The arithmetic intensity of 1 FLOP per byte is far below the GPU's operational intensity (ratio of compute to bandwidth), which exceeds 150 FLOPs per byte for the A100. This confirms that inference is deeply memory-bound. We load data constantly but barely compute anything with it. The GPU's thousands of CUDA cores sit idle most of the time, waiting for the next batch of weights to arrive from memory. No amount of additional compute cores changes this situation.

The key insight for speculative decoding emerges from this analysis: if we process multiple tokens simultaneously, we can amortize the memory loading cost. Loading the model weights once and processing 5 tokens takes nearly the same time as processing 1 token, but produces 5x the output. The challenge is that autoregressive generation seems inherently sequential: each token depends on all previous tokens, so we cannot generate token 4 without first knowing tokens 1, 2, and 3. Speculative decoding overcomes this apparent limitation through a clever draft-and-verify strategy that preserves sequential correctness while enabling parallel verification.