Weight Quantization Basics › Why Memory Bandwidth Limits Inference

Understanding why quantization matters requires a brief look at the physics of modern GPU architecture. We often think of GPUs as compute devices, but for LLM inference, they behave primarily as memory devices. The compute units sit idle while waiting for data, and quantization is the most direct tool for addressing this basic bottleneck.

Modern GPUs have enormous compute throughput that vastly outpaces their memory systems. An NVIDIA A100 delivers 312 teraflops of FP16 compute but only 2 TB/s of memory bandwidth. These numbers sound abstract, so think of it this way: for every byte loaded from memory, the GPU can perform roughly 156 floating-point operations. If an operation requires loading 1 byte and doing 1 computation, the GPU sits idle for the other 155 operation slots. The compute is effectively free compared to the cost of moving data.

During autoregressive generation, each new token requires exactly one forward pass through the entire model, loading essentially all of the model's weight matrices once. A single matrix multiplication of shape $(1, d_\text{model}) \times (d_\text{model}, d_\text{ffn})$, where $d_\text{model}$ is the model dimension and $d_\text{ffn}$ is the feed-forward dimension, loads $d_\text{model} \times d_\text{ffn}$ parameters but performs only $2 \times d_\text{model} \times d_\text{ffn}$ floating-point operations. This gives an arithmetic intensity of just 2 FLOPs per loaded element. When the hardware can sustain 156 FLOPs per byte, an arithmetic intensity of 2 means you are using roughly 1.3% of the available compute. The rest waits.

This situation, where memory bandwidth limits throughput rather than compute, is called memory-bound operation. It is the dominant regime for LLM inference at small batch sizes, which includes most real-world serving scenarios. Quantization directly attacks this bottleneck. Loading 4-bit weights instead of 16-bit weights moves four times less data across the memory bus. Even if dequantization adds some computational overhead, the memory savings dominate for memory-bound workloads. This explains why efficient LLM deployment now depends heavily on quantization.

The relationship between bit width and bandwidth is exactly linear: cutting bits in half cuts bandwidth in half. This linearity is what makes quantization so attractive as an optimization strategy. There is no engineering trick here, just arithmetic. Fewer bits per number means fewer bytes transferred, and fewer bytes transferred means faster inference.