Weight Quantization Basics › Summary

Quantization maps high-precision floating-point values to lower-precision integers, reducing memory footprint and letting faster computation for memory-bound inference workloads. The core insight is that LLM inference is primarily limited by memory bandwidth, not compute, and loading fewer bytes per parameter directly translates to faster token generation.

The basic operation involves two parameters: the scale $s$, which determines the step size between adjacent quantization levels, and the zero-point $z$, which shifts the integer range to align with the actual distribution of values. Together they define a linear mapping $q = \text{round}(x/s) + z$ from real values to integers, with inverse $\hat{x} = s(q-z)$ for dequantization.

Symmetric quantization forces $z=0$, simplifying computation but potentially wasting integer range when data is not zero-centered. Asymmetric quantization allows $z \neq 0$, using the full integer range for any distribution at the cost of additional arithmetic during dequantization. Weights are typically quantized symmetrically, while activations benefit from asymmetric schemes.

Quantization granularity ranges from per-tensor (one scale for all values) through per-channel (one scale per output dimension) to per-group (multiple scales within each channel). Finer granularity better handles distribution variation at the cost of additional scale storage, with per-channel adding negligible overhead while giving substantial accuracy improvements.

Calibration determines optimal quantization parameters by analyzing actual tensor values. Weight calibration is straightforward because weights are fixed. Activation calibration requires running on representative data and choosing aggregation strategies that balance outlier handling against precision for typical values.

The basic tradeoff in quantization is information loss versus compression. Fewer bits mean smaller models and faster inference but increased quantization error, with each additional bit reducing error by approximately 4x. Understanding this tradeoff, and the tools for managing it, prepares you for the specific quantization techniques in the chapters that follow.