INT8 Quantization › Summary

INT8 quantization gives a practical path to halving model memory and accelerating inference through hardware INT8 units. The full picture requires understanding several interconnected concepts, each building on the others:

• Absmax quantization maps floating-point values to INT8 using a single scale factor derived from the maximum absolute value, centering the quantization grid at zero to preserve exact zeros
• Asymmetric quantization introduces a zero-point offset to shift the grid for data distributions that are not centered at zero, improving efficiency for activations like ReLU outputs
• Per-channel quantization applies separate scales to each output channel, sharply reducing error when channels have different magnitude distributions while adding negligible storage overhead
• The outlier problem emerges in large language models where a small fraction of activation dimensions have values 10-100x larger than typical, making per-tensor quantization fail catastrophically for these models
• Smooth quantization solves the outlier problem by migrating quantization difficulty from activations to weights through a mathematically equivalent transformation, computable entirely offline and fusible into existing LayerNorm parameters at zero runtime cost
• The smoothing factor $s_j$ balances activation and weight ranges using a migration strength $\alpha$, with $\alpha = 0.5$ giving a balanced default that works well for most models

INT8 quantization with smooth quantization reaches near-FP16 accuracy on models up to hundreds of billions of parameters while giving 2x memory reduction and significant throughput improvements on compatible hardware. The techniques described here form the foundation for the inference infrastructure behind most large language model deployments today. The upcoming chapters on INT4 quantization, GPTQ, and AWQ explore techniques that push quantization to just 4 bits per weight, accepting somewhat more accuracy degradation in exchange for even greater memory efficiency.