INT8 Quantization › The Outlier Problem in Large Language Models

As language models scale to billions of parameters, a problematic pattern emerges: certain activation dimensions develop extreme outlier values. Research has shown that in models like OPT-175B and BLOOM-176B, a small number of hidden dimensions (sometimes called "large activations" or "emergent features") can have values 10-100x larger than typical activations. This phenomenon was not anticipated by early quantization research conducted on vision models and smaller transformers, and it poses a significant challenge to naive INT8 approaches.

The emergence of these outliers appears to be linked to how transformers process information at scale. Certain hidden dimensions seem to serve as "highways" for important information, developing consistently large activation magnitudes across many inputs. These are not random fluctuations but systematic features of the model's learned representations. The same channels show outlier behavior across different inputs and different layers, suggesting they encode something structurally important about how the model is information. While we are still working to fully understand why this happens, the practical implication is clear: any quantization strategy for large language models must account for these outliers.

The key insight is that outlier channels break the basic assumption behind per-tensor quantization. When we assume all channels have similar magnitude distributions and can share a single scale factor, we implicitly assume that no channel is sharply different from the others. The outlier phenomenon violates this assumption catastrophically. One channel's behavior forces a scale factor that is appropriate for that channel alone and wildly suboptimal for all others.

Consider a hidden state where 99.9% of values lie in $[-1, 1]$, but dimension 1847 consistently produces values around 50. Per-tensor absmax quantization would set $s = 50/127 \approx 0.39$, meaning values of magnitude 1 quantize to just 2-3 integers. The reconstruction error for these common values becomes unacceptable. A value of 0.5, which might be important for the model's computation, would quantize to either 1 or 2, introducing errors of up to 20% in its representation. This is not a minor precision loss that can be ignored. It can cause the model to make qualitatively wrong predictions.

The naive solutions all have significant drawbacks:

• Clipping outliers: Destroys important model information encoded in those dimensions. The outlier channels are apparently important, not artifacts.
• Per-channel activation quantization: Requires computing new scale factors for every token at runtime, adding latency and complexity.
• Mixed precision: Keeping outlier channels in FP16 complicates kernels and reduces throughput, partially defeating the purpose.

Each of these approaches involves either losing accuracy, losing speed, or adding significant implementation complexity. What was needed was a technique that could tame the outlier problem without sacrificing the efficiency benefits of uniform INT8 quantization. Smooth quantization gives exactly this.