INT8 Quantization › INT8 Accuracy in Practice

The accuracy impact of INT8 quantization varies materially between model architectures and parameter counts. It also depends on the task. Understanding these patterns helps you set realistic expectations and choose the right approach for your deployment scenario.

Models under 1 billion parameters typically quantize to INT8 with minimal degradation, usually less than 1% accuracy loss on standard benchmarks. The weight distributions are well-behaved, and activation outliers are rare or mild. For these models, simple per-channel weight quantization with per-tensor activation quantization usually suffices without any smoothing.

Models from 1 billion to 10 billion parameters show increasing sensitivity. Without techniques like smooth quantization, accuracy can degrade 2-5% on certain tasks. Perplexity increases are noticeable but often acceptable for deployment. At this scale, smooth quantization or careful calibration-based clipping is typically necessary to maintain acceptable quality.

Models above 10 billion parameters exhibit the emergent outlier features described earlier, making naive INT8 quantization fail noticeably. Smooth quantization or similar techniques become needed at this scale, and even with best practices, a small accuracy gap compared to FP16 usually remains. The LLM.int8() technique, which keeps outlier channels in FP16 while quantizing the rest to INT8, was the original solution to this problem and remains useful when smooth quantization is insufficient.

Task sensitivity also matters. Simple classification tasks are reliable to quantization because the decision boundary does not require high numerical precision. Open-ended generation shows more degradation as errors compound across hundreds or thousands of autoregressive steps. Mathematical reasoning and code generation are particularly sensitive because small numerical errors can cascade into completely wrong answers.

The calibration dataset is an often-overlooked factor in INT8 accuracy. The statistics used to set scale factors and smooth quantization parameters must come from data representative of the deployment distribution. A model calibrated on English text may produce suboptimal quantization parameters for code completion or multilingual inputs. Choosing calibration data carefully, and potentially using task-specific calibration, can materially improve quantized model quality.