INT8 Quantization › Key Parameters
Michael BrenndoerferJanuary 11, 20261 min read
Language AI HandbookAll chapters Part 42 of 67Inference Optimization
INT8 Quantization › Key ParametersOne section · Sections in this chapter11 / 15 ↓
Sections in this chapter15
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for the INT8 quantization implementation are:
- alpha: The migration strength in smooth quantization, typically 0.5. Controls how much quantization difficulty is shifted from activations to weights. Higher values (closer to 1.0) shift more difficulty to the weights; lower values leave more with the activations.
- scale: The step size of the quantization grid. Determined by the maximum absolute value (absmax) in the tensor or channel. Smaller scales give finer precision but only stand for values in a smaller range.
- smooth_scales: The channel-wise scaling factors derived from calibration data to balance activation and weight magnitudes. These are computed once and stored as part of the model.
- Calibration dataset size: Typically 256 to 512 samples suffice for collecting reliable activation statistics. Too few samples may miss worst-case outlier channels.