INT8 Quantization › Symmetric Quantization with Absmax

The simplest and most common approach to INT8 quantization is symmetric quantization, where we center the quantization grid at zero. The "absmax" method finds the maximum absolute value in the tensor and maps it to the largest representable integer. This approach derives its name from the key statistic it computes: the absolute maximum, which determines the extent of our quantization range in both positive and negative directions simultaneously.

Think of absmax quantization as stretching or compressing a rubber band. Your floating-point values are marked on the band, and you resize it until the most extreme mark lines up with the INT8 boundary at -128 or 127. The scale factor tells you how much you stretched or compressed. Every other value gets rescaled by the same factor, preserving relative distances. The reconstruction step is just reversing this scaling operation. The elegance of this approach is that a single number, the scale factor, captures everything you need to know about the quantization.

The intuition behind absmax quantization is straightforward. We want to ensure that no value in our tensor falls outside the representable range, which would force us to clip it and introduce potentially large errors. By finding the largest magnitude value in the tensor and designing our grid to just barely accommodate it, we guarantee that every original value can be represented without clipping. At the same time, we make the grid as fine as possible given this constraint, since using a larger range than necessary would waste precision on integers that never get used.

Given a tensor $\mathbf{x}$ with floating-point values, absmax quantization works as follows. First we compute the scale factor from the tensor's extreme value, then we apply rounding to obtain integer codes:

$$\begin{aligned} \alpha &= \max(|\mathbf{x}|) \\ s &= \frac{\alpha}{127} \\ \mathbf{x}_q &= \text{round}\left(\frac{\mathbf{x}}{s}\right) \end{aligned}$$

where:

• $\mathbf{x}$: the input floating-point tensor we want to quantize
• $\alpha$: the maximum absolute value in the input tensor, called the "absmax"
• $s$: the scale factor that determines how much floating-point distance each integer step is
• $127$: the maximum representable value in a signed 8-bit integer (we reserve -128 as an "overflow" value and map the positive side to $[0, 127]$)
• $\mathbf{x}_q$: the resulting tensor of quantized 8-bit integers, each in $[-128, 127]$

Let's trace through this process step by step. First, we scan the entire tensor to find $\alpha$, the largest magnitude value regardless of sign. This becomes our reference point for setting the scale. Next, we compute the scale factor $s$ by dividing $\alpha$ by 127, which tells us how much "floating-point distance" each integer step is. If $\alpha = 0.635$, then $s \approx 0.005$, meaning each integer step covers about 0.005 in floating-point space. Finally, we quantize each value by dividing it by the scale factor and rounding to the nearest integer. This rounding step is where the actual information loss occurs, as we snap each continuous value to its nearest discrete representation.

To recover an approximation of the original values, we simply multiply by the scale:

$$\hat{\mathbf{x}} = s \cdot \mathbf{x}_q$$

where:

• $\hat{\mathbf{x}}$: the reconstructed floating-point tensor, an approximation of the original $\mathbf{x}$
• $s$: the scale factor used during quantization (must be stored alongside the quantized tensor)
• $\mathbf{x}_q$: the quantized integer tensor we stored

The hat notation indicates this is a reconstruction, not the exact original values. The difference $\mathbf{x} - \hat{\mathbf{x}}$ is the quantization error. This error arises because the rounding operation discards fractional information. When we divided by $s$ and rounded, we lost the remainder, and multiplying the rounded result back by $s$ cannot restore what was lost. The beauty of this scheme is that the error is bounded: no reconstructed value can differ from its original by more than half a scale step, or $s/2$. For a tensor with absmax of 1.0, the scale is $1.0/127 \approx 0.0079$, so the maximum error is about 0.004. This is roughly 0.4% of the full range, which is small enough to be acceptable for most neural network weights.

Centering the grid at zero has a important computational advantage: the integer zero maps exactly to floating-point zero. This property might seem like a minor mathematical nicety, but it has significant practical implications for neural network inference.

This matters because many neural network activations are zero (from ReLU, dropout, and padding), zero-valued weights do not contribute to computations, and preserving exact zeros avoids accumulating unnecessary rounding errors. Consider what happens during a matrix multiplication when many elements are zero. If zero maps exactly to zero, these elements contribute nothing to the output, exactly as they should. But if zero were represented by some non-zero integer due to an offset, every "zero" element would contribute a small spurious value to the computation. Over millions of operations, these spurious contributions could accumulate into real errors.

Symmetric quantization also simplifies the mathematics during matrix multiplication. When computing $\mathbf{Y} = \mathbf{X}\mathbf{W}$ with quantized values, we only need to track scale factors, not additional offset terms. This simplification translates directly into faster inference, since the dequantization step requires only a single multiplication rather than a multiplication followed by an addition. When both scales are scalar constants (as they are for per-tensor quantization), this final rescaling is in effect free compared to the matrix multiplication itself.

Absmax quantization clips all values to the range $[-\alpha, \alpha]$. If $\alpha$ is dominated by a few outlier values, the quantization grid becomes coarse for the majority of values. This creates a basic tension in quantization design: we want to accommodate outliers to avoid clipping errors, but we also want a fine grid to minimize rounding errors for typical values.

Consider a tensor where 99% of values lie between -0.1 and 0.1, but one outlier is 5.0. Using absmax:

$$s = \frac{5.0}{127} \approx 0.0394$$

This scale means values around 0.1 get quantized to just $\text{round}(0.1/0.0394) = 3$. The reconstruction is $3 \times 0.0394 = 0.118$, introducing significant relative error for these common values. To understand the magnitude of this problem, note that a typical value of 0.1 experiences an 18% relative error, while the outlier value of 5.0 would be reconstructed nearly perfectly. We have sacrificed precision where it matters most, on the bulk of our values, to perfectly preserve a single outlier.

One solution is to clip outliers: choose $\alpha$ smaller than the true maximum, accepting that extreme values will be clipped. This trades outlier accuracy for better precision on the bulk of values. For example, if we set $\alpha = 0.5$ instead of 5.0, our scale becomes $s = 0.5/127 \approx 0.00394$, and a value of 0.1 now quantizes to integer 25 with reconstruction $25 \times 0.00394 = 0.0985$, a much smaller relative error. The outlier would be clipped to 127, reconstructing to 0.5 instead of 5.0, but this might be an acceptable trade-off depending on the model's sensitivity to those extreme values.

Finding the optimal clipping threshold is the basis for calibration-based quantization methods. These methods analyze the value distribution across representative data to find the clipping point that minimizes total reconstruction error, balancing the clipping errors from outliers against the rounding errors for typical values. Calibration is often as important as the quantization scheme itself, and models that fail to quantize well with default settings often succeed after proper calibration.