INT8 Quantization › Asymmetric Quantization

When tensor values are not centered around zero, symmetric quantization wastes representable range. ReLU activations, for example, are always non-negative. Mapping the range $[0, 3.5]$ symmetrically would waste half our integers representing negative values that never occur. This inefficiency becomes severe when the actual data distribution is heavily skewed or shifted away from zero. With 256 integers available and only half the range needed, symmetric quantization effectively reduces your precision from 8 bits to 7 bits for these tensors.

Asymmetric quantization solves this by introducing a zero-point offset $z$ that shifts the quantization grid to the data's range. Think of it as sliding our 256-slot number line left or right until it lines up with the data distribution, rather than always keeping it centered at zero. The grid spacing remains uniform, but we can position it anywhere. This flexibility comes at a cost: more complex arithmetic during inference, as we now have two parameters (scale and zero-point) to track rather than one.

The asymmetric quantization formulas show how to derive both parameters from the data range:

$$\begin{aligned} s &= \frac{\max(\mathbf{x}) - \min(\mathbf{x})}{255} \\ z &= -128 - \text{round}\left(\frac{\min(\mathbf{x})}{s}\right) \\ \mathbf{x}_q &= \text{round}\left(\frac{\mathbf{x}}{s}\right) + z \end{aligned}$$

where:

• $\mathbf{x}$: the input floating-point tensor
• $s$: the scale factor, computed from the full data range (minimum to maximum)
• $255$: the number of intervals in an 8-bit range ($2^8 - 1 = 255$ intervals for 256 values)
• $z$: the zero-point integer that shifts the grid to align with the data distribution
• $\mathbf{x}_q$: the resulting quantized tensor with values in $[-128, 127]$

The key difference from symmetric quantization lies in how we compute the scale and the introduction of the zero-point. Instead of basing the scale on the maximum absolute value, we use the full range from minimum to maximum. This keeps all 256 integers can potentially be used. The zero-point $z$ is an integer that shifts the quantization grid so that the minimum value maps to -128 and the maximum to 127. This shift allows us to "slide" our quantization grid along the number line to wherever the actual data lies.

Dequantization reverses this process by first undoing the zero-point shift and then applying the scale:

$$\hat{\mathbf{x}} = s \cdot (\mathbf{x}_q - z)$$

where:

• $\hat{\mathbf{x}}$: the reconstructed floating-point values
• $s$: the scale factor
• $\mathbf{x}_q$: the quantized integer values
• $z$: the zero-point offset, which is subtracted before scaling to recover the original range

While asymmetric quantization can stand for arbitrary ranges more efficiently, it complicates the mathematics during inference. Matrix multiplications now involve additional terms from the zero-point offsets. When we expand the matrix multiplication with asymmetric quantization, cross-terms involving the zero-points appear, requiring additional computation and memory access. For weights, which are typically centered around zero due to common initialization and regularization practices, symmetric quantization remains the standard choice. The computational overhead of asymmetric quantization is generally reserved for activations where the efficiency gains from better range utilization outweigh the additional complexity.