INT8 Quantization › Quantized Matrix Multiplication

Understanding how quantized operations execute helps clarify why INT8 gives speedup. The key mechanism is that we perform the heavy matrix multiplication entirely in integer arithmetic, then apply scale factors to recover floating-point values for the final output. This two-stage approach exploits the fact that integer operations are cheaper than floating-point operations, especially for the large matrix multiplications that dominate transformer inference.

When both weights and activations are in INT8, the matrix multiplication proceeds as follows. We multiply the integer tensors together, accumulating into 32-bit integers to prevent overflow, and then convert and rescale the result:

$$\mathbf{Y}_{\text{int32}} = \mathbf{X}_{\text{int8}} \mathbf{W}_{\text{int8}}$$

where:

• $\mathbf{Y}_{\text{int32}}$: the intermediate result matrix, accumulated in 32-bit integers
• $\mathbf{X}_{\text{int8}}$: the quantized input activations in 8-bit integer format
• $\mathbf{W}_{\text{int8}}$: the quantized weights in 8-bit integer format

The integer multiplication produces INT32 results to avoid overflow from accumulating many INT8 products. Each INT8 times INT8 multiplication produces a result up to $127 \times 127 = 16129$, which fits in INT16. However, a matrix multiplication accumulates thousands or millions of such products, and these sums can easily exceed the INT16 range. INT32 gives enough headroom to accumulate the typical number of products found in neural network layers without overflow. The hardware handles this automatically: dedicated INT8 multiply-accumulate (MAC) units produce 32-bit partial sums that can be safely added together.

After the integer matrix multiplication, we convert back to floating-point and apply the scale factors to recover approximate floating-point outputs:

$$\mathbf{Y}_{\text{fp}} = s_X \cdot s_W \cdot \mathbf{Y}_{\text{int32}}$$

where:

• $\mathbf{Y}_{\text{fp}}$: the final floating-point output tensor
• $s_X$: the quantization scale factor for the activations (scalar for per-tensor, vector for per-channel)
• $s_W$: the quantization scale factor for the weights (scalar or per-channel vector)
• $\mathbf{Y}_{\text{int32}}$: the integer output from the matrix multiplication, converted to float before scaling

The scale factors $s_X$ and $s_W$ are multiplied together before being applied, since the quantization error from both sources combines multiplicatively. Since scale factors are scalar (or per-channel), this final rescaling is cheap compared to the matrix multiplication itself. The computational cost of the rescaling step is $O(n \cdot m)$ for an output matrix of size $n \times m$, while the matrix multiplication itself is $O(n \cdot m \cdot k)$. Since $k$ is typically large (thousands of features), the rescaling overhead is less than 0.1% of the total computation.

Hardware INT8 units execute this pattern efficiently because of several compounding advantages. INT8 multiply-accumulate operations use simpler circuits than floating-point, with no need to handle exponent alignment, mantissa normalization, or special cases like NaN and infinity. Smaller operands allow higher parallelism in the same silicon area: the same chip real estate that fits one FP32 multiplier can fit four INT8 multipliers. Memory bandwidth is halved compared to FP16, reducing the primary bottleneck for large models where the GPU must stream weights from memory for every token.