INT8 Quantization › Worked Example: Quantizing a Small Weight Tensor

To build concrete intuition, let's trace through a complete quantization example with specific numbers. This will make the abstractions tangible and reveal where errors accumulate.

Suppose we have a small weight tensor (one row of a larger matrix):

$$\mathbf{w} = [0.84, -0.23, 1.12, -0.67, 0.05, -1.05, 0.33, -0.91]$$

Step 1: Compute the absmax. We scan the entire vector and find:

$$\alpha = \max(|0.84|, |-0.23|, |1.12|, |-0.67|, |0.05|, |-1.05|, |0.33|, |-0.91|) = 1.12$$

Step 2: Compute the scale factor. Dividing by 127 to cover the full INT8 range:

$$s = \frac{1.12}{127} \approx 0.008819$$

Step 3: Quantize each element. Divide by $s$ and round to the nearest integer:

$$\begin{aligned} 0.84 / 0.008819 &\approx 95.25 \to 95 \\ -0.23 / 0.008819 &\approx -26.08 \to -26 \\ 1.12 / 0.008819 &\approx 127.00 \to 127 \\ -0.67 / 0.008819 &\approx -75.97 \to -76 \\ 0.05 / 0.008819 &\approx 5.67 \to 6 \\ -1.05 / 0.008819 &\approx -119.06 \to -119 \\ 0.33 / 0.008819 &\approx 37.42 \to 37 \\ -0.91 / 0.008819 &\approx -103.19 \to -103 \end{aligned}$$

The quantized tensor is: $\mathbf{w}_q = [95, -26, 127, -76, 6, -119, 37, -103]$.

Step 4: Reconstruct and compute error. Multiply each integer by $s$:

$$\begin{aligned} 95 \times 0.008819 &= 0.8378 \quad \text{(error: } 0.0022\text{)} \\ -26 \times 0.008819 &= -0.2293 \quad \text{(error: } 0.0007\text{)} \\ 127 \times 0.008819 &= 1.1200 \quad \text{(error: } 0.0000\text{)} \\ -76 \times 0.008819 &= -0.6702 \quad \text{(error: } 0.0002\text{)} \\ 6 \times 0.008819 &= 0.0529 \quad \text{(error: } 0.0029\text{)} \\ -119 \times 0.008819 &= -1.0495 \quad \text{(error: } 0.0005\text{)} \\ 37 \times 0.008819 &= 0.3263 \quad \text{(error: } 0.0037\text{)} \\ -103 \times 0.008819 &= -0.9084 \quad \text{(error: } 0.0016\text{)} \end{aligned}$$

The maximum error is 0.0037, and the mean absolute error is about 0.0015. Notice that the largest element (1.12) reconstructs perfectly since it was the absmax and maps exactly to 127. The element with value 0.05 (the smallest magnitude) experiences the largest relative error: the rounding from 5.67 to 6 introduces a 5.9% relative error. This pattern is typical: absmax quantization gives best relative precision to large-magnitude values and worst to small ones.

The maximum possible error for any element is $s/2 = 0.008819/2 = 0.00441$, which all our actual errors satisfy. Storing the entire vector requires just 8 bytes (8 integers) plus 4 bytes for the float32 scale factor, totaling 12 bytes instead of the original 32 bytes (8 float32 values). That is 2.67x compression with bounded error.