INT4 Quantization › Worked Example: Tracing a Weight Through INT4 Quantization

To solidify your understanding of the full quantization pipeline, let's trace a single group of weights through the complete NF4 quantization and dequantization cycle. This numerical example shows exactly what happens at each step, making the abstractions concrete.

Suppose we have a small group of 8 weights (we use 8 for clarity; in practice groups are larger):

$$\mathbf{w} = [0.031, -0.054, 0.012, 0.087, -0.023, 0.065, -0.041, 0.019]$$

Step 1: Compute the group maximum absolute value.

The first step is to identify the scale of this group by finding the largest absolute weight value:

$$s_{\max} = \max(|\mathbf{w}|) = 0.087$$

Step 2: Normalize the group to the range expected by NF4.

NF4 is designed for data in the range approximately $[-1, 1]$. We normalize by dividing by $s_{\max}$:

$$\mathbf{w}_{\text{norm}} = \frac{\mathbf{w}}{s_{\max}} = [0.356, -0.621, 0.138, 1.000, -0.264, 0.747, -0.471, 0.218]$$

Note that the largest absolute value becomes exactly 1.0 after normalization, and all other values fall within $[-1, 1]$.

Step 3: Map each normalized weight to the nearest NF4 level.

We look up each normalized value in the NF4 table. Let's trace one value: the weight $-0.054$, which normalized to $-0.621$. Scanning the canonical NF4 codebook, the nearest level is approximately $-0.6962$, so this weight gets integer index 1.

After mapping all 8 weights:

Each NF4 index (0-15) can be stored in exactly 4 bits.

Step 4: Store the scale factor and quantized indices.

We store two things: the scale factor $s_{\max} = 0.087$ (in FP16) and the 8 quantized indices $[11, 1, 9, 15, 4, 14, 2, 10]$ (4 bits each). Total storage: 16 bits for the scale plus $8 \times 4 = 32$ bits for the indices, for 48 bits total. The original 8 FP16 weights used $8 \times 16 = 128$ bits. We achieved a 2.67x compression for this group.

Step 5: Dequantization.

When we need to use this weight during inference, we reconstruct by looking up each index in the NF4 table and multiplying by the scale:

$$\hat{w}_i = \text{NF4}[q_i] \times s_{\max}$$

For example, index 11 maps to NF4 level $\approx 0.3379$, giving reconstructed weight $0.3379 \times 0.087 \approx 0.0294$. The original was 0.031, so the absolute error is approximately 0.0016, well within the local quantization step size.

This worked example reveals several important properties of the NF4 + group-wise pipeline:

• The scale factor carries the magnitude information; the NF4 index carries the shape information
• Reconstruction error is bounded by the NF4 step size times the scale factor
• The largest weight in each group (here 0.087) always maps to the extreme NF4 level (index 15), guaranteeing zero error for the maximum
• Reconstruction accuracy is best for large weights and can be relatively coarser for small weights, but those small errors have less impact on the matrix product