INT4 Quantization › Group-Wise Quantization

The solution to the outlier problem at 4-bit precision is group-wise quantization, also called block-wise quantization. This technique represents a major change in how we approach the quantization problem: instead of asking "what single scale factor best represents this entire weight matrix?" we ask "what scale factor best represents each small neighborhood of weights?" Instead of using a single scale factor for an entire tensor or even a channel, we divide weights into small groups and compute separate scale factors for each group. This localized approach isolates the impact of outliers, preventing them from corrupting the quantization of the entire weight matrix.

Think of group-wise quantization as applying a different "zoom level" to different regions of the weight matrix. The parts of the matrix that contain mostly small, closely clustered weights get a fine-grained zoom that can distinguish subtle differences. The parts that happen to contain outliers get a wider zoom to accommodate them, but only those parts are affected. The rest of the matrix continues to benefit from fine resolution, oblivious to the outliers elsewhere.

This approach works because of a key statistical fact about how outliers appear in trained neural networks. Outlier weights do not distribute themselves uniformly throughout a weight matrix. They appear sporadically, concentrated in particular rows, columns, or small regions. When they do appear, they are surrounded by many non-outlier weights that have similar small magnitudes. By working with groups of 32, 64, or 128 consecutive weights, we ensure that any given group likely contains either zero or a small number of outliers. The groups containing outliers sacrifice precision for range; the groups without outliers maintain fine-grained precision. This spatial separation of outlier influence is the core mechanism that makes group-wise quantization so effective.

The memory overhead of group-wise quantization is modest and well-controlled. Each group requires one additional scale factor, typically stored in FP16 (16 bits). For a group of 128 weights, this adds 16 bits amortized over 128 four-bit weights, which is $16/128 = 0.125$ extra bits per weight. The total effective bit rate becomes $4 + 0.125 = 4.125$ bits per weight, barely above the nominal INT4 rate. As we will see with double quantization later in this chapter, even this small overhead can be further reduced.

To understand why group-wise quantization works so effectively, consider the spatial distribution of outliers within a weight tensor. In trained neural networks, outliers typically do not occur uniformly throughout a tensor. Instead, they appear sporadically, concentrated in certain locations while leaving large regions of the tensor relatively outlier-free. By using small groups, an outlier affects only the quantization of its own group, not the entire layer. The remaining groups, free from outlier contamination, can use their 16 quantization levels efficiently to represent their local weight distribution with high fidelity.

For a weight tensor, group-wise quantization proceeds through the following steps:

1. Divide the weights into groups of size $g$, where common choices are 32, 64, or 128 elements per group
2. Compute a separate scale factor (and optionally zero-point) for each group based only on the weights within that group
3. Quantize each group independently using its own scale factor

The key insight is statistical: within any sufficiently small group of weights, the probability of encountering an extreme outlier is much lower than across the entire tensor. When an outlier does appear in a group, it affects only the $g$ weights in that group, leaving the thousands or millions of other weights in the tensor unaffected. Containing outlier damage makes 4-bit quantization practical.

The formal description of group-wise symmetric quantization is straightforward. For a group of weights $\mathbf{w}_g = [w_1, w_2, \ldots, w_g]$, we compute:

$$s_g = \frac{\max(|\mathbf{w}_g|)}{q_{\max}}$$ $$q_i = \text{clip}\!\left(\text{round}\!\left(\frac{w_i}{s_g}\right), -q_{\max}, q_{\max}\right)$$

where:

• $s_g$: the scale factor for group $g$, computed from the weights in that group only
• $\max(|\mathbf{w}_g|)$: the maximum absolute weight value within the group
• $q_{\max}$: the maximum representable integer (7 for signed INT4 with range -7 to 7)
• $q_i$: the quantized integer representation of weight $w_i$
• $\text{round}(\cdot)$: rounding to the nearest integer
• $\text{clip}(\cdot, -q_{\max}, q_{\max})$: clamping to the valid integer range

Reconstruction is simply the inverse operation: $\hat{w}_i = q_i \cdot s_g$. The stored representation is the collection of integers $\{q_i\}$ along with the per-group scale factors $\{s_g\}$.

Group-wise quantization substantially reduces quantization error because outliers no longer dominate the scale factor for the majority of weights. Each group operates with a scale factor tailored to its local distribution. This keeps the 16 available quantization levels are deployed where they provide the most benefit for that particular subset of weights. The scatter plot makes this concrete: with uniform quantization, many near-zero weights collapse to the same reconstructed value, appearing as horizontal bands on the plot. With group-wise quantization, the reconstruction tracks the original faithfully even for tiny weights.

The choice of group size balances accuracy against memory overhead, and understanding this trade-off is needed for making informed deployment decisions. Smaller groups provide finer-grained adaptation to local weight statistics, but each group requires its own scale factor. These scale factors, typically stored in FP16 format (16 bits each), add to the overall memory footprint. Let's examine this trade-off quantitatively: