GPTQ › Group Quantization

Standard per-tensor or per-channel quantization uses a single scale and zero point for many weights. For weights with non-uniform distributions within a channel, a single scale factor forces the quantization grid to accommodate the full dynamic range, wasting precision on weights that are clustered in a small portion of that range. GPTQ often employs group quantization to address this limitation.

In group quantization, weights are divided into contiguous groups (commonly 128 weights each) that each have their own scale and zero-point parameters. This is a middle ground between per-tensor quantization (cheapest, least accurate) and per-weight quantization (most expensive, most accurate). The overhead of storing group-wise quantization parameters is small relative to the accuracy gains.

Think of group quantization as using local rulers instead of a single global ruler to measure distances. If you have a collection of objects ranging from 1 mm to 1 m in size, a single ruler calibrated to handle the full range will be very imprecise for the small objects. Multiple rulers, each calibrated for a smaller range, will be more precise throughout. Group quantization applies this same principle to weight distributions.

Group quantization gives a middle ground between accuracy and overhead:

• Per-tensor quantization. One scale for all weights in the layer. Minimum storage overhead but potentially high quantization error if weight distributions vary across channels or weight groups.

• Per-group quantization. Separate scales for groups of 128 weights. The overhead is small: one FP16 scale per 128 INT4 weights adds 16 bits for every 4 $\times$ 128 = 512 bits of weight data, which is 0.125 bits per weight, increasing the effective bit-width from 4.0 to 4.125 bits. This tiny overhead often yields substantially better accuracy.

• Per-weight quantization. Individual scales for each weight. Maximum accuracy but impractical overhead: storing one FP16 scale per INT4 weight doubles the memory usage, eliminating the benefit of quantization entirely.

With group size 128 and 4-bit weights, the effective storage is approximately 4.125 bits per weight, a tiny overhead for significant accuracy gains. In practice, group sizes of 64 to 128 offer the best tradeoff, with group size 128 being the most common choice in deployed systems.