INT4 Quantization › Summary

INT4 quantization pushes the boundaries of model compression, reducing memory requirements to roughly 4.5 bits per weight (including scale factors). The techniques in this chapter form a coherent system: each one addresses a specific failure mode that would prevent naive 4-bit quantization from working. Together, they enable deployment of frontier-scale models on consumer hardware.

The key insights from this chapter are:

• 16 levels are not enough for naive uniform quantization. The limited representational capacity requires sophisticated techniques to maintain model quality. A single outlier weight can corrupt the entire quantization scale, making naive INT4 unusable in practice.

• Group-wise quantization is needed for INT4 success. By using separate scale factors for groups of 32-128 weights, outliers affect only their local group rather than the entire tensor. This spatial isolation of outlier damage is the foundational innovation that makes 4-bit quantization practical.

• NF4 format outperforms uniform INT4 for normally distributed weights by placing quantization levels to match the weight distribution. By so each of the 16 levels is equally likely to be used, NF4 maximizes the information extracted from the 4-bit budget. This format, introduced with QLoRA, has become the de facto standard for 4-bit inference.

• Double quantization reduces the memory overhead from scale factors by quantizing the scales themselves. The two-level hierarchy (INT4 weights, INT8 scales, FP32 meta-scales) approaches true 4 bits per weight while maintaining the accuracy benefits of group-wise quantization.

• Model size and task complexity determine quantization tolerance. Larger models (30B+) typically maintain quality at INT4, while smaller models may need INT8 or mixed precision strategies. Tasks requiring precise numerical reasoning are more sensitive to quantization errors than tasks involving pattern recognition.

The next chapter covers GPTQ, a calibration-based quantization method that uses second-order information to find optimal weight quantization, often achieving better accuracy than the simpler methods discussed here. If NF4 is like using a well-designed ruler, GPTQ is like custom-designing a ruler for each specific layer based on data about how that layer is used.