INT4 Quantization › INT4 Quantization

The GPU memory requirements for large language models create a basic access problem. A 70-billion-parameter model in standard 16-bit floating-point format consumes roughly 140 GB of GPU memory, an amount that requires several A100 or H100 GPUs costing tens of thousands of dollars to purchase. For most researchers and practitioners, that puts frontier-scale models out of reach entirely. Quantization is the most powerful tool we have for closing this gap, and INT4 quantization in particular has enabled a remarkable democratization of LLM inference: models that once required a data center now run on consumer GPUs that cost a few hundred dollars.

Reducing weights from 16 bits to 8 bits cuts memory in half while preserving most model quality. The natural question is: can we go further? INT4 quantization promises to halve memory again, fitting a 70B parameter model into roughly 35 GB instead of 140 GB. But moving from 8 bits to 4 bits is not just "more of the same." It introduces basic challenges that require entirely new techniques to overcome. The step from INT8 to INT4 is qualitatively different from the step from FP16 to INT8, because 4-bit precision is so coarse that naive approaches simply destroy model quality.

As we discussed in the previous chapter on INT8 quantization, the basic idea of quantization is mapping continuous floating-point values to a discrete set of integers. INT8 gives us 256 possible values. INT4 gives us only 16. This dramatic reduction in representational capacity means that naive 4-bit quantization produces models that perform worse than much smaller models in full precision. The techniques we explore in this chapter, particularly group-wise quantization and specialized 4-bit formats like NF4, are what make aggressive quantization practical rather than destructive.

Think of quantization as converting a high-resolution photograph to a low-resolution thumbnail. Going from 16-bit to 8-bit is like reducing a 4K image to 1080p: you lose some detail, but the image remains clear and recognizable. Going from 8-bit to 4-bit is like reducing that 1080p image to 64x64 pixels: without special techniques, faces become unrecognizable blobs. The techniques in this chapter are like smart compression algorithms that identify which pixels carry the most information and preserve them at higher fidelity, making even the 64x64 version surprisingly usable.

This chapter builds on the group quantization concepts from our INT8 discussion and introduces three key innovations that make 4-bit quantization viable: group-wise scaling that isolates the damage from outliers, the NF4 (Normal Float 4) format that places quantization levels optimally for Gaussian-distributed weights, and double quantization that reduces the overhead from storing scale factors. Together, these ideas underpin systems like QLoRA and bitsandbytes, which have brought 70B-scale inference to consumer hardware. We also walk through a complete numerical worked example so you can trace exactly how a weight survives the round-trip from FP16 to INT4 and back.