INT8 Quantization › INT8 Quantization

In the previous chapter on weight quantization basics, we explored why representing neural network parameters with fewer bits can sharply reduce memory requirements and accelerate inference. We saw how the core ideas of scale factors and zero-points let us map floating-point values onto discrete integer grids. We now turn to the most widely deployed quantization format in production systems: 8-bit integers (INT8). This precision point is a sweet spot that the machine learning community discovered through years of experimentation: memory savings are substantial (2x compression from FP16, 4x from FP32), specialized hardware acceleration is widely available, and accuracy degradation remains manageable for most applications. Understanding INT8 quantization in depth means understanding every major LLM inference library used today, from bitsandbytes to TensorRT-LLM.

INT8 quantization is about storing numbers more compactly and running them faster. Modern GPUs and specialized AI accelerators include dedicated INT8 matrix multiplication units that can reach 2-4x higher throughput than their floating-point counterparts. NVIDIA's Tensor Cores, for example, perform INT8 operations at twice the rate of FP16 operations. Quantizing both weights and activations to INT8 therefore reduces memory use and can accelerate computation. The improvement comes from two compounding effects: smaller data types require less memory bandwidth to load, and dedicated integer arithmetic units can perform more operations per clock cycle than their floating-point equivalents. On memory-bandwidth-limited tasks like large-batch LLM inference, these effects can substantially increase throughput.

The challenge is that this sounds easier than it is. Squeezing the continuous range of floating-point values into just 256 discrete integers inevitably introduces quantization error. Every value you store must round to one of those 256 representable points, and the difference between the original and rounded value accumulates across millions of operations. For small models and simple tasks, this noise is negligible. For large language models with billions of parameters and complex dependencies between layers, naive INT8 quantization can cause large accuracy degradation. The field had to develop principled techniques for minimizing this error before INT8 LLM inference became practical.

The journey to effective INT8 quantization for LLMs involved a surprising discovery: the problem was not the weights but the activations. Weights are static tensors we can analyze carefully offline, adjusting our quantization scheme to fit their distribution. Activations are dynamic, varying with every input, and in large language models they develop pathological outliers that resist standard quantization approaches. A small number of hidden dimensions produce values sharply larger than all other dimensions, which makes it impossible to choose a quantization scale that works well for both the outliers and the typical values. This observation, and the elegant mathematical fix for it, forms the intellectual core of this chapter.

We will explore the mathematics of range mapping, the practical absmax quantization scheme that forms the foundation, the granularity choices between per-tensor and per-channel quantization, the emergent outlier problem in large language models, and the smooth quantization technique that makes INT8 viable even for models with hundreds of billions of parameters. By the end you will understand how to apply these techniques, why each design choice was made, and what happens when things go wrong.