AWQ › AWQ: Activation-Aware Weight Quantization

In the previous chapter, we explored GPTQ's approach to quantization, which uses second-order Hessian information to minimize quantization error layer by layer. While GPTQ reaches impressive results, its reliance on expensive matrix inversions and sequential weight updates creates computational overhead that scales with model size. Activation-Aware Weight Quantization (AWQ) takes a fundamentally different approach: instead of trying to optimally quantize all weights, it focuses on identifying and protecting the small subset of weights that matter most for output quality.

The motivating question behind AWQ sounds simple: are all weights equally important? If you have a 7-billion parameter model and you need to quantize it to 4 bits, does every one of those seven billion numbers deserve the same level of careful treatment? Intuition says no. Some weights sit at important junctions in the network where small errors compound through many subsequent computations. Others contribute so little that even large errors in their values barely register in the final output. AWQ turns this intuition into a practical algorithm by identifying important weights and giving them preferential treatment under quantization.

The mechanism AWQ uses to rank weights is elegant in its simplicity. Rather than computing expensive curvature information or running lengthy optimization procedures, AWQ looks at the activations that each weight multiplies. Think of it this way: a weight's importance depends on its own magnitude and on the magnitude of the signal it amplifies. A small weight that consistently multiplies large activations can have enormous influence on the output. A large weight that typically multiplies near-zero activations contributes almost nothing. By collecting activation statistics from a small calibration dataset, AWQ builds a picture of which input channels carry the most signal and therefore which weights deserve protection from quantization error.

AWQ protects the identified salient weights without changing their bit width. Naive approaches would keep important weights in higher precision, say 8 bits or 16 bits, while quantizing everything else to 4 bits. But this mixed-precision approach creates hardware headaches: kernels optimized for uniform 4-bit computation cannot efficiently process a mixture of precisions, and the irregular memory access patterns that result from mixed-precision weights degrade throughput. Instead, AWQ applies per-channel scaling factors that redistribute quantization precision. Important weights get scaled up before quantization, which effectively gives them finer resolution on the integer grid, and the inverse scaling is absorbed into the preceding layer. The final model is pure INT4, but the precision is concentrated where it matters.

This chapter builds on the quantization fundamentals from earlier sections and the specific techniques introduced in the GPTQ chapter. We will examine the theory of activation-aware quantization in depth, derive the scaling factor formula, walk through a concrete numerical example, and compare AWQ against GPTQ across multiple dimensions. By the end, you will have a thorough understanding of why AWQ works, when to prefer it over alternatives, and how to apply it in practice.