AWQ › Summary

AWQ is a shift in quantization philosophy, from optimizing all weights equally to protecting the small subset that matters most. This chapter covered:

Salient weight identification. AWQ observes that weights multiplying large activations have disproportionate impact on model outputs. By analyzing activation magnitudes from a calibration dataset, it identifies which weights deserve protection from quantization error. The saliency metric is simple: average absolute activation magnitude across the calibration set.

Per-channel scaling. Rather than keeping salient weights at higher precision, AWQ uses per-channel scaling factors to effectively give them more resolution in the quantized representation. Weights are scaled up before quantization, which stretches them to occupy more of the INT4 grid. The inverse scaling is fused into preceding layers so inference runs at uniform INT4 precision with no extra operations.

The scaling formula. The scaling exponent $\alpha$ controls how aggressively AWQ differentiates between salient and non-salient channels. Small $\alpha$ values give gentle, broadly uniform scaling. Large $\alpha$ values give strong protection for the highest-activation channels. Per-layer grid search finds the optimal $\alpha$ efficiently.

Efficient algorithm. AWQ avoids GPTQ's expensive Hessian computations in favor of simple activation statistics and grid search over a handful of scaling hyperparameters per layer. This makes possible faster quantization of large models, often 2 to 4 times faster than GPTQ on comparable hardware.

Practical tradeoffs. AWQ and GPTQ reach similar accuracy on standard benchmarks, but AWQ often generalizes better to out-of-distribution inputs because its saliency estimates are based on reliable statistical measurements rather than sample-specific optimization. GPTQ has broader ecosystem support due to its earlier release, but AWQ is rapidly gaining adoption across major inference frameworks.

The next chapter explores the GGUF format, which gives a standardized way to store and distribute quantized models across different inference engines and hardware platforms.