AWQ › Limitations

AWQ's activation-awareness, while powerful, rests on assumptions that do not always hold. Understanding these limitations helps you anticipate when AWQ might underperform and how to mitigate potential issues.

The most basic limitation is that AWQ's saliency metric (average activation magnitude across a calibration set) is a statistical proxy for weight importance, not a direct measure of it. A weight might be necessary to the model's performance but happen to see moderate activation magnitudes on the calibration set. This can occur when the model handles certain input patterns rarely but critically, and those patterns are not well-represented in the calibration data. For example, a model that is occasionally called upon to solve a specific type of math problem might have weights that activate strongly for that pattern but appear dormant in a general-purpose calibration corpus. AWQ would not protect those weights, and performance on math problems might degrade more than on general text.

This calibration-distribution mismatch can also arise from domain shift. If a model is quantized using English text but deployed on multilingual inputs, the activation patterns for non-English languages may identify different channels as salient than the English calibration data would suggest. Similarly, if a model is fine-tuned after quantization or evaluated on tasks that differ substantially from the calibration distribution, the per-channel scaling factors optimized for the calibration distribution may be suboptimal for the deployment setting.

AWQ's per-channel scaling formulation also captures only one dimension of weight importance: the column-wise activation magnitude. In reality, weight saliency can be more complex. GPTQ's Hessian-based approach captures second-order interactions between weights, identifying cases where quantizing one weight affects the optimal treatment of another. AWQ has no mechanism for detecting these cross-weight dependencies. In practice this limitation is minor for most models and tasks, because the dominant factor in weight importance is indeed the activation magnitude of the corresponding input channel. But for models with unusual weight structure or highly correlated weight patterns, GPTQ's more complete analysis can occasionally give better results.

Additionally, AWQ's scaling factors are computed once at quantization time and fixed thereafter. If the model's effective activation distribution shifts during inference (due to input distribution shift, prompt engineering, or special tokens), the pre-computed scaling factors may no longer be optimal. This is a limitation shared by all static post-training quantization methods, but it is worth keeping in mind for deployments where the input distribution is expected to be very different from anything in the calibration set.

Finally, AWQ gives per-channel scaling at the granularity of input channels to linear layers. It does not address quantization artifacts in activations themselves (activation quantization), in embedding tables, or in the logit computation. For deployments targeting extremely low precision (below 4 bits) or requiring very tight accuracy guarantees, additional techniques such as activation quantization or outlier suppression may be needed in combination with AWQ.

For most practical deployments, these limitations are minor. AWQ performs well across diverse models and tasks, and its speed and simplicity make it a practical default for INT4 weight quantization.