GPTQ › Limitations and Practical Tradeoffs

GPTQ is a major advance over naive quantization, but understanding its limitations is needed for deploying it effectively. Each limitation also suggests where further research has gone, and where you might need to supplement GPTQ with additional techniques.

Calibration sensitivity. Quantization quality depends on how well the calibration data matches the model's deployment inputs. If you calibrate a code-generation model on web text, the Hessian will reflect the statistics of web text activations, not code activations. Code has more structured patterns, longer token dependencies, and different vocabulary distributions. A model quantized with mismatched calibration data will precisely quantize weights that matter for web text while representing weights important for code less accurately. For general-purpose models used on general text, the standard C4 or WikiText calibration works well. For specialized deployments, invest in domain-specific calibration data.

Computational cost. GPTQ quantization is materially slower than naive round-to-nearest. Quantizing a 7B model takes around 10 to 30 minutes on a modern GPU, and the time scales roughly as $O(d^3)$ per layer with the Hessian computation and Cholesky factorization. Quantizing a 65B model can take several hours. This is acceptable for one-time quantization, but makes GPTQ unsuitable for anything requiring on-the-fly quantization during inference. You quantize once and deploy; you do not requantize per request. The computational cost is also why GPTQ is usually run on a GPU: CPU-based Cholesky factorization for large hidden dimensions would be impractically slow.

Layer-wise approximation. By optimizing each layer independently, GPTQ ignores how quantization errors compound across layers. In a deep transformer, a small error introduced at layer 1 might be amplified by subsequent attention and FFN computations, reaching layer 20 in a magnified form. GPTQ's per-layer optimization minimizes each layer's individual error without accounting for this cross-layer amplification. This approximation is empirically reasonable (the results in the perplexity table above show this), but it is not exact. Methods that incorporate cross-layer information, while more expensive, can in principle do better.

Activation quantization not included. GPTQ only quantizes weights, not activations. For inference on specialized hardware accelerators that require both weight and activation quantization (such as certain edge devices or custom chips designed for INT4 compute), GPTQ alone is insufficient. Additional quantization-aware training or separate activation quantization techniques are needed to reach full INT4 inference on such hardware.

Outlier sensitivity. The Hessian estimation can be skewed by outlier activations. As discussed in the chapter on INT8 quantization, transformer models sometimes produce extreme activation values in certain dimensions that can be orders of magnitude larger than typical values. These outliers disproportionately influence the Hessian: a few very large activations in one dimension will make the Hessian diagonal for that dimension enormous, leading GPTQ to treat it as critically sensitive and focus excessive precision there, potentially neglecting other dimensions. The dampening factor helps by reducing the relative influence of outliers, but does not eliminate it. Techniques like SmoothQuant, which pre-scales activations to reduce outlier magnitude, can be combined with GPTQ to address this issue.

Despite these limitations, GPTQ remains one of the most effective post-training quantization methods for LLMs. Its combination of theoretical foundation (optimal error compensation from second-order information) and practical efficiency (Cholesky-based linear updates) set the standard for weight quantization. The success of GPTQ inspired subsequent work like AWQ (Activation-aware Weight Quantization), which we will explore in the next chapter. AWQ takes a complementary approach: instead of compensating for quantization errors, it identifies and protects the most important weights from quantization entirely, often achieving even better results on models with strong activation outliers.