GPTQ › Summary

GPTQ turns weight quantization from a simple rounding problem into a principled optimization problem grounded in second-order information theory. By compensating for each weight's quantization error through Hessian-guided updates to remaining weights, GPTQ reaches reconstruction error far below what naive quantization can manage.

The key elements that make GPTQ work are:

• The layer-wise reconstruction objective. Optimizing the difference between original and quantized layer outputs (rather than weights directly) minimizes the metric relevant to downstream model performance.
• The Hessian matrix. Computed cheaply from a single forward pass on calibration data, the Hessian captures which weights are important by measuring the curvature of the loss with respect to each weight. Weights connected to frequently active input features have high curvature and must be quantized precisely.
• The optimal compensation formula. When a weight is quantized, a closed-form formula distributes the rounding error across all remaining weights proportionally to their correlations with the quantized weight via the inverse Hessian. This redistribution recovers much of the accuracy lost to rounding.
• Cholesky-based efficiency. Precomputing the Cholesky factorization of the inverse Hessian reduces the per-column update cost from $O(d^2)$ to $O(d)$, making the entire quantization procedure tractable at scale with $O(d^3)$ total complexity per layer.
• Group quantization. Using separate scale parameters for groups of 128 weights captures local weight distribution variations, adding only 0.125 bits per weight in overhead while substantially improving accuracy.
• ActOrder. Processing high-sensitivity columns first, when more remaining weights are available for compensation, improves perplexity by 0.1 to 0.5 points at 4-bit precision.

For you, GPTQ means that 4-bit quantization is practical for production LLM deployment. Models lose minimal accuracy while fitting in a fraction of the original memory. A 65B parameter model that required a multi-GPU server in FP16 now runs on a single consumer GPU. The calibration data requirements and quantization time are modest one-time costs for large and permanent efficiency gains.

Understanding GPTQ also gives insight into the broader space of quantization methods. The principle of error compensation, rather than pure rounding, appears in various forms across modern quantization techniques. The Hessian as a measure of weight importance recurs in AWQ, QuIP, and AQLM. The layer-wise decomposition appears in nearly every practical post-training quantization method. Whether you are deploying a quantized model or developing new efficiency methods, the foundations established by GPTQ remain needed knowledge for anyone working in this space.