GPTQ: Optimizing 4-Bit Weight Quantization for LLMs

In the previous chapters on INT8 and INT4 quantization, we explored straightforward approaches to weight compression: round each weight to the nearest representable value in the target format. While these methods work reasonably well, they treat each weight independently, ignoring the complex interactions between weights in a neural network. A small rounding error in one weight might be catastrophic for model quality, while a larger error in another weight might barely matter. The round-to-nearest strategy has no mechanism to distinguish between these cases, and so it ends up treating a harmless rounding as equally important as a damaging one.

GPTQ (GPT Quantization) takes a fundamentally different approach. Rather than treating quantization as a simple rounding problem, GPTQ frames it as an optimization problem: given a layer's weights, find the quantized values that minimize the layer's output error. The key insight is that when you quantize one weight, you can partially compensate for the resulting error by adjusting the remaining weights before they too are quantized. This sequential compensation mechanism propagates corrections through the weight matrix, sharply reducing the accumulated error that plagues naive methods.

Think of GPTQ as a team of engineers managing a large construction project where every measurement must be rounded to the nearest centimeter. A naive approach would round each measurement independently and accept whatever cumulative error results. A smarter approach, the one GPTQ takes, observes each rounding error as it happens and adjusts subsequent measurements to keep the overall structure as close as possible to the original blueprint. Each small correction nudges the final result back toward accuracy, so that by the time you've finished rounding all measurements, the cumulative deviation is far smaller than if you had rounded everything in isolation.

This compensation mechanism, combined with algorithmic optimizations, allows GPTQ to reach quantization error far below what naive methods can manage. Models quantized with GPTQ to 4 bits often perform nearly as well as their full-precision counterparts, letting LLMs with tens of billions of parameters to run on consumer GPUs. GPTQ was one of the first methods to make models like LLaMA-65B practically usable on hardware that would otherwise be unable to load them, compressing 130 GB of FP16 weights into under 35 GB of INT4 data.

The mathematical foundation of GPTQ draws from classical work in neural network compression, particularly the Optimal Brain Surgeon framework developed in the early 1990s for pruning. GPTQ's authors recognized that the same machinery developed to remove weights from a network could be repurposed to quantize weights with minimal damage. This cross-pollination of ideas from pruning into quantization is a recurring theme in modern efficiency research: the core question in both settings is how to perturb a network's weights while preserving its behavior.

Understanding GPTQ deeply will serve you well beyond knowing how to call the library. The concepts here, particularly the Hessian matrix as a measure of weight importance and the closed-form compensation formula, appear throughout modern quantization research. AWQ, QuIP, and AQLM all build on ideas that GPTQ establishes. By the end of this chapter, you will have the theoretical grounding to understand GPTQ and the entire family of second-order quantization methods that have followed it.

To understand how GPTQ approaches quantization, we must first establish what it means to quantize well. The basic question is: what objective should we optimize? One natural answer might be to minimize the difference between original and quantized weights directly. However, this ignores a important insight: not all weight errors are equally harmful. What truly matters is how quantization affects the layer's output, because the output is what downstream computations depend upon. A weight that multiplies a small input contributes little to the output regardless of its quantization error; a weight that multiplies a large, frequently-occurring input is necessary and must be handled with care.

GPTQ operates on one layer at a time, treating each layer's quantization as an independent optimization problem. This layer-wise decomposition is both a practical necessity and a reasonable approximation. It is practical because computing gradients through the entire model for every quantization decision would require a full backward pass per weight, which is completely infeasible for models with billions of parameters. It is a reasonable approximation because the output of each layer is the input to the next, and minimizing the error introduced at each layer independently tends to minimize the total error propagated through the network.

Think of it as optimizing each department of a large company independently. You cannot simultaneously optimize every decision across every team, but if each department minimizes its own errors, the overall company performance tends to be much better than if no one cared about local quality. The layer-wise approach gives GPTQ a tractable decomposition of an otherwise intractable global problem.

Consider a linear layer with weights $\mathbf{W} \in \mathbb{R}^{d_{out} \times d_{in}}$ and inputs $\mathbf{X} \in \mathbb{R}^{d_{in} \times n}$, where $n$ is the number of calibration tokens we use to estimate the layer's behavior. The goal is to find quantized weights $\hat{\mathbf{W}}$ that minimize the reconstruction error:

where:

• $\mathcal{L}(\hat{\mathbf{W}})$: the reconstruction error (loss) for the quantized weights
• $\mathbf{W}$: the original high-precision weight matrix
• $\hat{\mathbf{W}}$: the quantized weight matrix we are searching for
• $\mathbf{X}$: the input matrix containing calibration data (each column is one token's activations)
• $||\cdot||_F$: the Frobenius norm, equal to the square root of the sum of all squared elements

This loss function measures how much the layer's output changes due to quantization. We are comparing outputs, not weights directly. This distinction is needed: a weight that interacts with large input values will have a greater impact on the output than a weight that sees only small inputs. Two weights might differ by the same absolute amount from their quantized versions, but if one multiplies inputs that are ten times larger, it creates ten times more output distortion. The Frobenius norm aggregates all of these output differences into a single scalar that we can minimize.

The reconstruction error can be rewritten in a form that reveals important mathematical structure. This reformulation exposes the role of input correlations in determining which weights matter most. Expanding the Frobenius norm using the identity $||\mathbf{A}||_F^2 = \text{Tr}(\mathbf{A}\mathbf{A}^T)$:

where:

• $\text{Tr}[\cdot]$: the trace operator, which sums the diagonal elements of a square matrix
• $(\mathbf{W} - \hat{\mathbf{W}})$: the weight error matrix, capturing how much each weight was shifted by quantization
• $\mathbf{X}^T$: the transpose of the input matrix

The trace operation sums the diagonal elements of the resulting matrix. What emerges from this manipulation is that the loss depends on the weight errors not in isolation, but as weighted by the matrix $\mathbf{X}\mathbf{X}^T$. This matrix captures how the inputs correlate with each other: if two input dimensions tend to activate together, errors in their corresponding weights will interact. The matrix $\mathbf{X}\mathbf{X}^T$ is in effect the unnormalized sample covariance of the layer's inputs, computed across all calibration tokens.

Define $\mathbf{H} = 2\mathbf{X}\mathbf{X}^T$, which we call the Hessian matrix (we'll explain why this name is appropriate in the next section). The loss becomes:

where:

• $\text{Tr}[\cdot]$: the trace operator
• $(\mathbf{W} - \hat{\mathbf{W}})$: the weight error matrix
• $\mathbf{H}$: the Hessian matrix ($2\mathbf{X}\mathbf{X}^T$) that captures input correlations
• $\frac{1}{2}$: a scaling factor arising from the definition of $\mathbf{H}$

This quadratic form indicates that the loss is a weighted sum of squared errors, where the weighting comes from the Hessian matrix. Errors in weights that correspond to highly active or correlated inputs are penalized more heavily than errors in weights that rarely contribute to the output. The quadratic structure is not merely convenient notation: it is what lets GPTQ to compute closed-form solutions for optimal corrections rather than relying on iterative gradient descent.

Since the rows of $\mathbf{W}$ don't interact in this expression (each output dimension is computed independently as a dot product with the inputs), we can optimize each row separately. This further simplifies our problem: instead of optimizing over all $d_{out} \times d_{in}$ weights simultaneously, we can solve $d_{out}$ independent problems, each involving only $d_{in}$ weights. For a single row $\mathbf{w} \in \mathbb{R}^{d_{in}}$:

where:

• $\mathbf{w}$: a single row of the original weight matrix (one output neuron's incoming weights)
• $\hat{\mathbf{w}}$: the corresponding row of quantized weights
• $\mathbf{H}$: the Hessian matrix, shared across all rows because all output neurons see the same inputs
• $(\mathbf{w} - \hat{\mathbf{w}})$: the error vector for this row

This weighted squared error gives us the foundation for GPTQ's optimization strategy. The quadratic structure of this loss function is important because it lets closed-form solutions for optimal weight updates. We do not need to run gradient descent or perform any iterative optimization; we can compute the exact optimal correction in a single step, which is what makes GPTQ both theoretically clean and computationally efficient.

The matrix $\mathbf{H} = 2\mathbf{X}\mathbf{X}^T$ is called the Hessian because it equals the second derivative of the reconstruction loss with respect to the weights. This naming shows a deep connection to optimization theory. The Hessian matrix characterizes the local curvature of the loss landscape, telling us how rapidly the loss changes as we move in different directions through weight space. High curvature in a particular direction means that small weight changes in that direction cause large changes in the loss; low curvature means the loss is relatively insensitive to perturbations in that direction.

Think of the Hessian as a topographic map of the loss landscape around the current weights. A standard map shows elevation as a function of position; the Hessian shows how steeply the loss rises in every direction from the current point. Directions with high curvature correspond to "steep valleys" where we must be precise. Directions with low curvature correspond to "flat plains" where we have freedom to move without much consequence. GPTQ uses this map to prioritize accuracy in the steep valleys while accepting more error on the flat plains.

To see why this matrix is second derivatives, consider the loss for a single output neuron. We want to find weights that make the neuron's output match what it would produce with the original weights:

where:

• $\mathbf{w}$: the weight vector being optimized
• $\mathbf{x}_i$: the $i$-th input vector from the calibration set
• $y_i$: the target output scalar, computed using the original weights $\mathbf{w}^*$ applied to $\mathbf{x}_i$
• $n$: the total number of calibration tokens

This is a standard least-squares objective. To find its minimum, we compute the gradient by differentiating with respect to each weight component:

where:

• $\frac{\partial \mathcal{L}}{\partial \mathbf{w}}$: the gradient vector of the loss with respect to the weights
• $(\mathbf{w} \cdot \mathbf{x}_i - y_i)$: the prediction error for the $i$-th calibration sample
• $\mathbf{x}_i$: the $i$-th input vector, which scales how each prediction error affects each weight

Taking the derivative once more, we obtain the Hessian, which tells us how the gradient itself changes as we modify the weights:

where:

• $\mathbf{H}$: the Hessian matrix (second-order derivative of the loss)
• $\frac{\partial^2 \mathcal{L}}{\partial \mathbf{w}^2}$: the matrix of all second partial derivatives
• $\mathbf{x}_i \mathbf{x}_i^T$: the outer product of each input with itself, a $d_{in} \times d_{in}$ matrix
• $\mathbf{X}$: the matrix of all calibration inputs stacked as columns

The Hessian captures the curvature of the loss landscape around the current weights. Each element of this matrix has a specific interpretation. The diagonal elements $H_{qq}$ indicate how sensitive the loss is to changes in weight $w_q$: a large value means that small perturbations to this weight cause large changes in the output, making this weight "important" in the sense that we must quantize it carefully. The off-diagonal elements $H_{qj}$ capture interactions between weights: they tell us how changes in one weight affect the optimal value of another. When two weights have a large off-diagonal element, their quantization errors are not independent, and an error in one can be partially offset by adjusting the other.

The key insight here is that the Hessian tells us which weights are important and how their importance relates to the actual data distribution. A weight connected to an input feature that never activates has zero curvature; no amount of error there will hurt the model. A weight connected to a feature that activates strongly and frequently has high curvature; errors there propagate directly into output degradation. GPTQ uses these curvature measurements as a lens through which every quantization decision is filtered.

This observation has significant practical implications. Computing the Hessian requires only a forward pass through the network on calibration data. We never need to differentiate through the model or compute any target labels. The Hessian emerges naturally from the statistics of the inputs that the layer observes during normal operation. This makes GPTQ a true post-training quantization method: we take a pre-trained model, run it on some representative data, and use the resulting statistics to guide quantization. No fine-tuning, no gradient computation through the model, and no labeled data are required.

GPTQ builds on a framework called Optimal Brain Quantization (OBQ), which itself descends from classical work on neural network pruning. The historical connection is illuminating: pruning and quantization are closely related problems. In pruning, we set certain weights exactly to zero. In quantization, we round weights to the nearest value in a discrete set. Both operations introduce error, and in both cases we want to minimize the impact on the network's output. The mathematics that describes the optimal pruning correction turns out to describe the optimal quantization correction as well.

The core insight of OBQ is that when you quantize one weight, you can compute the optimal adjustment to all remaining weights that minimizes the resulting increase in loss. This is not a heuristic or an approximation: given the quadratic structure of our loss function, there exists a closed-form formula for the best possible compensation. The formula emerges from setting the derivative of the loss (after fixing one weight to its quantized value) to zero with respect to the remaining weights and solving analytically.

Think of OBQ as a chess player who, after being forced to sacrifice a piece, immediately recalculates the optimal position for all remaining pieces. The sacrifice introduces a loss, but the player compensates by repositioning everything else to minimize the damage. In GPTQ, the "sacrifice" is the rounding error introduced when a weight is quantized, and the "repositioning" is the adjustment applied to all remaining weights.

Suppose we have decided to quantize weight $w_q$ to value $\hat{w}_q$. We want to adjust the remaining weights $\mathbf{w}_F$ (where $F$ denotes the set of weights not yet quantized) to minimize the resulting loss. Taking the derivative of the loss with respect to $\delta\mathbf{w}_F$ and setting it to zero yields the optimal adjustment:

where:

• $\delta\mathbf{w}_F$: the optimal adjustment vector for the remaining unquantized weights
• $w_q - \hat{w}_q$: the quantization error for the current weight $q$, the difference between what we wanted and what we got after rounding
• $[\mathbf{H}^{-1}]_{qq}$: the diagonal element of the inverse Hessian corresponding to weight $q$
• $[\mathbf{H}^{-1}]_{F,q}$: the elements of the $q$-th column of the inverse Hessian for all remaining weights $F$

Let us unpack this formula carefully to build intuition for what each component contributes. The numerator $w_q - \hat{w}_q$ is the quantization error for weight $q$: simply the difference between what we wanted and what we got after rounding. Larger errors require larger compensations.

The denominator $[\mathbf{H}^{-1}]_{qq}$ is the inverse curvature, measuring how flexible the loss landscape is with respect to weight $q$. When this value is large, the curvature of the loss with respect to $w_q$ is small, meaning errors in this weight are relatively easy to absorb through small adjustments elsewhere. When this value is small, the curvature is high, meaning errors in this weight are costly and difficult to compensate.

The vector $[\mathbf{H}^{-1}]_{F,q}$ determines how this error should be distributed across remaining weights. It acts like a routing map: weights that are strongly correlated with $w_q$ through the inverse Hessian receive larger adjustments, because adjusting them has the greatest effect on undoing the damage caused by the rounding error in $w_q$.

The resulting increase in loss from quantizing weight $q$ (after optimal compensation) is:

where:

• $\Delta\mathcal{L}_q$: the unavoidable increase in reconstruction error caused by quantizing weight $q$
• $(w_q - \hat{w}_q)^2$: the squared quantization error (larger errors cost more)
• $[\mathbf{H}^{-1}]_{qq}$: the inverse curvature, representing the "stiffness" of weight $q$; smaller values mean more damage per unit of rounding error

This formula tells us exactly how much each quantization decision costs, accounting for optimal compensation. It has an elegant structure: cost is proportional to the squared rounding error and inversely proportional to the inverse curvature. Weights where $[\mathbf{H}^{-1}]_{qq}$ is large can be quantized with relatively low cost even if the rounding error is substantial. Conversely, weights with small $[\mathbf{H}^{-1}]_{qq}$ values are stiff: even small errors cause significant loss increases. This cost formula is exactly what OBQ uses to decide which weight to quantize next, always choosing the one with minimum $\Delta\mathcal{L}_q$.

To make the OBQ compensation formula concrete, let us walk through a tiny numerical example with just three weights. This example strips away all the complexity to show exactly how error compensation works step by step.

Suppose a row has three weights $\mathbf{w} = [0.7, 0.3, -0.5]$ and the calibration data gives the following inverse Hessian:

We use 4-bit symmetric quantization with scale $s = 0.1$ (so quantized values are integer multiples of $0.1$ in the range $[-0.8, 0.7]$).

Step 1: Quantize $w_0 = 0.7$.

Rounding $0.7$ to the nearest multiple of $0.1$ gives $\hat{w}_0 = 0.7$. The error is $e_0 = 0.7 - 0.7 = 0.0$. Because the error is zero, no compensation is needed. The other weights remain unchanged: $\mathbf{w} = [0.7, 0.3, -0.5]$.

Step 2: Quantize $w_1 = 0.3$.

Rounding $0.3$ gives $\hat{w}_1 = 0.3$. Again the error is zero. Still no compensation needed.

Step 3: Quantize $w_2 = -0.5$.

Rounding $-0.5$ gives $\hat{w}_2 = -0.5$. Error is zero. This example happened to be perfectly aligned with the grid.

Let us now change the example slightly to see real compensation. Suppose instead $\mathbf{w} = [0.73, 0.28, -0.53]$.

Step 1: Quantize $w_0 = 0.73$.

Rounding to the nearest $0.1$ gives $\hat{w}_0 = 0.7$. The error is $e_0 = 0.73 - 0.70 = 0.03$.

The optimal compensation for the remaining weights $\{w_1, w_2\}$ is:

After compensation, the adjusted weights are $w_1 = 0.28 - 0.0075 = 0.2725$ and $w_2 = -0.53 + 0.0045 = -0.5255$.

Step 2: Quantize adjusted $w_1 = 0.2725$.

Rounding to the nearest $0.1$ gives $\hat{w}_1 = 0.3$. The error is $e_1 = 0.2725 - 0.30 = -0.0275$.

The optimal compensation for $\{w_2\}$ (the only remaining weight) is:

After compensation, $w_2 = -0.5255 + 0.00367 = -0.52183$.

Step 3: Quantize adjusted $w_2 = -0.52183$.

Rounding gives $\hat{w}_2 = -0.5$. The error is $e_2 = -0.52183 - (-0.50) = -0.02183$.

No compensation is needed because there are no remaining weights.

Comparing the two approaches. With naive round-to-nearest, the quantized weights would be $\hat{\mathbf{w}} = [0.7, 0.3, -0.5]$ with total squared weight error $(0.03)^2 + (0.02)^2 + (0.03)^2 = 0.0022$. With GPTQ, the quantized weights are $[0.7, 0.3, -0.5]$ with the same integer representations, but GPTQ's intermediate adjustments ensured that each rounding decision accounted for the correlations between weights in the Hessian. In a more complex example with finer grid spacing or larger off-diagonal Hessian elements, GPTQ would choose different integer values than naive rounding for weights $w_1$ and $w_2$ precisely because the compensation shifted their effective values before rounding. The key point is that the compensation changes the value of each weight before it is rounded, so later weights end up being rounded to different and better integer codes than they would have been under naive quantization.

The naive OBQ approach would quantize weights one at a time, choosing at each step the weight whose quantization causes the smallest loss increase $\Delta\mathcal{L}_q$. While this greedy strategy seems sensible, its computational cost is prohibitive. Each step requires examining all remaining weights to find the best candidate, and after quantizing each weight, we must update the inverse Hessian. This requires $O(d_{in}^2)$ operations per weight, yielding $O(d_{in}^3)$ complexity per row and $O(d_{in}^4)$ for the entire layer. For a model where $d_{in}$ can be 4096 or larger, $O(d_{in}^4)$ operations are completely impractical.

GPTQ makes three key modifications that reduce the total complexity to $O(d_{in}^3)$ while maintaining nearly identical accuracy. The central observation is that the optimal quantization order matters less than one might expect. The compensation mechanism is powerful enough that processing weights in a fixed sequential order yields results almost as good as the optimal greedy order. What matters most is performing the compensation correctly after each quantization step.

Think of the three optimizations as different aspects of making the same core algorithm more efficient, like optimizing a factory assembly line. Column-wise processing batches the work across all output neurons simultaneously (vectorization). Lazy batch updates reduce the number of times we touch memory by accumulating changes before applying them (cache efficiency). Cholesky-based inverse updates replace a naive matrix inversion with a factorization that can be maintained cheaply as we process each column (algorithmic efficiency).

Column-wise processing. Instead of quantizing weights one at a time in optimal order, GPTQ processes all weights in a fixed left-to-right column order. The important insight is that all rows of the weight matrix share the same Hessian, because all output neurons see the same inputs. This means we can process all rows simultaneously for a given column, quantizing column $q$ of every output neuron at once. This vectorization turns a sequential loop over rows into a single matrix operation, achieving a large constant-factor speedup.

Lazy batch updates. Rather than updating all remaining weights after each individual column quantization, GPTQ accumulates updates in blocks (typically of size 128) and applies them all at once when the block is complete. This improves cache efficiency sharply because memory access patterns become more localized. Weight matrix rows are stored contiguously in memory, and processing a full block before writing updates means each memory location is accessed sequentially rather than randomly.

Cholesky-based inverse updates. Computing and maintaining the full inverse Hessian naively would cost $O(d_{in}^2)$ per column after the initial $O(d_{in}^3)$ inversion, yielding $O(d_{in}^3)$ total work just for the updates. GPTQ instead precomputes the Cholesky decomposition $\mathbf{H}^{-1} = \mathbf{L}\mathbf{L}^T$ where $\mathbf{L}$ is lower triangular. The triangular structure of $\mathbf{L}$ directly encodes the sequential dependencies between weights: the $q$-th column of $\mathbf{L}$ gives the compensation coefficients needed when processing column $q$. After processing column $q$, we simply move to the next column of $\mathbf{L}$ without any additional update. This reduces the per-column cost from $O(d_{in}^2)$ to $O(d_{in})$ after the initial factorization.

Here is the complete GPTQ procedure in detail:

1. Collect calibration data. Run a small set of representative examples through the model, using forward hooks to record each layer's inputs $\mathbf{X}$ as the data flows through.

2. Compute the Hessian. For each layer, compute $\mathbf{H} = 2\mathbf{X}\mathbf{X}^T$ and add a small diagonal dampening term $\lambda \mathbf{I}$ for numerical stability. Compute the inverse $\mathbf{H}^{-1}$ via Cholesky decomposition.

3. Process each row of the weight matrix. For each row $\mathbf{w}$:

   a. For each column $q$ from 0 to $d_{in} - 1$:

   • Quantize: $\hat{w}_q = \text{quantize}(w_q)$
   • Compute error: $\delta_q = w_q - \hat{w}_q$
   • Update remaining weights: $w_{q+1:} \leftarrow w_{q+1:} - \frac{\delta_q}{[\mathbf{H}^{-1}]_{qq}} \cdot [\mathbf{H}^{-1}]_{q+1:,q}$

   b. Store the quantized row $\hat{\mathbf{w}}$ along with per-group scale and zero-point parameters.

4. Replace the layer's weights with their quantized integer values plus the associated dequantization metadata.

The order in which columns are processed affects accuracy. The default left-to-right order works reasonably well, but ActOrder (processing columns in decreasing order of Hessian diagonal) often improves results by quantizing the most important weights first, when there are still many remaining weights available to absorb compensation. We will explore ActOrder in detail later in this chapter.

The inverse Hessian update is a necessary mathematical component that lets efficient processing. After quantizing column $q$, we need to update the inverse Hessian to reflect that column $q$ is no longer a free variable. The weight at column $q$ has been fixed to its quantized value; it can no longer be adjusted to compensate for future quantization errors. Mathematically, we need to compute a new inverse Hessian that covers only the remaining unquantized weights $F \setminus \{q\}$.

Let $\mathbf{H}_F^{-1}$ denote the inverse Hessian restricted to the remaining (unquantized) weights. After removing column and row $q$, the new inverse is given by the Schur complement formula:

where:

• $[\mathbf{H}_{F \setminus q}^{-1}]_{ij}$: the updated inverse Hessian element at row $i$, column $j$, after removing weight $q$
• $[\mathbf{H}_F^{-1}]_{ij}$: the current inverse Hessian element before the update
• $[\mathbf{H}_F^{-1}]_{iq}, [\mathbf{H}_F^{-1}]_{qj}$: elements from the $q$-th column and row of the current inverse Hessian
• $[\mathbf{H}_F^{-1}]_{qq}$: the diagonal element corresponding to weight $q$
• $i, j$: indices ranging over the remaining unquantized weights in $F \setminus \{q\}$

This formula is a rank-one update, subtracting an outer product that corresponds to removing the degree of freedom at position $q$. The intuition is that by fixing weight $q$, we eliminate one dimension from our optimization space. The remaining weights must "account for" the fact that they can no longer rely on $w_q$ to absorb any future compensation. The Schur complement formula computes exactly how the correlations between remaining weights change when one weight is removed from consideration.

Computing this update naively as a dense matrix operation would cost $O(d^2)$ per column. Over all $d$ columns, this would total $O(d^3)$, which is the same as just re-inverting the Hessian from scratch each time. The Cholesky decomposition gives the escape from this cost.

The Cholesky factorization gives us $\mathbf{H}^{-1} = \mathbf{L}\mathbf{L}^T$ where $\mathbf{L}$ is lower triangular. This factorization is particularly useful because the triangular structure directly encodes the sequential dependencies between weights. For column $q$, the required compensation coefficients $[\mathbf{H}^{-1}]_{q+1:,q}$ are exactly the entries in the $q$-th column of $\mathbf{L}$ (below the diagonal). The update formula simplifies to reading off the next column of $\mathbf{L}$ rather than performing a full matrix update, reducing the per-column cost to $O(d)$ after the initial $O(d^3)$ Cholesky factorization. This is why the total complexity of GPTQ is $O(d_{in}^3)$ per layer: dominated by the one-time Cholesky factorization, with all subsequent operations being linear in $d_{in}$.

GPTQ requires calibration data to estimate the Hessian. The quality and quantity of this data affect quantization results, yet practitioners often treat calibration data as an afterthought. Calibration is a significant design decision.

The calibration data does not need labels. GPTQ only performs forward passes through the model to collect layer activations, so any unlabeled text will do. What matters is that the calibration data covers the kinds of inputs the model will encounter at deployment. The Hessian is an empirical estimate of input statistics: run calibration on text that looks nothing like your deployment distribution, and the Hessian will mis-estimate which weights matter, leading to worse quantization choices.

Think of the calibration data as a "map of typical inputs." If you are quantizing a general-purpose language model, a diverse sample that combines web and book text with source code will give the Hessian a realistic picture of the model's inputs. If you are quantizing a specialized code-generation model, calibrating on web text means the Hessian will reflect a distribution very different from the model's actual use, and the quantization will be suboptimal for code generation tasks even if it looks fine on general language benchmarks.

The key parameters for calibration are:

• Quantity. GPTQ typically uses 128 to 1,024 samples. More samples give a better Hessian estimate, reducing variance in the diagonal and off-diagonal elements. In practice, 128 samples of 2,048 tokens each (totaling 262,144 tokens) is sufficient for most models, with diminishing returns beyond 512 samples.

• Representativeness. The calibration data should resemble the data the model will see at inference. Using random text to calibrate a code model yields worse results than using code samples, because the Hessian will reflect different activation patterns.

• Sequence length. Longer sequences give more tokens per sample, leading to a more accurate Hessian estimate. Most implementations use 2,048 tokens per sequence.

Common choices include random samples from C4 (a large web text corpus), WikiText (cleaned Wikipedia text), or domain-specific data for specialized models. The C4 and WikiText defaults work well for general-purpose models. For specialized deployments, investing in domain-specific calibration data consistently improves results.

Standard per-tensor or per-channel quantization uses a single scale and zero point for many weights. For weights with non-uniform distributions within a channel, a single scale factor forces the quantization grid to accommodate the full dynamic range, wasting precision on weights that are clustered in a small portion of that range. GPTQ often employs group quantization to address this limitation.

In group quantization, weights are divided into contiguous groups (commonly 128 weights each) that each have their own scale and zero-point parameters. This is a middle ground between per-tensor quantization (cheapest, least accurate) and per-weight quantization (most expensive, most accurate). The overhead of storing group-wise quantization parameters is small relative to the accuracy gains.

Think of group quantization as using local rulers instead of a single global ruler to measure distances. If you have a collection of objects ranging from 1 mm to 1 m in size, a single ruler calibrated to handle the full range will be very imprecise for the small objects. Multiple rulers, each calibrated for a smaller range, will be more precise throughout. Group quantization applies this same principle to weight distributions.

Group quantization gives a middle ground between accuracy and overhead:

• Per-tensor quantization. One scale for all weights in the layer. Minimum storage overhead but potentially high quantization error if weight distributions vary across channels or weight groups.

• Per-group quantization. Separate scales for groups of 128 weights. The overhead is small: one FP16 scale per 128 INT4 weights adds 16 bits for every 4 $\times$ 128 = 512 bits of weight data, which is 0.125 bits per weight, increasing the effective bit-width from 4.0 to 4.125 bits. This tiny overhead often yields substantially better accuracy.

• Per-weight quantization. Individual scales for each weight. Maximum accuracy but impractical overhead: storing one FP16 scale per INT4 weight doubles the memory usage, eliminating the benefit of quantization entirely.

With group size 128 and 4-bit weights, the effective storage is approximately 4.125 bits per weight, a tiny overhead for significant accuracy gains. In practice, group sizes of 64 to 128 offer the best tradeoff, with group size 128 being the most common choice in deployed systems.

Let us see GPTQ in practice. We will start by implementing the core algorithm from scratch to build understanding, then show how to use the AutoGPTQ library for production use.

Layer output error is a better measure of quantization quality than weight error. Weight error measures how much the weights changed; output error measures how much the layer's predictions changed. Since the model's downstream behavior depends entirely on layer outputs rather than raw weights, output error is the relevant metric.

We want to verify that GPTQ's weight-space optimization translates into output-space improvements. If it did not, the Hessian-weighted objective would be clever mathematics without practical payoff. In practice, the improvement in output error is even more pronounced than the improvement in weight error, precisely because the Hessian weighting ensures we concentrate accuracy where the outputs are most sensitive.

For production use, the AutoGPTQ library gives an optimized implementation with GPU acceleration, including CUDA kernels for fast INT4 matrix multiplication during inference. The library handles all the details of Hessian computation, Cholesky factorization, and group quantization, exposing a clean API that takes a model and calibration data as inputs and produces a quantized model ready for deployment.

The workflow has three phases. In the first phase, you prepare calibration data by tokenizing representative text samples and collecting them into a list. In the second phase, you configure the quantization parameters (bits, group size, column ordering) and run the quantization algorithm. In the third phase, you save the quantized model to disk so it can be loaded quickly for inference without repeating the quantization process.

Understanding the key AutoGPTQ parameters helps you make informed tradeoffs when quantizing a model for deployment:

• bits: Target bit width, typically 4 or 8. 4-bit is the most common choice for deployment. It offers the best size reduction. 3-bit is possible but causes more accuracy degradation for most models.
• group_size: Number of weights sharing quantization parameters. 128 is the standard choice. It offers a good balance between accuracy and overhead. Smaller group sizes (64, 32) improve accuracy at the cost of slightly more storage for scale parameters.
• desc_act: Whether to use ActOrder (activation ordering), which processes columns by decreasing Hessian diagonal. Enabling this typically improves perplexity by 0.1 to 0.5 points but slightly increases quantization time.
• damp_percent: Dampening factor added to the Hessian diagonal as a fraction of its mean value. This prevents numerical instability when the Hessian is nearly singular. The default of 0.01 (1%) works well for most models.

The order in which GPTQ processes columns affects the final quantization error. The default left-to-right order treats all weights equally, but weights differ substantially in their importance as measured by the Hessian diagonal. Processing a highly important weight late in the sequence, after many others have already been quantized and their compensation capacity used up, is suboptimal. That important weight receives compensation from fewer remaining weights than it would have if processed earlier.

ActOrder (activation ordering) addresses this by processing columns in decreasing order of their Hessian diagonal values $H_{qq}$. The intuition is simple: weights with larger $H_{qq}$ have greater impact on the output and should be quantized first, when the maximum number of other weights are still available to absorb compensation. As important weights are quantized early, their errors are distributed among many remaining weights, each absorbing a small fraction. Less important weights are quantized later, when fewer candidates remain for compensation, but this matters less because their errors have smaller impact on the output anyway.

Think of ActOrder as a surgeon strategy for high-stakes operations. A surgical team handles the most necessary, time-sensitive procedures first, while the available support staff is at its fullest. Routine procedures are handled later, when some staff may have moved on, because the consequences of imperfect execution are smaller.

ActOrder typically improves perplexity by 0.1 to 0.5 points at 4-bit precision, with the benefit more pronounced for smaller models where each individual weight matters more. For large models like LLaMA-65B, the baseline GPTQ without ActOrder already performs well, and the marginal improvement from ActOrder is smaller. For 7B and 13B models, ActOrder is often worth enabling.

The ultimate test of quantization quality is model performance on downstream tasks. For language models, perplexity on held-out text gives a principled proxy: it measures how well the model predicts unseen text, capturing both fluency and factual accuracy in a single number. Lower perplexity means better predictions.

Perplexity is defined as the exponentiated average negative log-likelihood per token:

where:

• $\text{PPL}$: perplexity, where lower values indicate better language modeling
• $N$: the total number of tokens in the evaluation set
• $x_i$: the $i$-th token in the sequence
• $P(x_i \mid x_{<i})$: the model's predicted probability for token $x_i$ given all preceding tokens
• $\exp(\cdot)$: the exponential function that converts average log-probability into a perplexity score

A perplexity of 5 means the model is, on average, as uncertain as if it had to choose uniformly among 5 equally likely options for each token. A perplexity of 4.5 instead of 5.0 is a real improvement in language understanding. The following table shows typical GPTQ performance for LLaMA models evaluated on WikiText-2:

The perplexity degradation shrinks with model size, and this is not a coincidence. Larger models have more parameters, which means more redundancy in the weight space. More redundancy means that the compensation mechanism has more "slack" to work with: when one weight is rounded away from its optimal value, there are more other weights available to absorb the error. At 65B parameters, the difference between FP16 and GPTQ 4-bit is barely measurable on most benchmarks. The effective information density of a quantized 65B model exceeds that of a full-precision 7B model, making GPTQ quantization a worthwhile trade at every scale.

The relationship between model size and quantization robustness has an important practical implication: if you must choose between a smaller full-precision model and a larger quantized model of similar memory size, the larger quantized model usually wins on accuracy. A GPTQ 4-bit LLaMA-13B fits in roughly the same GPU memory as an FP16 LLaMA-3B, but performs substantially better on almost every benchmark.

GPTQ is a major advance over naive quantization, but understanding its limitations is needed for deploying it effectively. Each limitation also suggests where further research has gone, and where you might need to supplement GPTQ with additional techniques.

Calibration sensitivity. Quantization quality depends on how well the calibration data matches the model's deployment inputs. If you calibrate a code-generation model on web text, the Hessian will reflect the statistics of web text activations, not code activations. Code has more structured patterns, longer token dependencies, and different vocabulary distributions. A model quantized with mismatched calibration data will precisely quantize weights that matter for web text while representing weights important for code less accurately. For general-purpose models used on general text, the standard C4 or WikiText calibration works well. For specialized deployments, invest in domain-specific calibration data.

Computational cost. GPTQ quantization is materially slower than naive round-to-nearest. Quantizing a 7B model takes around 10 to 30 minutes on a modern GPU, and the time scales roughly as $O(d^3)$ per layer with the Hessian computation and Cholesky factorization. Quantizing a 65B model can take several hours. This is acceptable for one-time quantization, but makes GPTQ unsuitable for anything requiring on-the-fly quantization during inference. You quantize once and deploy; you do not requantize per request. The computational cost is also why GPTQ is usually run on a GPU: CPU-based Cholesky factorization for large hidden dimensions would be impractically slow.

Layer-wise approximation. By optimizing each layer independently, GPTQ ignores how quantization errors compound across layers. In a deep transformer, a small error introduced at layer 1 might be amplified by subsequent attention and FFN computations, reaching layer 20 in a magnified form. GPTQ's per-layer optimization minimizes each layer's individual error without accounting for this cross-layer amplification. This approximation is empirically reasonable (the results in the perplexity table above show this), but it is not exact. Methods that incorporate cross-layer information, while more expensive, can in principle do better.

Activation quantization not included. GPTQ only quantizes weights, not activations. For inference on specialized hardware accelerators that require both weight and activation quantization (such as certain edge devices or custom chips designed for INT4 compute), GPTQ alone is insufficient. Additional quantization-aware training or separate activation quantization techniques are needed to reach full INT4 inference on such hardware.

Outlier sensitivity. The Hessian estimation can be skewed by outlier activations. As discussed in the chapter on INT8 quantization, transformer models sometimes produce extreme activation values in certain dimensions that can be orders of magnitude larger than typical values. These outliers disproportionately influence the Hessian: a few very large activations in one dimension will make the Hessian diagonal for that dimension enormous, leading GPTQ to treat it as critically sensitive and focus excessive precision there, potentially neglecting other dimensions. The dampening factor helps by reducing the relative influence of outliers, but does not eliminate it. Techniques like SmoothQuant, which pre-scales activations to reduce outlier magnitude, can be combined with GPTQ to address this issue.

Despite these limitations, GPTQ remains one of the most effective post-training quantization methods for LLMs. Its combination of theoretical foundation (optimal error compensation from second-order information) and practical efficiency (Cholesky-based linear updates) set the standard for weight quantization. The success of GPTQ inspired subsequent work like AWQ (Activation-aware Weight Quantization), which we will explore in the next chapter. AWQ takes a complementary approach: instead of compensating for quantization errors, it identifies and protects the most important weights from quantization entirely, often achieving even better results on models with strong activation outliers.

GPTQ turns weight quantization from a simple rounding problem into a principled optimization problem grounded in second-order information theory. By compensating for each weight's quantization error through Hessian-guided updates to remaining weights, GPTQ reaches reconstruction error far below what naive quantization can manage.

The key elements that make GPTQ work are:

• The layer-wise reconstruction objective. Optimizing the difference between original and quantized layer outputs (rather than weights directly) minimizes the metric relevant to downstream model performance.
• The Hessian matrix. Computed cheaply from a single forward pass on calibration data, the Hessian captures which weights are important by measuring the curvature of the loss with respect to each weight. Weights connected to frequently active input features have high curvature and must be quantized precisely.
• The optimal compensation formula. When a weight is quantized, a closed-form formula distributes the rounding error across all remaining weights proportionally to their correlations with the quantized weight via the inverse Hessian. This redistribution recovers much of the accuracy lost to rounding.
• Cholesky-based efficiency. Precomputing the Cholesky factorization of the inverse Hessian reduces the per-column update cost from $O(d^2)$ to $O(d)$, making the entire quantization procedure tractable at scale with $O(d^3)$ total complexity per layer.
• Group quantization. Using separate scale parameters for groups of 128 weights captures local weight distribution variations, adding only 0.125 bits per weight in overhead while substantially improving accuracy.
• ActOrder. Processing high-sensitivity columns first, when more remaining weights are available for compensation, improves perplexity by 0.1 to 0.5 points at 4-bit precision.

For you, GPTQ means that 4-bit quantization is practical for production LLM deployment. Models lose minimal accuracy while fitting in a fraction of the original memory. A 65B parameter model that required a multi-GPU server in FP16 now runs on a single consumer GPU. The calibration data requirements and quantization time are modest one-time costs for large and permanent efficiency gains.

Understanding GPTQ also gives insight into the broader space of quantization methods. The principle of error compensation, rather than pure rounding, appears in various forms across modern quantization techniques. The Hessian as a measure of weight importance recurs in AWQ, QuIP, and AQLM. The layer-wise decomposition appears in nearly every practical post-training quantization method. Whether you are deploying a quantized model or developing new efficiency methods, the foundations established by GPTQ remain needed knowledge for anyone working in this space.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about GPTQ.