GPTQ › The Hessian and Its Role

The matrix $\mathbf{H} = 2\mathbf{X}\mathbf{X}^T$ is called the Hessian because it equals the second derivative of the reconstruction loss with respect to the weights. This naming shows a deep connection to optimization theory. The Hessian matrix characterizes the local curvature of the loss landscape, telling us how rapidly the loss changes as we move in different directions through weight space. High curvature in a particular direction means that small weight changes in that direction cause large changes in the loss; low curvature means the loss is relatively insensitive to perturbations in that direction.

Think of the Hessian as a topographic map of the loss landscape around the current weights. A standard map shows elevation as a function of position; the Hessian shows how steeply the loss rises in every direction from the current point. Directions with high curvature correspond to "steep valleys" where we must be precise. Directions with low curvature correspond to "flat plains" where we have freedom to move without much consequence. GPTQ uses this map to prioritize accuracy in the steep valleys while accepting more error on the flat plains.

To see why this matrix is second derivatives, consider the loss for a single output neuron. We want to find weights that make the neuron's output match what it would produce with the original weights:

$$\mathcal{L}(\mathbf{w}) = \sum_{i=1}^{n} (\mathbf{w} \cdot \mathbf{x}_i - y_i)^2$$

where:

• $\mathbf{w}$: the weight vector being optimized
• $\mathbf{x}_i$: the $i$-th input vector from the calibration set
• $y_i$: the target output scalar, computed using the original weights $\mathbf{w}^*$ applied to $\mathbf{x}_i$
• $n$: the total number of calibration tokens

This is a standard least-squares objective. To find its minimum, we compute the gradient by differentiating with respect to each weight component:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{w}} = 2\sum_{i=1}^{n} (\mathbf{w} \cdot \mathbf{x}_i - y_i)\mathbf{x}_i$$

where:

• $\frac{\partial \mathcal{L}}{\partial \mathbf{w}}$: the gradient vector of the loss with respect to the weights
• $(\mathbf{w} \cdot \mathbf{x}_i - y_i)$: the prediction error for the $i$-th calibration sample
• $\mathbf{x}_i$: the $i$-th input vector, which scales how each prediction error affects each weight

Taking the derivative once more, we obtain the Hessian, which tells us how the gradient itself changes as we modify the weights:

$$\mathbf{H} = \frac{\partial^2 \mathcal{L}}{\partial \mathbf{w}^2} = 2\sum_{i=1}^{n} \mathbf{x}_i \mathbf{x}_i^T = 2\mathbf{X}\mathbf{X}^T$$

where:

• $\mathbf{H}$: the Hessian matrix (second-order derivative of the loss)
• $\frac{\partial^2 \mathcal{L}}{\partial \mathbf{w}^2}$: the matrix of all second partial derivatives
• $\mathbf{x}_i \mathbf{x}_i^T$: the outer product of each input with itself, a $d_{in} \times d_{in}$ matrix
• $\mathbf{X}$: the matrix of all calibration inputs stacked as columns

The Hessian captures the curvature of the loss landscape around the current weights. Each element of this matrix has a specific interpretation. The diagonal elements $H_{qq}$ indicate how sensitive the loss is to changes in weight $w_q$: a large value means that small perturbations to this weight cause large changes in the output, making this weight "important" in the sense that we must quantize it carefully. The off-diagonal elements $H_{qj}$ capture interactions between weights: they tell us how changes in one weight affect the optimal value of another. When two weights have a large off-diagonal element, their quantization errors are not independent, and an error in one can be partially offset by adjusting the other.

The key insight here is that the Hessian tells us which weights are important and how their importance relates to the actual data distribution. A weight connected to an input feature that never activates has zero curvature; no amount of error there will hurt the model. A weight connected to a feature that activates strongly and frequently has high curvature; errors there propagate directly into output degradation. GPTQ uses these curvature measurements as a lens through which every quantization decision is filtered.

This observation has significant practical implications. Computing the Hessian requires only a forward pass through the network on calibration data. We never need to differentiate through the model or compute any target labels. The Hessian emerges naturally from the statistics of the inputs that the layer observes during normal operation. This makes GPTQ a true post-training quantization method: we take a pre-trained model, run it on some representative data, and use the resulting statistics to guide quantization. No fine-tuning, no gradient computation through the model, and no labeled data are required.