GPTQ › Worked Example: Error Compensation in Three Weights

To make the OBQ compensation formula concrete, let us walk through a tiny numerical example with just three weights. This example strips away all the complexity to show exactly how error compensation works step by step.

Suppose a row has three weights $\mathbf{w} = [0.7, 0.3, -0.5]$ and the calibration data gives the following inverse Hessian:

$$\mathbf{H}^{-1} = \begin{pmatrix} 2.0 & 0.5 & -0.3 \\ 0.5 & 1.5 & 0.2 \\ -0.3 & 0.2 & 1.0 \end{pmatrix}$$

We use 4-bit symmetric quantization with scale $s = 0.1$ (so quantized values are integer multiples of $0.1$ in the range $[-0.8, 0.7]$).

Step 1: Quantize $w_0 = 0.7$.

Rounding $0.7$ to the nearest multiple of $0.1$ gives $\hat{w}_0 = 0.7$. The error is $e_0 = 0.7 - 0.7 = 0.0$. Because the error is zero, no compensation is needed. The other weights remain unchanged: $\mathbf{w} = [0.7, 0.3, -0.5]$.

Step 2: Quantize $w_1 = 0.3$.

Rounding $0.3$ gives $\hat{w}_1 = 0.3$. Again the error is zero. Still no compensation needed.

Step 3: Quantize $w_2 = -0.5$.

Rounding $-0.5$ gives $\hat{w}_2 = -0.5$. Error is zero. This example happened to be perfectly aligned with the grid.

Let us now change the example slightly to see real compensation. Suppose instead $\mathbf{w} = [0.73, 0.28, -0.53]$.

Step 1: Quantize $w_0 = 0.73$.

Rounding to the nearest $0.1$ gives $\hat{w}_0 = 0.7$. The error is $e_0 = 0.73 - 0.70 = 0.03$.

The optimal compensation for the remaining weights $\{w_1, w_2\}$ is:

$$\delta\mathbf{w}_{\{1,2\}} = -\frac{e_0}{[\mathbf{H}^{-1}]_{00}} \cdot [\mathbf{H}^{-1}]_{\{1,2\},0} = -\frac{0.03}{2.0} \cdot \begin{pmatrix} 0.5 \\ -0.3 \end{pmatrix} = \begin{pmatrix} -0.0075 \\ 0.0045 \end{pmatrix}$$

After compensation, the adjusted weights are $w_1 = 0.28 - 0.0075 = 0.2725$ and $w_2 = -0.53 + 0.0045 = -0.5255$.

Step 2: Quantize adjusted $w_1 = 0.2725$.

Rounding to the nearest $0.1$ gives $\hat{w}_1 = 0.3$. The error is $e_1 = 0.2725 - 0.30 = -0.0275$.

The optimal compensation for $\{w_2\}$ (the only remaining weight) is:

$$\delta w_2 = -\frac{-0.0275}{[\mathbf{H}^{-1}]_{11}} \cdot [\mathbf{H}^{-1}]_{2,1} = -\frac{-0.0275}{1.5} \cdot 0.2 = 0.00367$$

After compensation, $w_2 = -0.5255 + 0.00367 = -0.52183$.

Step 3: Quantize adjusted $w_2 = -0.52183$.

Rounding gives $\hat{w}_2 = -0.5$. The error is $e_2 = -0.52183 - (-0.50) = -0.02183$.

No compensation is needed because there are no remaining weights.

Comparing the two approaches. With naive round-to-nearest, the quantized weights would be $\hat{\mathbf{w}} = [0.7, 0.3, -0.5]$ with total squared weight error $(0.03)^2 + (0.02)^2 + (0.03)^2 = 0.0022$. With GPTQ, the quantized weights are $[0.7, 0.3, -0.5]$ with the same integer representations, but GPTQ's intermediate adjustments ensured that each rounding decision accounted for the correlations between weights in the Hessian. In a more complex example with finer grid spacing or larger off-diagonal Hessian elements, GPTQ would choose different integer values than naive rounding for weights $w_1$ and $w_2$ precisely because the compensation shifted their effective values before rounding. The key point is that the compensation changes the value of each weight before it is rounded, so later weights end up being rounded to different and better integer codes than they would have been under naive quantization.