GPTQ › Implementation with AutoGPTQ
Language AI HandbookAll chapters GPTQ › Implementation with AutoGPTQOne section · Sections in this chapter10 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us see GPTQ in practice. We will start by implementing the core algorithm from scratch to build understanding, then show how to use the AutoGPTQ library for production use.
First, we load a small model to examine before quantization:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load a small model for demonstration
model_name = "facebook/opt-125m"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name, torch_dtype=torch.float32
)Let us examine the original model's memory footprint to establish a baseline:
def count_parameters(model):
"""Count total parameters and calculate memory in FP16."""
total_params = sum(p.numel() for p in model.parameters())
memory_fp16_mb = total_params * 2 / (1024**2) # 2 bytes per FP16
return total_params, memory_fp16_mb
total_params, memory_mb = count_parameters(model)
int4_memory_mb = memory_mb / 4Total parameters: 125,239,296 FP16 memory: 238.88 MB Estimated INT4 memory: 59.72 MB
The output shows that 4-bit quantization reduces the memory footprint to a quarter of its FP16 size. For this 125M parameter model, this means dropping from over 200 MB to roughly 60 MB. For a 7B parameter model, the same ratio shrinks 14 GB to 3.5 GB, making the model runnable on a laptop GPU. For a 65B model, it means fitting 130 GB into about 35 GB on a single high-end consumer GPU.
Now let us implement the core GPTQ components. We start with the Hessian computation:
def compute_hessian(inputs: torch.Tensor) -> torch.Tensor:
"""
Compute the Hessian matrix H = 2 * X @ X^T for GPTQ.
Args:
inputs: Layer inputs of shape (n_samples, seq_len, hidden_dim)
or (n_tokens, hidden_dim)
Returns:
Hessian matrix of shape (hidden_dim, hidden_dim)
"""
# Flatten to (n_tokens, hidden_dim)
if inputs.dim() == 3:
inputs = inputs.reshape(-1, inputs.shape[-1])
# H = 2 * X^T @ X (note: we transpose so each row is a feature)
H = 2 * inputs.T @ inputs
# Add small diagonal for numerical stability
H += 1e-4 * torch.eye(H.shape[0], device=H.device, dtype=H.dtype)
return HThe 1e-4 diagonal dampening prevents numerical issues when the Hessian is nearly singular, which can happen when some input features are nearly linearly dependent or rarely active. In practice, the dampening factor is typically set as a fraction of the mean Hessian diagonal (0.1% to 1%), but the absolute value works well for our demonstration.
Next, we implement the quantization and dequantization primitives:
def quantize_weight(
weight: float, scale: float, zero_point: int, n_bits: int = 4
) -> int:
"""Quantize a single weight value to n-bit integer."""
qmin, qmax = 0, (2**n_bits) - 1
q = round(weight / scale + zero_point)
return max(qmin, min(qmax, q))
def dequantize_weight(q: int, scale: float, zero_point: int) -> float:
"""Convert quantized integer back to float."""
return scale * (q - zero_point)Here is the core GPTQ algorithm for a single row of weights. The key loop quantizes each column, computes the error, and immediately applies the compensation to all remaining weights:
def gptq_quantize_row(
weights: torch.Tensor, # Shape: (d_in,)
H_inv: torch.Tensor, # Inverse Hessian, shape: (d_in, d_in)
n_bits: int = 4,
group_size: int = 128,
) -> tuple[torch.Tensor, list]:
"""
Apply GPTQ quantization to a single row of weights.
Returns:
Tuple of (quantized_weights, quantization_params)
"""
d_in = weights.shape[0]
weights = weights.clone().float()
quantized = torch.zeros_like(weights, dtype=torch.int8)
params = [] # Store (scale, zero_point) for each group
# Process columns left to right
for col in range(d_in):
# Determine group for this column
group_idx = col // group_size
group_start = group_idx * group_size
group_end = min(group_start + group_size, d_in)
# Compute scale and zero point for this group (if at group boundary)
if col == group_start:
group_weights = weights[group_start:group_end]
w_min, w_max = (
group_weights.min().item(),
group_weights.max().item(),
)
# Asymmetric quantization
qmin, qmax = 0, (2**n_bits) - 1
scale = (w_max - w_min) / (qmax - qmin) if w_max > w_min else 1.0
zero_point = round(-w_min / scale) if scale > 0 else 0
zero_point = max(qmin, min(qmax, zero_point))
params.append((scale, zero_point))
scale, zero_point = params[group_idx]
# Quantize current weight
w = weights[col].item()
q = quantize_weight(w, scale, zero_point, n_bits)
quantized[col] = q
# Compute quantization error
w_hat = dequantize_weight(q, scale, zero_point)
error = w - w_hat
# Update remaining weights to compensate for error
if col < d_in - 1 and H_inv[col, col] > 1e-10:
# Optimal update: delta_w_F = -error / H_inv[q,q] * H_inv[F,q]
update = -error / H_inv[col, col] * H_inv[col + 1 :, col]
weights[col + 1 :] += update
return quantized, paramsNotice how the compensation update modifies weights[col + 1:] in place. Each subsequent weight is adjusted before it is rounded, so the rounding decision for column is made on the corrected value rather than the original. This sequential correction is the essence of GPTQ: every quantization step immediately propagates its error into the remaining weights, giving them a chance to compensate.
Let us show this on a sample weight matrix:
# Create a sample weight matrix and random inputs
torch.manual_seed(42)
d_out, d_in = 128, 256
sample_weights = torch.randn(d_out, d_in) * 0.1
sample_inputs = torch.randn(512, d_in) # 512 calibration tokens
# Compute Hessian and its inverse
H = compute_hessian(sample_inputs)
H_inv = torch.linalg.inv(H)# Quantize one row as demonstration
original_row = sample_weights[0]
quantized_row, quant_params = gptq_quantize_row(original_row, H_inv)
# Dequantize to compare
dequantized_row = torch.zeros_like(original_row)
group_size = 128
for col in range(d_in):
group_idx = col // group_size
scale, zero_point = quant_params[group_idx]
dequantized_row[col] = dequantize_weight(
quantized_row[col].item(), scale, zero_point
)
# Compute reconstruction error
mse = ((original_row - dequantized_row) ** 2).mean().item()
# Compare with naive round-to-nearest
naive_scale = (original_row.max() - original_row.min()) / 15
naive_zp = round(-original_row.min().item() / naive_scale.item())
naive_q = torch.round(original_row / naive_scale + naive_zp).clamp(0, 15)
naive_deq = naive_scale * (naive_q - naive_zp)
naive_mse = ((original_row - naive_deq) ** 2).mean().item()
weight_mse_ratio = mse / naive_mseGPTQ quantization MSE: 0.000135 Naive quantization MSE: 0.000086 GPTQ raw weight MSE is 1.56x naive MSE
GPTQ does not minimize unweighted error between the original and quantized weights. In this example, its raw weight MSE is higher than naive round-to-nearest. GPTQ instead uses the Hessian to spend the error budget where it has the least effect on layer outputs. The output-space comparison below is therefore the meaningful test of whether compensation worked.
