GPTQ › Using AutoGPTQ in Practice
Language AI HandbookAll chapters GPTQ › Using AutoGPTQ in PracticeOne section · Sections in this chapter12 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
For production use, the AutoGPTQ library gives an optimized implementation with GPU acceleration, including CUDA kernels for fast INT4 matrix multiplication during inference. The library handles all the details of Hessian computation, Cholesky factorization, and group quantization, exposing a clean API that takes a model and calibration data as inputs and produces a quantized model ready for deployment.
The workflow has three phases. In the first phase, you prepare calibration data by tokenizing representative text samples and collecting them into a list. In the second phase, you configure the quantization parameters (bits, group size, column ordering) and run the quantization algorithm. In the third phase, you save the quantized model to disk so it can be loaded quickly for inference without repeating the quantization process.
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model and tokenizer
model_name = "meta-llama/Llama-2-7b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name, torch_dtype=torch.float16, device_map="auto"
)
# Prepare calibration data
calibration_data = load_dataset("c4", "en", split="train", streaming=True)
calibration_samples = []
for sample in calibration_data.take(128):
tokenized = tokenizer(sample["text"], truncation=True, max_length=2048)
calibration_samples.append(tokenized["input_ids"])
# Configure quantization
quantize_config = BaseQuantizeConfig(
bits=4, # 4-bit quantization
group_size=128, # Group size for scales
desc_act=True, # Enable ActOrder (activation ordering)
damp_percent=0.01, # Dampening for numerical stability
)
# Quantize the model
model = AutoGPTQForCausalLM.from_pretrained(model_name, quantize_config)
model.quantize(calibration_samples)
# Save the quantized model
model.save_quantized("llama-2-7b-gptq-4bit")Key Parameters
Understanding the key AutoGPTQ parameters helps you make informed tradeoffs when quantizing a model for deployment:
- bits: Target bit width, typically 4 or 8. 4-bit is the most common choice for deployment. It offers the best size reduction. 3-bit is possible but causes more accuracy degradation for most models.
- group_size: Number of weights sharing quantization parameters. 128 is the standard choice. It offers a good balance between accuracy and overhead. Smaller group sizes (64, 32) improve accuracy at the cost of slightly more storage for scale parameters.
- desc_act: Whether to use ActOrder (activation ordering), which processes columns by decreasing Hessian diagonal. Enabling this typically improves perplexity by 0.1 to 0.5 points but slightly increases quantization time.
- damp_percent: Dampening factor added to the Hessian diagonal as a fraction of its mean value. This prevents numerical instability when the Hessian is nearly singular. The default of 0.01 (1%) works well for most models.