GPTQ › Calibration Data
Language AI HandbookAll chapters GPTQ › Calibration DataOne section · Sections in this chapter8 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
GPTQ requires calibration data to estimate the Hessian. The quality and quantity of this data affect quantization results, yet practitioners often treat calibration data as an afterthought. Calibration is a significant design decision.
The calibration data does not need labels. GPTQ only performs forward passes through the model to collect layer activations, so any unlabeled text will do. What matters is that the calibration data covers the kinds of inputs the model will encounter at deployment. The Hessian is an empirical estimate of input statistics: run calibration on text that looks nothing like your deployment distribution, and the Hessian will mis-estimate which weights matter, leading to worse quantization choices.
Think of the calibration data as a "map of typical inputs." If you are quantizing a general-purpose language model, a diverse sample that combines web and book text with source code will give the Hessian a realistic picture of the model's inputs. If you are quantizing a specialized code-generation model, calibrating on web text means the Hessian will reflect a distribution very different from the model's actual use, and the quantization will be suboptimal for code generation tasks even if it looks fine on general language benchmarks.
The key parameters for calibration are:
-
Quantity. GPTQ typically uses 128 to 1,024 samples. More samples give a better Hessian estimate, reducing variance in the diagonal and off-diagonal elements. In practice, 128 samples of 2,048 tokens each (totaling 262,144 tokens) is sufficient for most models, with diminishing returns beyond 512 samples.
-
Representativeness. The calibration data should resemble the data the model will see at inference. Using random text to calibrate a code model yields worse results than using code samples, because the Hessian will reflect different activation patterns.
-
Sequence length. Longer sequences give more tokens per sample, leading to a more accurate Hessian estimate. Most implementations use 2,048 tokens per sequence.
Common choices include random samples from C4 (a large web text corpus), WikiText (cleaned Wikipedia text), or domain-specific data for specialized models. The C4 and WikiText defaults work well for general-purpose models. For specialized deployments, investing in domain-specific calibration data consistently improves results.