Weight Quantization Basics › Calibration

We have assumed we know the minimum and maximum values of each tensor, but finding these optimally requires care. Simply taking the observed min/max may not be ideal, especially for activations that vary with input data. Calibration is the process of determining these parameters systematically, and the quality of calibration directly impacts quantized model accuracy. Poor calibration can turn a theoretically excellent quantization scheme into a practical failure.

Think of calibration as tuning an instrument before a performance. You could tune once using the manufacturer's default settings, but a skilled musician tunes to the specific acoustic conditions of the hall and the other instruments they will play with. Similarly, a quantization scheme calibrated on representative data for the target deployment scenario will outperform one that uses generic or theoretical parameter choices.

Weights are fixed after training, so their distributions are known exactly. For per-tensor quantization, we can directly compute global min/max. For per-channel or per-group quantization, we compute statistics for each partition. Since weights do not change during inference, calibration happens once and the parameters are stored with the model, adding negligible size overhead.

However, raw min/max may be skewed by rare extreme values. Some methods clip the range to exclude extreme outliers, accepting large errors on those values in exchange for better precision on typical values. The reasoning is that if only 0.1% of values are extreme, accepting large errors on those values while improving precision for the other 99.9% reduces overall error. Common approaches include:

• Percentile clipping: Use the $p$-th percentile instead of the true maximum, sacrificing outlier accuracy for finer granularity on typical values. Common choices are the 99th or 99.9th percentile.
• MSE minimization: Search for the clipping threshold that minimizes mean squared quantization error over the entire tensor. This is optimal in the mean-squared-error sense but requires a search procedure.
• Entropy-based: Choose thresholds that minimize KL divergence between the original and quantized value distributions. This preserves the statistical character of the tensor rather than minimizing a single error metric.

The right choice depends on your application. For tasks sensitive to rare values (factual recall, where the model needs to retrieve specific numbers), clipping extreme values aggressively may hurt. For tasks dominated by typical values (general text generation), aggressive clipping often helps by improving precision for the common case.

Activations present a harder problem because their distributions depend on inputs. A model might produce activations ranging from $-10$ to $10$ on one input and $-5$ to $50$ on another. We need to find quantization parameters that work well across the expected input distribution, not just for any single input. Static activation quantization requires making a decision once that must generalize to all future inputs.

The standard approach runs the model on a calibration dataset: a small representative sample of inputs from the target domain. We collect activation statistics (min, max, percentiles, or full histograms) across this dataset, then compute quantization parameters from the aggregated statistics. The calibration dataset is a proxy for the true inference distribution, and its quality directly determines calibration quality.

Common aggregation strategies include:

• Running min/max: Track the most extreme values seen across all calibration samples. This ensures no calibration input will be clipped, but may produce overly large scales that waste precision on typical inputs.
• Moving average: Smooth the statistics across samples to reduce sensitivity to outliers in the calibration set itself. A commonly used formula is $\hat{x}_\text{max} = \alpha \cdot x_\text{max,new} + (1 - \alpha) \cdot \hat{x}_\text{max,old}$, with $\alpha \approx 0.01$.
• Histogram-based: Build full histograms and choose optimal thresholds by minimizing reconstruction error. This is the most accurate but also the most expensive approach.

The calibration dataset should resemble actual inference inputs. It should cover similar content and sequence lengths while matching the expected writing style. Using the wrong distribution during calibration can lead to systematic clipping or range waste during deployment. A chatbot calibrated on encyclopedia text may have poorly tuned activations for conversational inputs. A few hundred to a few thousand samples typically suffice for stable statistics, though the exact number depends on how variable the activations are and how sensitive the model is to quantization.

Calibration determines whether quantization is static or dynamic, and this distinction has major practical implications for inference systems.

Static quantization fixes all quantization parameters at calibration time. Inference uses predetermined scales and zero-points, with no runtime overhead for parameter computation. This is efficient and predictable but may lose accuracy if inference distributions differ from calibration. Static quantization is the default for production deployment: calibrate once, deploy everywhere.

Dynamic quantization computes activation quantization parameters on-the-fly for each input. This adapts to the actual values in each tensor. This provides optimal parameters regardless of the input distribution. The cost is computing statistics (at minimum, a min/max scan) over each activation tensor during inference, adding latency and overhead. Dynamic quantization is commonly used when activation distributions are highly variable, when calibration data is unavailable, or when maximum accuracy is required with no deployment pipeline for offline calibration.

Weight quantization is almost always static since weights do not change. The question is whether to quantize weights permanently (storing integers) or to quantize them on-the-fly from stored floating-point weights (saving memory but doing the conversion at runtime). Permanent weight quantization is more common for deployed models. Activation quantization may be either static or dynamic, depending on the accuracy-efficiency tradeoff desired.

A third approach, quantization-aware training (QAT), trains the model while simulating quantization effects, letting the model to adapt its weights to minimize quantized error. QAT typically outperforms post-training quantization but requires access to training data and significant compute. We will cover QAT in detail in a later chapter.