Weight Quantization Basics › Limitations and Practical Considerations

Quantization's appeal lies in its simplicity, but several challenges arise in practice that require careful handling. Understanding these limitations is needed for successfully deploying quantized models and for understanding why the more sophisticated methods covered in subsequent chapters were developed.

Outlier sensitivity remains the primary obstacle to aggressive quantization. Transformer models frequently develop weight and activation outliers during training, particularly in attention layers and certain projection matrices. Research on large transformers has found that some activation dimensions can take values an order of magnitude larger than the mean, and these outliers become more pronounced as model scale increases. A single extreme value in a tensor forces a large scale, wasting precision for all other values. Per-channel and per-group quantization mitigate this for weights, but handling activation outliers often requires more sophisticated techniques. Methods like LLM.int8() detect outliers at runtime and process them in higher precision, while AWQ and GPTQ use careful calibration to minimize outlier impact. The challenge is that these outliers are not random: they tend to concentrate in specific channels and follow patterns that can be exploited, but exploiting them requires analysis beyond simple min/max calibration.

Accuracy degradation varies dramatically across models and tasks. Simple tasks like classification often tolerate aggressive quantization with minimal accuracy loss, because the model needs only to distinguish between a small number of output classes and can absorb substantial weight perturbations. Generation tasks, especially those requiring precise reasoning or factual recall, prove more sensitive. A model that achieves acceptable perplexity after 4-bit quantization may show subtle but significant degradation in multi-step reasoning, mathematical calculations, or factual question answering. The difficulty is that standard benchmarks may not reveal this degradation. A model might score well on perplexity while generating subtly incorrect or less coherent text. Always evaluate on task-specific metrics, not just generic benchmarks, and use human evaluation for generation quality when possible.

Calibration dataset selection significantly impacts results, and this dependence is often underestimated. Using the wrong distribution during calibration can cause systematic clipping or range waste during deployment. For general-purpose models, calibration data should span the expected content and input lengths while reflecting the likely writing styles. For specialized applications, calibration on domain-specific data often yields better results than generic calibration. A model deployed for medical record analysis should be calibrated on medical text, not Wikipedia. The mismatch between calibration distribution and deployment distribution is a common source of unexpected accuracy loss that is difficult to diagnose after deployment.

Hardware support determines practical speedup, and this is where theoretical analysis often diverges from empirical results. Integer operations are only faster if the hardware has efficient integer compute units and the software stack (CUDA kernels, runtime libraries) is optimized to use them. Modern GPUs offer accelerated INT8 and sometimes INT4 operations through tensor cores, but not all hardware supports all quantization formats, and not all quantization schemes are supported by the available kernels. Some quantization schemes that theoretically reduce compute find no actual speedup due to missing hardware support or kernel availability. Before committing to a quantization approach for production deployment, verify actual inference latency on the target hardware. The theoretical memory reduction is guaranteed, but the compute speedup depends on the specific hardware and software stack.

Despite these challenges, quantization has enabled an explosion in LLM accessibility. Models that required enterprise-grade hardware now run on laptops and smartphones. The memory savings cascade into reduced serving costs, making AI applications economically viable at scales previously impossible. The next chapters will explore specific quantization methods, INT8 and INT4 techniques, and formats like GPTQ and AWQ that push compression further while maintaining accuracy. The foundational concepts in this chapter, scale, zero-point, granularity, and calibration, underpin all of those methods.