INT8 Quantization › Limitations and Practical Considerations

INT8 quantization is a practical and widely-deployed optimization, but it comes with important limitations that you should understand before deploying in production. The limitations are not reasons to avoid quantization, but they are constraints that must be accounted for in system design.

The most basic limitation is that some model information is permanently lost during quantization. While smooth quantization and careful calibration minimize this loss, they cannot eliminate it entirely. For safety-necessary applications or tasks requiring the highest possible accuracy, the small degradation from INT8 may be unacceptable. This is why many production systems maintain FP16 or FP32 models for evaluation and fine-tuning while deploying INT8 for inference. The irreversibility is by design: if we could recover the original values losslessly from INT8, we would not have achieved any compression. The compression and the information loss are two sides of the same coin.

Calibration data sensitivity presents another practical challenge. Smooth quantization computes channel-wise statistics from a calibration dataset, and if this data does not stand for the actual deployment distribution, the smoothing factors may be suboptimal. A model calibrated on English text may perform poorly when quantized for code completion or multilingual inputs. In production, this means you need calibration pipelines that are maintained alongside the model and updated when the deployment distribution shifts. It also means that models with broad deployment contexts (such as general-purpose chatbots serving diverse user queries) may need larger and more diverse calibration sets to reach reliable quantization quality.

Hardware support, while widespread, is not universal. INT8 acceleration requires specific hardware features such as NVIDIA's Tensor Cores or ARM's NEON instructions, and the speedup varies by platform and use case. On CPUs without INT8 vectorization, quantized inference may be slower than FP32 due to the quantization and dequantization overhead. On GPUs, the speedup is most pronounced for large matrix multiplications (typical of large-batch inference) but may be minimal for small-batch or latency-sensitive inference where compute is not the bottleneck. Always benchmark on your target hardware before committing to a quantization strategy.

Attention layers present a special challenge that pure weight quantization does not solve. The key-value cache that accumulates during autoregressive generation stores activation values, not weights. As sequences grow longer, the KV cache grows proportionally and can become the dominant memory consumer. Quantizing the KV cache requires activation quantization, which is harder than weight quantization because activation distributions vary with the input. Dedicated techniques for KV cache quantization are an active area of development and are addressed in later chapters.

Finally, INT8 may not give sufficient compression for edge deployment scenarios. With 8 bits per weight, a 7B parameter model still requires 7 GB of storage, which exceeds the capabilities of most mobile devices and many edge accelerators. For mobile or embedded applications, the more aggressive INT4 quantization techniques we will explore in the next chapter become necessary, accepting greater accuracy trade-offs in exchange for smaller model sizes that fit in constrained hardware.