INT4 Quantization › Accuracy Trade-offs
Language AI HandbookAll chapters INT4 Quantization › Accuracy Trade-offsOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
How much does INT4 quantization affect model quality? The answer depends heavily on the model and task, including the specific quantization technique used. Understanding these dependencies is important for making informed decisions about when and how to deploy 4-bit quantized models. The good news is that for large models (30B parameters and above), the accuracy degradation from well-implemented INT4 quantization is often small enough to be practically invisible on most tasks. The less good news is that for smaller models and certain sensitive tasks, the degradation can be meaningful.
The basic reason that accuracy varies so much with model scale is information redundancy. In a 70B parameter model, each layer has many more parameters than strictly necessary to encode its function. The model has learned to be reliable to small perturbations because during training, stochastic gradient descent effectively explores a broad region of parameter space. Quantization is, in a sense, just another small perturbation. A 7B model, by contrast, operates closer to its minimum necessary capacity: each parameter carries more unique information, and corrupting that information through quantization causes proportionally larger degradation.
Let's explore the key factors that determine INT4 quantization success.
Model Size Matters
Larger models tolerate quantization better than smaller ones, a phenomenon that has been consistently observed across many model families and quantization methods. A 70B parameter model quantized to INT4 often performs comparably to its FP16 version, while a 7B model may show noticeable degradation. Larger models are more redundant and tolerate information loss from quantization better.
Larger models are more reliable because they have more parameters to encode information. When quantization introduces errors into some of these parameters, the remaining parameters can compensate, maintaining the overall behavior of the network. Smaller models lack this redundancy: each parameter carries more information, and corrupting that information through quantization has proportionally larger effects. This observation has practical implications: if you are choosing between a 7B INT4 model and a 13B INT8 model of similar memory footprint, the 13B INT8 model will often provide better quality despite using fewer bits per weight, because the larger model size compensates for the coarser quantization.
# Simulate quantization impact vs model size
# Based on empirical observations from the literature
model_sizes = [1, 3, 7, 13, 30, 70] # Billions of parameters
fp16_perplexity = [15.0, 10.0, 7.5, 6.5, 5.5, 5.0] # Baseline
# INT4 degradation diminishes with scale
int4_degradation_pct = [25, 15, 8, 5, 3, 2] # Percentage increase in perplexity
int4_perplexity = [
p * (1 + d / 100) for p, d in zip(fp16_perplexity, int4_degradation_pct)
]
Task Sensitivity
Different tasks have different sensitivity to quantization errors, and understanding this variation is important for deployment decisions. Tasks requiring precise numerical reasoning or factual recall tend to suffer more from quantization than tasks involving broader pattern recognition.
The sensitivity hierarchy roughly goes:
- More sensitive: Mathematical reasoning, code generation, factual question answering
- Less sensitive: Text summarization, sentiment analysis, general conversation
This sensitivity stems from how errors propagate through the model during different types of computation. For mathematical reasoning, a small error in intermediate calculations can compound into a wrong final answer. The model must maintain precise numerical relationships across many computation steps, and quantization errors can accumulate or interfere with these delicate calculations. For summarization, slightly imprecise representations still capture the overall meaning, and the model's task is more about recognizing patterns and relationships than performing exact computation.
There is also a subtler effect related to the context window. In tasks with long contexts, small per-token errors in the attention computation can accumulate across hundreds of attention operations. Quantization errors that are individually small can compound through repeated matrix multiplications, affecting the model's ability to faithfully propagate information from distant context. This is one reason why quantized models sometimes perform disproportionately worse on long-context tasks.
Layer-Specific Quantization
Not all layers in a transformer contribute equally to model quality, and this observation changes the interpretation for quantization strategy. Research has shown that certain layers are more sensitive to quantization errors than others:
- Attention projections (Q, K, V, and output projections): Often sensitive, especially in earlier layers, because they determine how information flows between positions in the sequence
- Feed-forward networks: Generally more reliable to quantization, perhaps because they perform more local computations that are less affected by small perturbations
- Embedding layers: Very sensitive, often kept at higher precision, because they form the foundation upon which all subsequent computation builds
This observation leads to mixed-precision strategies where sensitive layers use INT8 while reliable layers use INT4. Such strategies can achieve much of the memory savings of uniform INT4 quantization while retaining much of the accuracy of INT8 or even FP16 for necessary operations:
# Example of a layer sensitivity analysis
layer_types = [
"Embedding",
"Q Projection",
"K Projection",
"V Projection",
"Attention Out",
"FFN Up",
"FFN Down",
"Layer Norm",
"LM Head",
]
# Relative sensitivity (1 = baseline, higher = more sensitive)
sensitivity_scores = [2.5, 1.8, 1.6, 1.5, 1.4, 1.0, 1.0, 2.0, 2.2]
# Parameter count percentage (rough estimates)
param_percentages = [5, 8, 8, 8, 8, 25, 25, 0.1, 5]
# Generate recommendations
recommendations = []
for sens in sensitivity_scores:
if sens >= 2.0:
recommendations.append("FP16 or INT8")
elif sens >= 1.4:
recommendations.append("INT8")
else:
recommendations.append("INT4")Layer Quantization Sensitivity Analysis: ----------------------------------------------------------------- Layer Type Sensitivity % Params Recommended ----------------------------------------------------------------- Embedding 2.5 5.0% FP16 or INT8 Q Projection 1.8 8.0% INT8 K Projection 1.6 8.0% INT8 V Projection 1.5 8.0% INT8 Attention Out 1.4 8.0% INT8 FFN Up 1.0 25.0% INT4 FFN Down 1.0 25.0% INT4 Layer Norm 2.0 0.1% FP16 or INT8 LM Head 2.2 5.0% FP16 or INT8
The results suggest a hybrid approach: retain high precision for embeddings and attention outputs, but aggressively quantize the feed-forward networks that contain the majority of parameters. Since FFN layers often account for roughly half of a transformer's parameters, quantizing them to INT4 while keeping other layers at INT8 can still provide substantial memory savings with minimal accuracy loss. In practice, libraries like bitsandbytes allow you to specify which layers to quantize and at what precision, giving you fine-grained control over this trade-off.