INT4 Quantization › The Challenge of 4-Bit Precision
Language AI HandbookAll chapters INT4 Quantization › The Challenge of 4-Bit PrecisionOne section · Sections in this chapter2 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Understanding why 4-bit quantization is hard requires understanding what those 16 levels mean in practice. With only 4 bits, we can represent just 16 distinct values, and every weight in the neural network must map to exactly one of these 16 values. To understand why this limitation is so severe, consider the basic mathematics at play. If we use signed integers, this gives us a range from -8 to 7 (or -7 to 7 with a symmetric scheme). Every weight in a neural network, regardless of its original floating-point precision, must map to one of these 16 values. This is an extremely coarse discretization of the original continuous space, and the consequences propagate through every matrix multiplication in the model.
Think of it this way: INT4 quantization is like trying to represent every color in a photograph using only 16 distinct shades. A grayscale photograph with 256 shades (INT8) already loses some detail, but you can still recognize faces and read text. With only 16 shades, fine details merge together and subtle distinctions disappear. For neural network weights, those subtle distinctions between similar-but-not-identical small weights are precisely what encode the model's learned knowledge, and losing them changes model behavior in unpredictable ways.
Consider what this means for a typical weight distribution. Neural network weights after training often follow an approximately normal distribution centered near zero. The majority of weights cluster around small magnitudes, with the distribution tapering off symmetrically toward the tails. With INT8, we have enough resolution to capture the shape of this distribution reasonably well, since 256 quantization levels can provide fine granularity even in the densely populated region near zero. With INT4, we are essentially creating a 16-bucket histogram to represent a continuous distribution. Each bucket must absorb a wide swath of the original weight values, collapsing potentially meaningful differences into a single quantized output.
When we quantize a value, we introduce a quantization error equal to the difference between the original value and its quantized representation. With 256 levels, the maximum error for any single weight is bounded by half the step size between adjacent levels. With only 16 levels, that step size becomes 16 times larger, and the maximum quantization error grows proportionally. This error propagates through every matrix multiplication in the network, accumulating and potentially compounding in ways that can fundamentally alter the model's behavior. The errors are not random noise that averages out: they are systematic biases that can shift the model's predictions in consistent but unexpected directions.
The key insight is that the severity of quantization error depends on how many levels are available and how well those levels match the data distribution. Most of the problems with naive 4-bit quantization stem from a mismatch: the quantization grid is designed for uniform data, but neural network weights are emphatically non-uniform. This mismatch motivates both the group-wise approach (which adapts the scale locally) and the NF4 format (which adapts the level placement globally).
import numpy as np
# Simulate a typical neural network weight distribution deterministically
weight_rng = np.random.default_rng(42)
weights = weight_rng.standard_normal(10000) * 0.02 # Typical small weight scale
# What INT8 and INT4 quantization "see"
int8_bins = 256
int4_bins = 16
The loss of precision becomes more severe when we consider outliers. Recall from our INT8 discussion that neural networks often have a small number of outlier weights with magnitudes much larger than the majority. These outliers, though rare, can be critically important for the model's computations, often encoding strong feature responses or needed biases. With INT8, we can absorb some outlier impact because we have 256 levels to work with. This provides reasonable resolution even after accommodating extreme values. With INT4, outliers become catastrophic because the limited number of levels cannot simultaneously span a wide range and maintain adequate precision in the dense central region.
The Outlier Problem Intensified
To build intuition for why outliers cause such severe problems at 4-bit precision, imagine a layer where 99% of weights fall between -0.1 and 0.1, but a few outliers reach values of 0.5 or higher. If we set our quantization range to cover the outliers, most of our 16 quantization levels will be "wasted" on the outlier range, leaving very few levels to distinguish the majority of weights near zero. This is a basic trade-off that becomes increasingly painful as the number of available levels decreases.
Think of this problem as a zoom lens on a camera. If your frame must fit both a distant mountain and a nearby flower, the flower becomes just a few indistinct pixels. The "zoom level" of your quantization scale is forced to accommodate the most extreme value, compressing everything else into a tiny slice of your available range. With 256 levels (INT8), you still have reasonable detail on the flower even when the mountain is in frame. With 16 levels (INT4), the flower becomes essentially invisible.
Consider the mathematics of this trade-off. Suppose our 16 quantization levels must span from -0.5 to 0.5 to accommodate outliers. The step size between adjacent levels is then 1.0 divided by 15 (since we have 16 levels creating 15 intervals), giving approximately 0.067 per step. For the 99% of weights living in the range of -0.1 to 0.1, this means they have access to only about 3 distinct quantization levels. The subtle differences between small weights, which are important for model computations, are lost in the coarseness imposed by the outliers.
More formally, for symmetric quantization, the scale factor is determined by the maximum absolute value in the tensor:
where:
- : the vector of weights being quantized
- : the maximum absolute value across all weights in the group
- : the maximum representable integer (7 for signed INT4 with range -7 to 7)
A single outlier weight with magnitude 0.6 among thousands of weights with magnitude 0.02 forces . For a weight of 0.02, the quantized index is . Nearly every small weight gets mapped to integer 0, making them indistinguishable from each other and from true zeros. The entire information content of the small-weight region collapses to a single point.
# Demonstrate outlier impact on INT4 quantization
outlier_rng = np.random.default_rng(42)
# Normal weights with a few outliers
normal_weights = outlier_rng.standard_normal(1000) * 0.02
outliers = np.array([0.5, -0.4, 0.6, -0.5]) # Just 4 outliers
weights_with_outliers = np.concatenate([normal_weights, outliers])
# Uniform quantization across full range
w_min, w_max = weights_with_outliers.min(), weights_with_outliers.max()
scale = (w_max - w_min) / 15 # 16 levels (0-15) for unsigned INT4
def quantize_uniform(weights, scale, w_min):
"""Uniform quantization to 4-bit range (0-15)"""
q = np.round((weights - w_min) / scale).astype(int)
q = np.clip(q, 0, 15)
return q
def dequantize_uniform(q, scale, w_min):
"""Dequantize back to float"""
return q * scale + w_min
q_weights = quantize_uniform(weights_with_outliers, scale, w_min)
reconstructed = dequantize_uniform(q_weights, scale, w_min)
# Calculate error metrics
mse = np.mean((weights_with_outliers - reconstructed) ** 2)
max_error = np.max(np.abs(weights_with_outliers - reconstructed))Weight range: [-0.5000, 0.6000] Scale factor: 0.073333 Mean squared error: 0.00039370 Maximum error: 0.036582 Number of unique quantized values used: 7
With a scale of approximately 0.07, each quantization step is large relative to our typical weights near zero. Most small weights get mapped to the same few quantization levels, destroying the subtle differences between them. The quantization process discards the details that distinguish small weights, even though these differences affect inference. Notice also how few unique quantized values appear: with 1004 weights and 16 possible levels, we would expect all 16 to be used, but outliers force most values into just a handful of central bins.
