QLoRA › Quantization Fundamentals

Quantization maps high-precision values to a smaller set of discrete levels. For neural network weights, the goal is to reduce storage and computation costs while minimizing the accuracy loss from this approximation. To understand why this works at all, consider that neural networks tolerate some noise. Small perturbations to weights typically produce small changes in outputs, which means we have room to introduce controlled approximation errors. This tolerance follows from averaging across millions of parameters, where individual inaccuracies tend to cancel rather than compound.

The core question in quantization design is how to allocate limited precision most effectively. Going from 16 bits to 4 bits means representing values with only 16 distinct levels instead of 65,536. Choosing those 16 levels poorly can degrade model quality significantly. Choosing them wisely can preserve model capability almost completely. The difference between a good quantization scheme and a bad one isn't in the number of bits used, it's in how intelligently the available levels are positioned relative to the actual distribution of values being quantized.

The simplest approach is uniform quantization, where the value range is divided into equally-spaced bins. This approach treats all parts of the value range as equally important, allocating the same precision to every region. To quantize a tensor $W$ to $k$ bits (yielding $2^k$ levels), we first normalize the weights to the $[0, 1]$ range and then scale them to the integer range $[0, 2^k - 1]$. The normalization step ensures that all weights fall within a predictable range, while the scaling step maps these normalized values to discrete integer codes that can be stored efficiently. The calculation is:

$$W_{\text{quant}} = \text{round}\left(\frac{W - W_{\min}}{W_{\max} - W_{\min}} \times (2^k - 1)\right)$$

where:

• $W_{\text{quant}}$: the resulting quantized integer tensor
• $W$: the input high-precision weight tensor
• $W_{\min}$: the minimum value in the input tensor $W$
• $W_{\max}$: the maximum value in the input tensor $W$
• $k$: the bit-width used for quantization
• $2^k - 1$: the maximum integer value representable with $k$ bits

The term $(W - W_{\min}) / (W_{\max} - W_{\min})$ shifts and scales the weights so that the minimum value maps to 0 and the maximum maps to 1. Multiplying by $(2^k - 1)$ then stretches this unit interval to span the full range of representable integers. The rounding operation introduces quantization error by snapping each continuous value to its nearest discrete level.

Dequantization reverses this process, reconstructing an approximation of the original weights from the quantized codes:

$$W_{\text{dequant}} = \frac{W_{\text{quant}}}{2^k - 1} \times (W_{\max} - W_{\min}) + W_{\min}$$

where:

• $W_{\text{dequant}}$: the reconstructed high-precision weight tensor
• $W_{\text{quant}}$: the quantized integer tensor
• $W_{\min}, W_{\max}$: the quantization constants (min and max values) stored alongside the weights
• $k$: the bit-width used for quantization

Notice that we must store $W_{\min}$ and $W_{\max}$ alongside the quantized weights. Without these constants, we cannot recover the original scale and offset. This is a common pattern in quantization: we trade most of the precision for storage savings while keeping a small amount of metadata to enable reconstruction.

Uniform quantization works reasonably well for uniformly distributed data, but neural network weights are decidedly not uniform. As we'll see, this mismatch creates unnecessary quantization error.

Computing a single scale for an entire weight matrix is problematic because outlier values stretch the quantization range, wasting precision on unused regions. Imagine a weight matrix where most values lie between -0.5 and 0.5, but a few outliers reach plus or minus 3.0. Uniform quantization must spread its limited precision across the entire [-3.0, 3.0] range, even though most of this range is sparsely populated. The densely populated central region ends up with coarse granularity, leading to large quantization errors for the majority of weights.

Think of this as trying to measure the heights of a classroom of students using a ruler designed for basketball players. If the ruler goes from 0 to 8 feet but all your students are between 4 and 6 feet, you're wasting half the precision of your ruler on heights that don't exist in your data. You'd get much better measurements if you used a ruler calibrated specifically for the range you care about. Block-wise quantization takes this approach: it gives each small group of weights its own ruler, calibrated to that group's specific range.

Block-wise quantization addresses this by dividing weights into blocks of size $B$ (typically 64 or 128 elements) and computing separate quantization constants for each block. Within each block, the local minimum and maximum determine the quantization range. This allows each block to adapt to its own value distribution, placing precision where it is needed. A block of weights that happens to have a narrow range gets fine-grained precision within that range, even if adjacent blocks have wider ranges.

This improves accuracy but introduces overhead: we must store $N/B$ scaling factors in addition to the quantized weights. For 64-element blocks with FP32 scales, that's an additional 0.5 bits per parameter. This overhead represents the cost of adaptive precision allocation. The accuracy improvement typically outweighs this cost, but as we'll see, QLoRA introduces techniques to minimize even this overhead through a second round of compression.