QLoRA › Double Quantization

Block-wise quantization requires storing a scaling factor for each block. These scaling factors enable each block to adapt to its local value distribution, but they come with a storage cost that is easy to underestimate. Consider that for a 70B parameter model with 64-element blocks, we need approximately 1.09 billion scaling factors. Each factor is a 32-bit float, so the scales alone require 4.4 GB of memory. This is a significant fraction of the memory we're trying to save through quantization, and it means our effective bit-rate for storage is higher than 4 bits per weight. With 64-element blocks and FP32 scales, the overhead per parameter is:

$$\begin{aligned} O_s &= \frac{S_{\text{bits}}}{B_{\text{block}}} \\ &= \frac{32}{64} \\ &= 0.5 \text{ bits/element} \end{aligned}$$

where:

• $O_s$: the memory overhead per parameter due to storing scales
• $S_{\text{bits}}$: the size of a single scaling factor in bits (32 for FP32)
• $B_{\text{block}}$: the number of weights per quantization block (typically 64)

This formula captures the basic trade-off in block-wise quantization. Each 64-element block requires one 32-bit scaling factor, so the overhead per element is simply the scale size divided by the block size. For a 70B parameter model, this overhead alone requires 4.4 GB of memory, which is a significant fraction of the memory we're trying to save through quantization.

QLoRA's second innovation is double quantization: quantizing the quantization constants themselves. The key insight is that the scaling factors, while necessary for accurate reconstruction, are themselves just numbers that can be compressed. If the scaling factors have their own predictable distribution, we can quantize them too, at the cost of introducing a small additional layer of approximation. The question is whether the memory savings from compressing the scales justify the tiny accuracy loss from less precise scale reconstruction. The answer, it turns out, is yes by a comfortable margin.

The scale factors across blocks form their own distribution. Since each scale represents the range of values in a block of normally distributed weights, these scales tend to be positive numbers clustered around a characteristic value determined by the layer's weight statistics. We can apply a second round of quantization to these scales:

1. First quantization: Quantize weight blocks using NF4 with FP32 scales
2. Second quantization: Quantize the FP32 scales to 8-bit with a single global scale

The scales are typically positive values (since we normalize weights to have unit variance before quantization), so 8-bit unsigned integers work well. Using 256 levels for the scales provides enough precision to avoid meaningful accuracy loss while reducing storage by 4 times. The global scale needed to reconstruct the 8-bit scales is just a single FP32 number per tensor, contributing negligibly to total memory.

Think of double quantization as a Russian nesting doll of compression. The first doll represents the main weights, compressed from 16 bits to 4 bits using NF4. The second doll represents the scaling constants for the first compression, themselves compressed from 32 bits to 8 bits. The third doll is just a single number, the global constant needed to reconstruct the 8-bit scales. The outer dolls protect the precision of the inner dolls, and each layer of nesting adds only a tiny amount of storage overhead.

The minimal Mean Squared Error (MSE) confirms that 8-bit quantization preserves the scale information with high accuracy. This suggests we can safely compress the scales without degrading model performance. The error is so small because the scales vary smoothly across blocks, making them well-suited to quantization with many levels.

Let's calculate the total memory reduction. The overhead calculation changes because we now store 8-bit scales instead of 32-bit scales, with one additional 32-bit global scale per tensor: