Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

INT4 Quantization › Double Quantization

Michael BrenndoerferJanuary 12, 20266 min read

Language AI HandbookAll chapters

Part 42 of 67Inference Optimization

Chapter 319INT4 Quantization › Double Quantization

Reading section 5 of 11 · Double Quantization · 6 minSection 5 of 11 · 6 min

Section 5 of 11 · 6 min read

INT4 Quantization › Double QuantizationOne section · Sections in this chapter5 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A technique called double quantization, also introduced with QLoRA, further reduces memory overhead by quantizing the scale factors themselves. This recursive application of quantization addresses a subtle but important issue with group-wise quantization. In standard group-wise quantization, we store one FP16 scale factor per group. With groups of 128, this adds 0.125 bits per weight (16 bits divided by 128 weights per group). While this overhead may seem modest, it becomes significant when the goal is to minimize memory footprint as aggressively as possible.

Think of double quantization as the same trick applied recursively. We have weights, so we quantize them. We have scale factors that take up memory, so we quantize those too. The scale factors are themselves a set of floating-point numbers that can be represented more compactly, and their statistical properties (they tend to be positive, relatively smooth, and bounded) make them good candidates for further compression.

The key question is: does quantizing the scale factors introduce significant error? The answer is generally no, for a subtle reason. Scale factors interact with weights multiplicatively: the reconstructed weight is $\hat{w}_i = q_i \cdot s_g$ . If the scale factor has a small relative error $\delta$ , so $\hat{s}_g = s_g (1 + \delta)$ , then the reconstructed weight is $\hat{w}_i = q_i \cdot s_g (1 + \delta)$ . This is equivalent to a uniform scaling of all weights in the group by factor $(1 + \delta)$ . For small $\delta$ (say, 1-2%), this scaling is nearly invisible to the model, because the model's behavior depends on weight ratios more than absolute magnitudes for many operations. Scale factors can therefore be quantized quite aggressively without much impact on final model quality.

Double quantization applies a second round of quantization to these scale factors, treating them as a new quantization problem unto themselves:

Collect all scale factors from the first quantization into a vector
Group these scale factors together, typically in groups of 256
Quantize the scale factors to FP8 or INT8 precision, which is coarser than FP16 but still accurate enough for scale factors
Store a single FP32 scale factor per group of scale factors to enable reconstruction

The key insight letting double quantization is that scale factors themselves exhibit predictable statistical properties. Within a neural network, scale factors for different groups tend to fall within a relatively narrow range, making them amenable to quantization without significant loss of information. The second-level scale factors (the "meta-scales") require only FP32 precision and are few in number, adding negligible overhead.

In[31]:

Code

def double_quantize(weights, group_size=64, scale_group_size=256):
    """
    Double quantization: quantize weights, then quantize the scales.
    """
    # First quantization: weights to INT4
    flat_weights = weights.flatten()
    n = len(flat_weights)

    # Pad for grouping
    pad_len = (group_size - n % group_size) % group_size
    if pad_len > 0:
        flat_weights = np.concatenate([flat_weights, np.zeros(pad_len)])

    n_groups = len(flat_weights) // group_size
    groups = flat_weights.reshape(n_groups, group_size)

    # Compute scales (first level)
    scales_fp32 = np.max(np.abs(groups), axis=1) / 7.0
    scales_fp32 = np.where(scales_fp32 == 0, 1.0, scales_fp32)

    # Quantize weights using scales
    q_weights = np.round(groups / scales_fp32[:, np.newaxis]).astype(np.int8)
    q_weights = np.clip(q_weights, -8, 7)

    # Second quantization: scales to INT8
    # Pad scales for grouping
    n_scales = len(scales_fp32)
    scale_pad = (
        scale_group_size - n_scales % scale_group_size
    ) % scale_group_size
    if scale_pad > 0:
        scales_padded = np.concatenate([scales_fp32, np.zeros(scale_pad)])
    else:
        scales_padded = scales_fp32

    n_scale_groups = len(scales_padded) // scale_group_size
    scale_groups = scales_padded.reshape(n_scale_groups, scale_group_size)

    # Compute meta-scales (second level)
    meta_scales = np.max(np.abs(scale_groups), axis=1) / 127.0
    meta_scales = np.where(meta_scales == 0, 1.0, meta_scales)

    # Quantize scales to INT8
    q_scales = np.round(scale_groups / meta_scales[:, np.newaxis]).astype(
        np.int8
    )
    q_scales = np.clip(q_scales, -128, 127)

    return {
        "q_weights": q_weights.flatten()[:n],
        "q_scales": q_scales.flatten()[:n_scales],
        "meta_scales": meta_scales,
        "group_size": group_size,
        "scale_group_size": scale_group_size,
        "original_len": n,
    }

In[32]:

Code

# Calculate memory savings
memory_rng = np.random.default_rng(42)
large_weights = memory_rng.standard_normal(1_000_000) * 0.02

result = double_quantize(large_weights, group_size=16, scale_group_size=256)

n_weights = len(large_weights)
n_scales = len(result["q_scales"])
n_meta_scales = len(result["meta_scales"])

# Calculate memory usage in bits
original_bits = n_weights * 32  # FP32
single_quant_bits = n_weights * 4 + n_scales * 16  # INT4 weights + FP16 scales
double_quant_bits = n_weights * 4 + n_scales * 8 + n_meta_scales * 32

Out[33]:

Console

Memory Analysis for 1M Weights:
--------------------------------------------------
Original FP32:           4.00 MB
Single quantization:     0.62 MB
Double quantization:     0.56 MB
--------------------------------------------------
Effective bits/weight (single): 5.000
Effective bits/weight (double): 4.508

Out[34]:

Visualization

Bar chart comparing FP32, single quantization, and double quantization total memory in MB, showing progressively smaller memory usage. — Total memory usage for 1 million weights across three precision levels. Double quantization (green) achieves the smallest footprint by quantizing both weights and their scale factors, approaching but not quite reaching the theoretical 0.5 MB floor for pure INT4.

Out[35]:

Visualization

Stacked bar chart showing effective bits per weight for single and double quantization, with INT4 weight bits as base and decreasing overhead from scale factor storage. — Effective bits per weight breakdown for single and double quantization. The base INT4 weight bits (blue) are identical in both cases. Double quantization (right) replaces the larger FP16 scale overhead (orange) with a smaller INT8 scale overhead (green) plus negligible FP32 meta-scale bits (yellow).

Double quantization reduces the overhead from scale factors, getting closer to true 4 bits per weight while maintaining the benefits of group-wise quantization. The additional complexity in the dequantization path (needing to reconstruct scales before reconstructing weights) is modest and well worth the memory savings for memory-constrained deployments. In practice, the QLoRA paper reports that double quantization reduces memory usage from approximately 4.5 bits per weight to approximately 4.13 bits per weight on average, a savings of about 0.37 bits that adds up to over 3 GB for a 70B parameter model.