Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

QLoRA › The NormalFloat Data Type

Michael BrenndoerferDecember 4, 202512 min read

Language AI HandbookAll chapters

Part 35 of 67Parameter-Efficient Fine-tuning

Chapter 259QLoRA › The NormalFloat Data Type

Reading section 4 of 15 · The NormalFloat Data Type · 12 minSection 4 of 15 · 12 min

Section 4 of 15 · 12 min read

QLoRA › The NormalFloat Data TypeOne section · Sections in this chapter4 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

QLoRA's first and most important innovation is recognizing that neural network weights follow approximately normal distributions. This insight enables a quantization scheme that places more precision where the density is highest. Rather than treating all value ranges as equally important, NormalFloat allocates quantization levels according to where weights concentrate. The result is a data type that is provably optimal for normally distributed data, in the sense that it minimizes the expected quantization error.

The key insight is that we can design a data type around the statistical properties of the data it will store. This is unusual in computer architecture: standard data types like INT8 or FP16 are designed to be general-purpose, making no assumptions about the distribution of stored values. But when we know that neural network weights follow a specific distribution, we can exploit that knowledge to pack more information into fewer bits. This is the same principle that underlies compression algorithms like Huffman coding, which assigns shorter codes to more frequent symbols. NormalFloat assigns more precise levels to more probable weight values.

Think of NF4 as a custom ruler with non-uniform tick marks. A standard ruler has evenly spaced ticks, which is appropriate when you're measuring things that could be any size. But if you knew you'd only ever measure heights between 5 feet and 6 feet, you'd get far better precision by cramming all your tick marks into that one-foot range and ignoring everything outside it. NF4 takes this principle and applies it to the shape of the normal distribution: most tick marks cluster near zero where most weights live, and fewer tick marks cover the sparse tails.

Why Normal Distributions MatterLink Copied

When we initialize neural networks using methods like Xavier or Kaiming initialization, weights are drawn from normal distributions. These initialization schemes are designed to maintain stable gradient flow through deep networks, and they achieve this by carefully controlling the variance of each layer's weights. Training updates through gradient descent tend to maintain this approximate normality, because gradient updates are themselves sums of many small contributions, and sums of random variables tend toward normality by the central limit theorem. Let's verify this empirically:

In[4]:

Code

import numpy as np
from scipy import stats

# Simulate trained neural network weights
np.random.seed(42)

# Weights often follow normal distributions after training
# with slightly heavier tails
simulated_weights = np.random.normal(0, 1, 100000)

# Test for normality
statistic, p_value = stats.normaltest(simulated_weights)
mean_val = simulated_weights.mean()
std_val = simulated_weights.std()

Out[5]:

Console

Normality test p-value: 0.8560
Mean: 0.0010
Std: 1.0009

The high p-value confirms that the weights follow a normal distribution, while the mean and standard deviation align with the initialization parameters. This normality assumption is the foundation for the NormalFloat data type. If we know in advance what distribution our data follows, we can design a quantization scheme specifically optimized for that distribution.

Out[6]:

Visualization

Histogram of weight values overlaid with normal distribution curve showing close alignment. — Distribution of simulated neural network weights following standard normal initialization. The histogram closely matches the theoretical normal distribution (neutral curve), validating the assumption that underlies NormalFloat quantization.

Quantile QuantizationLink Copied

For normally distributed data, uniform quantization is inefficient. Many quantization levels fall in the distribution's tails where few values exist, while the dense center region has insufficient precision. Consider what happens with 16 uniformly spaced levels across the range [-3, 3]: each level covers an interval of width 0.375. But for a standard normal distribution, approximately 68% of values fall within one standard deviation of the mean, packed into just 16% of the total range. Uniform quantization wastes most of its levels on the sparse tails.

NormalFloat uses quantile quantization, a fundamentally different approach. The $2^k$ quantization levels are placed at equal quantiles of the standard normal distribution. For 4-bit quantization with 16 levels, each level corresponds to a region containing exactly $\frac{1}{16}$ of the probability mass. This ensures that each quantization bin is equally likely to be used, maximizing the information content of each bit.

The quantile function (inverse CDF) of the standard normal distribution gives us these boundaries. The quantile function $\Phi^{-1}(p)$ returns the value $x$ such that a standard normal random variable has probability $p$ of being less than $x$ :

q_i = \Phi^{-1}\left(\frac{i}{2^k}\right) \quad \text{for } i \in \{0, 1, ..., 2^k\}

where:

$q_i$ : the value of the $i$ -th quantization boundary
$\Phi^{-1}$ : the inverse cumulative distribution function (quantile function) of the standard normal distribution
$i$ : the index of the boundary
$2^k$ : the total number of quantization bins (determined by the bit-width $k$ )

For example, with 16 bins, we compute boundaries at probabilities 0, 1/16, 2/16, ..., 15/16, 1. The boundary between bins 7 and 8 falls at the median (probability 0.5), which for a standard normal is exactly 0. The boundaries spread out in the tails and cluster tightly near zero, precisely matching where the probability density is highest.

In[7]:

Code

from scipy.stats import norm


def create_nf4_levels():
    """Create the 16 quantization levels for NF4."""
    n_levels = 16

    # Compute quantile boundaries (17 boundaries for 16 bins)
    quantiles = np.linspace(0, 1, n_levels + 1)
    boundaries = norm.ppf(quantiles)

    # Quantization levels are midpoints of each bin
    # Handle infinite boundaries at edges
    boundaries[0] = -3.5  # Practical lower bound
    boundaries[-1] = 3.5  # Practical upper bound

    # Representative value for each bin is the conditional mean
    # For a truncated normal, this is the expected value within each quantile bin
    levels = []
    for i in range(n_levels):
        low, high = boundaries[i], boundaries[i + 1]
        # Expected value of standard normal truncated to [low, high]
        expected = (norm.pdf(low) - norm.pdf(high)) / (
            norm.cdf(high) - norm.cdf(low)
        )
        levels.append(expected)

    return np.array(levels)


nf4_levels = create_nf4_levels()

Out[8]:

Console

NF4 quantization levels:
  Level  0: -1.9611
  Level  1: -1.3259
  Level  2: -1.0129
  Level  3: -0.7779
  Level  4: -0.5800
  Level  5: -0.4027
  Level  6: -0.2375
  Level  7: -0.0785
  Level  8: +0.0785
  Level  9: +0.2375
  Level 10: +0.4027
  Level 11: +0.5800
  Level 12: +0.7779
  Level 13: +1.0129
  Level 14: +1.3259
  Level 15: +1.9611

Notice how the levels are more densely packed near zero where most weight values concentrate, and more sparse in the tails. The spacing between levels 7 and 8 (which straddle zero) is much smaller than the spacing between levels 0 and 1 (in the far left tail). This is the key advantage of NF4 over uniform INT4: precision is allocated according to need rather than uniformly.

Out[9]:

Visualization

Stem plot showing uniform INT4 quantization levels evenly spaced from -4 to 4. — Uniform INT4 quantization level spacing: all 16 levels are evenly distributed across the value range, allocating equal precision to sparse tails and dense center alike.

Stem plot showing NF4 quantization levels densely packed near zero and sparse in the tails. — NormalFloat NF4 quantization level spacing: levels cluster densely near zero where most weights occur and spread out in the tails, matching the normal distribution''s probability density.

Visualizing the DifferenceLink Copied

Let's compare uniform and NF4 quantization for normally distributed data:

Out[10]:

Visualization

Histogram of normally distributed weights with uniform INT4 quantization levels shown as evenly spaced red vertical lines. — Uniform INT4 quantization applied to normally distributed weights: red vertical lines show evenly spaced quantization levels that waste precision in the sparse tails.

Histogram of normally distributed weights with NF4 quantization levels shown as clustered green vertical lines near zero. — NormalFloat NF4 quantization applied to normally distributed weights: green vertical lines show levels clustered near zero where the probability density is highest, minimizing reconstruction error.

The uniform scheme wastes quantization levels in the sparse tails while giving coarse granularity where it matters most. NF4 allocates precision according to where values occur. Uniform levels appear evenly distributed regardless of the underlying data, while NF4 levels cluster where the histogram is tallest.

Quantization Error ComparisonLink Copied

Let's measure the reconstruction error for both approaches:

In[11]:

Code

def quantize_uniform(values, n_bits=4):
    """Uniform quantization to n_bits."""
    n_levels = 2**n_bits
    v_min, v_max = values.min(), values.max()

    # Quantize
    normalized = (values - v_min) / (v_max - v_min)
    quantized = np.round(normalized * (n_levels - 1)).astype(int)
    quantized = np.clip(quantized, 0, n_levels - 1)

    # Dequantize
    dequantized = quantized / (n_levels - 1) * (v_max - v_min) + v_min
    return dequantized


def quantize_nf4(values, levels):
    """NF4 quantization using precomputed levels."""
    # Normalize to unit variance
    scale = np.std(values)
    normalized = values / scale

    # Find nearest level for each value
    distances = np.abs(normalized[:, np.newaxis] - levels[np.newaxis, :])
    indices = np.argmin(distances, axis=1)

    # Dequantize
    dequantized = levels[indices] * scale
    return dequantized


# Generate test weights
test_weights = np.random.normal(0, 1, 50000)

# Compute reconstruction errors
uniform_recon = quantize_uniform(test_weights)
nf4_recon = quantize_nf4(test_weights, nf4_levels)

uniform_mse = np.mean((test_weights - uniform_recon) ** 2)
nf4_mse = np.mean((test_weights - nf4_recon) ** 2)
error_reduction = (1 - nf4_mse / uniform_mse) * 100

Out[12]:

Console

Uniform (INT4) MSE: 0.027274
NormalFloat (NF4) MSE: 0.019758
NF4 error reduction: 27.6%

NF4's information-theoretic optimality for normally distributed data translates to substantially lower quantization error. This seemingly technical improvement has major practical implications: lower quantization error means the 4-bit model behaves more like the original FP16 model, preserving capability during fine-tuning. Every reduction in quantization error translates directly to better preservation of the knowledge encoded in the original model weights.

Out[13]:

Visualization

Overlapping histograms comparing error distributions of uniform vs NF4 quantization. — Quantization error distributions for uniform INT4 (orange) and NormalFloat NF4 (green). The NF4 distribution is taller and narrower, showing that errors are highly concentrated near zero, whereas the uniform approach produces a broader spread of errors with heavier tails.