Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

QLoRA › Worked Example: Step-by-Step NF4 Quantization

Michael BrenndoerferDecember 4, 20254 min read

Language AI HandbookAll chapters

Part 35 of 67Parameter-Efficient Fine-tuning

Chapter 259QLoRA › Worked Example: Step-by-Step NF4 Quantization

Reading section 6 of 15 · Worked Example: Step-by-Step NF4 Quantization · 4 minSection 6 of 15 · 4 min

Section 6 of 15 · 4 min read

QLoRA › Worked Example: Step-by-Step NF4 QuantizationOne section · Sections in this chapter6 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To make the quantization process concrete, let's walk through a small numerical example from start to finish. We'll quantize a toy weight vector using NF4 with double quantization, showing every intermediate step. The goal is to demonstrate that quantization is not a black box but a series of well-defined arithmetic operations that we can trace exactly.

Consider a weight block with 8 values (we'll use a small block for illustration; real QLoRA uses blocks of 64 or 128):

W = [-1.23,\ 0.45,\ -0.12,\ 0.89,\ -0.67,\ 0.23,\ 1.34,\ -0.34]

Step 1: Compute the block scale. We use the maximum absolute value in the block as the scale, which maps the largest magnitude weight to plus or minus 1.0 after normalization. This preserves sign information and ensures the normalized values fall in the range $[-1, 1]$ :

s = \max(|W|) = \max(1.23,\ 0.45,\ 0.12,\ 0.89,\ 0.67,\ 0.23,\ 1.34,\ 0.34) = 1.34

Step 2: Normalize the block. Divide each weight by the scale:

W_{\text{norm}} = W / s = [-0.918,\ 0.336,\ -0.090,\ 0.664,\ -0.500,\ 0.172,\ 1.000,\ -0.254]

All values now lie in the range $[-1, 1]$ , which matches the range covered by the NF4 levels after we scale them to unit maximum.

Step 3: Look up the nearest NF4 level. For each normalized weight, find the closest of the 16 NF4 levels. The NF4 levels are placed at equal quantiles of the standard normal distribution, then normalized to the range $[-1, 1]$ . For example, the normalized weight $-0.918$ maps to the NF4 level approximately equal to $-0.929$ (level index 1), and the weight $0.336$ maps to the level approximately equal to $0.356$ (level index 10). This step converts each floating-point value into the 4-bit integer index (0 through 15) stored in memory.

Step 4: Store the 4-bit indices and the scale. Instead of 8 FP32 values (32 bytes total), we now store 8 4-bit integers (4 bytes) plus one FP32 scale (4 bytes), totaling 8 bytes instead of 32. This is the 4x compression ratio. With double quantization applied across many blocks, the FP32 scale would itself be further compressed to INT8, but the savings from that second compression only become meaningful at scale.

Step 5: Dequantize for computation. When we need the weights for a forward pass, we look up the NF4 level values by index and multiply by the scale:

W_{\text{recon}} = \text{NF4\_levels}[\text{indices}] \times s

The reconstructed weights will be close to but not identical to the originals, with small quantization errors. The maximum absolute error is typically less than 4% of the scale value, which is small enough to preserve model behavior during fine-tuning.

In[18]:

Code

# Worked example: NF4 quantization of a small weight block
W_example = np.array([-1.23, 0.45, -0.12, 0.89, -0.67, 0.23, 1.34, -0.34])

# Step 1: Compute scale (max absolute value)
scale_example = np.abs(W_example).max()

# Step 2: Normalize to [-1, 1]
W_norm_ex = W_example / scale_example

# Step 3: Find nearest NF4 level for each normalized weight
# First normalize nf4_levels to [-1, 1] range
nf4_scaled = nf4_levels / np.abs(nf4_levels).max()
distances_ex = np.abs(W_norm_ex[:, np.newaxis] - nf4_scaled[np.newaxis, :])
indices_ex = np.argmin(distances_ex, axis=1)

# Step 4: Dequantize
W_recon_ex = nf4_scaled[indices_ex] * scale_example

# Step 5: Compute reconstruction error
recon_error_ex = W_example - W_recon_ex

Out[19]:

Console

NF4 Quantization Worked Example
============================================================
Original weights:      [-1.23  0.45 -0.12  0.89 -0.67  0.23  1.34 -0.34]
Scale:                 1.3400
Normalized weights:    [-0.918  0.336 -0.09   0.664 -0.5    0.172  1.    -0.254]
Quantized indices:     [ 0 11  6 14  2 10 15  4]
Reconstructed weights: [-1.34    0.3963 -0.1623  0.906  -0.6921  0.2752  1.34   -0.3963]
Reconstruction errors: [ 0.11    0.0537  0.0423 -0.016   0.0221 -0.0452  0.      0.0563]
Max absolute error:    0.1100
Mean squared error:    0.002841

The worked example confirms our theoretical analysis: NF4 achieves accurate reconstruction with 4-bit storage. The reconstruction errors are small relative to the weight magnitudes, and the process is fully invertible given the stored scale and index values. In a real model with thousands of weight blocks, these small errors are distributed broadly enough that they don't accumulate into meaningful degradation of model capability.