Weight Quantization Basics › Implementation
Language AI HandbookAll chapters Weight Quantization Basics › ImplementationOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let us implement the core quantization operations to solidify these concepts. We will start with basic symmetric quantization, then extend to asymmetric and per-channel variants. Working through concrete code helps build intuition for how the mathematical formulas translate to practice, and reveals the engineering choices that pure mathematical treatment leaves implicit.
import torch
# Create a sample weight tensor with a realistic distribution. The shared setup
# seed keeps every render variant deterministic.
weights = torch.randn(64, 128) * 0.02 # Typical initialization scaleWeight tensor shape: torch.Size([64, 128]) Value range: [-0.0767, 0.0689] Mean: 0.000130, Std: 0.0200
The weights follow a roughly symmetric distribution centered near zero, typical of neural network parameters after initialization or training. This distribution is well-suited to symmetric quantization, which we will implement first. The standard deviation of approximately 0.02 matches common initialization scales, and the range is several standard deviations wide, with occasional outliers from the tails of the Gaussian.
Symmetric Per-Tensor Quantization
The symmetric per-tensor implementation demonstrates the simplest form of quantization. We compute a single scale from the maximum absolute value, then apply the same quantization formula to every element in the tensor. The implementation is only a few lines of code, which reflects the mathematical simplicity of the symmetric scheme.
def quantize_symmetric(tensor, num_bits=8):
"""Symmetric per-tensor quantization."""
qmax = 2 ** (num_bits - 1) - 1 # For signed: 127 for 8-bit
# Compute scale from maximum absolute value
max_abs = tensor.abs().max()
scale = max_abs / qmax
# Quantize: round(x / scale)
q_tensor = torch.round(tensor / scale).clamp(-qmax - 1, qmax)
q_tensor = q_tensor.to(torch.int8)
return q_tensor, scale
def dequantize_symmetric(q_tensor, scale):
"""Dequantize symmetrically quantized tensor."""
return q_tensor.float() * scale# Quantize and dequantize
q_weights, scale = quantize_symmetric(weights, num_bits=8)
reconstructed = dequantize_symmetric(q_weights, scale)
# Compute quantization error
error = (weights - reconstructed).abs()Scale: 0.000604 Quantized values range: [-127, 114] Mean absolute error: 0.000149 Max absolute error: 0.000302 Relative error (vs std): 0.7448%
The mean absolute error is small compared to the weight standard deviation. 8-bit symmetric quantization preserves weights well when the distribution is roughly symmetric. The relative error of less than 1% indicates that the quantized weights closely approximate the originals, which bodes well for maintaining model accuracy. The scale parameter encodes exactly how much real-valued range each integer step represents, and comparing it to the weight standard deviation gives a sense of quantization resolution relative to the spread of the data.
Asymmetric Per-Tensor Quantization
The asymmetric implementation adds complexity by computing and tracking a zero-point. This extra parameter allows better use of the integer range when values are not centered at zero. The implementation shows how the zero-point changes both quantization and dequantization, and how careful handling is needed to avoid edge cases like division by zero for constant tensors.
def quantize_asymmetric(tensor, num_bits=8):
"""Asymmetric per-tensor quantization."""
qmin, qmax = 0, 2**num_bits - 1 # Unsigned: 0 to 255
# Compute scale and zero-point
x_min, x_max = tensor.min(), tensor.max()
scale = (x_max - x_min) / (qmax - qmin)
# Avoid division by zero for constant tensors
if scale == 0:
scale = torch.tensor(1.0)
zero_point = qmin - torch.round(x_min / scale)
zero_point = zero_point.clamp(qmin, qmax)
# Quantize
q_tensor = torch.round(tensor / scale) + zero_point
q_tensor = q_tensor.clamp(qmin, qmax).to(torch.uint8)
return q_tensor, scale, zero_point
def dequantize_asymmetric(q_tensor, scale, zero_point):
"""Dequantize asymmetrically quantized tensor."""
return (q_tensor.float() - zero_point) * scale# Quantize asymmetrically
q_weights_asym, scale_asym, zp = quantize_asymmetric(weights, num_bits=8)
reconstructed_asym = dequantize_asymmetric(q_weights_asym, scale_asym, zp)
error_asym = (weights - reconstructed_asym).abs()Scale: 0.000571 Zero-point: 134 Quantized values range: [0, 255] Mean absolute error: 0.000143 Relative error (vs std): 0.7129%
For this symmetric weight distribution, asymmetric quantization provides similar accuracy to symmetric. The zero-point is near 128, roughly the middle of the unsigned range, as expected for zero-centered data. When the data is already symmetric, asymmetric quantization offers little benefit but also causes no harm. The computation confirms our theoretical expectation: for symmetric data, both schemes use the integer range similarly and produce comparable errors.
Per-Channel Quantization
Per-channel quantization computes separate scales for each row of the weight matrix, letting each output channel to use its full precision budget independently. The implementation uses PyTorch's broadcasting-aware operations to handle all channels simultaneously.
def quantize_per_channel(tensor, num_bits=8, axis=0):
"""Symmetric per-channel quantization along specified axis."""
qmax = 2 ** (num_bits - 1) - 1
# Compute scale per channel (reduce across the non-channel dimensions)
# For 2D matrices: if axis=0 (rows), reduce dim 1. If axis=1 (cols), reduce dim 0.
reduce_dim = 1 if axis == 0 else 0
max_abs = tensor.abs().amax(dim=reduce_dim, keepdim=True)
scale = max_abs / qmax
scale = torch.where(scale == 0, torch.ones_like(scale), scale)
# Quantize
q_tensor = torch.round(tensor / scale).clamp(-qmax - 1, qmax)
q_tensor = q_tensor.to(torch.int8)
return q_tensor, scale
def dequantize_per_channel(q_tensor, scale, axis=0):
"""Dequantize per-channel quantized tensor."""
# Scale already has correct broadcasting shape from quantize
return q_tensor.float() * scaleTo illustrate the benefits of per-channel quantization, we create a tensor where different channels have dramatically different value ranges. This simulates the real-world situation where some neurons develop larger weights than others during training. In practice, weight magnitude heterogeneity is common and can be severe: some neurons act as feature detectors for rare but important patterns and develop large weights, while others handle common patterns with smaller weights.
# Create a tensor with varying scales per channel
weights_varied = weights.clone()
weights_varied[0] *= 10 # First channel has larger values
weights_varied[1] *= 0.1 # Second channel has smaller values
# Compare per-tensor vs per-channel
q_pt, s_pt = quantize_symmetric(weights_varied)
recon_pt = dequantize_symmetric(q_pt, s_pt)
q_pc, s_pc = quantize_per_channel(weights_varied, axis=0)
recon_pc = dequantize_per_channel(q_pc, s_pc, axis=0)
# Compute quantization errors
error_pt = (weights_varied - recon_pt).abs()
error_pc = (weights_varied - recon_pc).abs()Per-tensor quantization: Single scale: 0.003952 Channel 0 error (large weights): 0.001089 Channel 1 error (small weights): 0.000978 Per-channel quantization: Channel 0 scale: 0.003952 Channel 1 scale: 0.000039 Channel 0 error: 0.001089 Channel 1 error: 0.000010
Per-channel quantization substantially reduces error for channel 1. With per-tensor quantization, the large channel forces a big scale, causing the small channel to quantize coarsely. Per-channel quantization gives each channel an appropriate scale tailored to its own value range. The error reduction for channel 1 is several orders of magnitude. This shows concretely why per-channel quantization is needed for high-accuracy quantization when weight distributions vary across channels.
Visualizing Quantization Error
Visualization helps build intuition about how quantization errors distribute across values. The following plots show both the error distribution and the relationship between original and reconstructed values.
import matplotlib.pyplot as plt
use_book_style()
plt.rcParams["figure.figsize"] = (6.0, 4.4)
plt.figure()
ax = plt.gca()
ax.hist(error.flatten().numpy(), bins=50, edgecolor=PALETTE["ink"], alpha=0.7)
ax.axvline(
scale.item() / 2,
color=theme_color("red"),
linestyle="--",
label=f"Scale/2 = {scale.item() / 2:.5f}",
)
ax.set_xlabel("Absolute Error")
ax.set_ylabel("Frequency")
ax.set_title("Quantization Error Distribution")
ax.legend(
loc="upper center",
bbox_to_anchor=(0.5, -0.16),
borderaxespad=0,
)
polish_axes(ax)
plt.show()
import matplotlib.pyplot as plt
import numpy as np
use_book_style()
plt.rcParams["figure.figsize"] = (6.0, 4.0)
plt.figure()
sample_idx = np.random.choice(weights.numel(), 1000)
orig_sample = weights.flatten()[sample_idx].numpy()
recon_sample = reconstructed.flatten()[sample_idx].numpy()
plt.scatter(orig_sample, recon_sample, alpha=0.5, s=10)
plt.plot(
[orig_sample.min(), orig_sample.max()],
[orig_sample.min(), orig_sample.max()],
color=PALETTE["coral"],
linestyle="--",
label="Perfect reconstruction",
)
plt.xlabel("Original Weight")
plt.ylabel("Reconstructed Weight")
plt.title("Original vs Reconstructed Values")
plt.legend()
polish_axes(plt.gca())
plt.show()
The error histogram shows that quantization errors are bounded and mostly small. The maximum possible error is half the scale (when a value falls exactly between two quantization levels), and the red dashed line marks this theoretical bound. The scatter plot confirms tight correlation between original and reconstructed values, with points clustering closely around the diagonal line that represents perfect reconstruction. The slight discretization visible in the reconstructed values (they can only take values that are multiples of the scale) creates the characteristic "staircase" pattern visible in this plot.
Simulating Calibration
Calibration in practice involves running the model on representative data to collect statistics. The following code simulates this process for a simple model. This shows both minmax and percentile-based calibration approaches.
def calibrate_activations(
model_fn, calibration_data, num_bits=8, method="minmax"
):
"""
Calibrate quantization parameters for activations.
model_fn: Function that takes input and returns activations
calibration_data: List of input tensors
method: 'minmax' or 'percentile'
"""
all_activations = []
# Collect activations across calibration set
for data in calibration_data:
activations = model_fn(data)
all_activations.append(activations)
all_activations = torch.cat(all_activations, dim=0)
if method == "minmax":
x_min = all_activations.min()
x_max = all_activations.max()
elif method == "percentile":
# Use 99.9th percentile to exclude outliers
x_min = torch.quantile(all_activations.float(), 0.001)
x_max = torch.quantile(all_activations.float(), 0.999)
# Compute asymmetric quantization parameters
qmin, qmax = 0, 2**num_bits - 1
scale = (x_max - x_min) / (qmax - qmin)
zero_point = qmin - torch.round(x_min / scale)
return scale, zero_point, x_min, x_max# Define fixed weights for the mock model
input_dim = 128
hidden_dim = 64
W = torch.randn(input_dim, hidden_dim) * 0.1
b = torch.randn(hidden_dim) * 0.01
# Simulate a simple activation function
def mock_model(x):
# Simulates ReLU-like activations (mostly positive with some outliers)
# Using fixed weights ensures consistent behavior across calls
return torch.relu(x @ W + b)
# Generate calibration data
calibration_data = [torch.randn(32, 128) for _ in range(10)]
# Calibrate with both methods
scale_mm, zp_mm, min_mm, max_mm = calibrate_activations(
mock_model, calibration_data, method="minmax"
)
scale_pct, zp_pct, min_pct, max_pct = calibrate_activations(
mock_model, calibration_data, method="percentile"
)MinMax Calibration: Range: [0.0000, 4.1836] Scale: 0.016406 Percentile Calibration (99.9%): Range: [0.0000, 3.4641] Scale: 0.013585 Scale reduction from percentile clipping: 17.2%
Percentile-based calibration produces a smaller scale. This provides finer quantization resolution for the bulk of activations at the cost of clipping rare outliers. The scale reduction directly translates to smaller quantization errors for typical values, which usually improves overall model accuracy. The trade is explicit: we accept large errors for the 0.1% most extreme activation values in exchange for smaller errors for the other 99.9%. Whether this is a good trade depends on how much those extreme values matter for your application.
Comparing Bit Widths
The following experiment measures how quantization error changes with bit width. This shows the exponential relationship between precision and error. This relationship is basic to understanding why moving from 8-bit to 4-bit quantization is much more challenging than moving from 32-bit to 16-bit.
def measure_quantization_error(tensor, num_bits):
"""Measure mean squared error for given bit width."""
q, s = quantize_symmetric(tensor, num_bits)
recon = dequantize_symmetric(q, s)
mse = ((tensor - recon) ** 2).mean()
return mse.item()
# Test different bit widths
bit_widths = [2, 3, 4, 5, 6, 7, 8]
errors = [measure_quantization_error(weights, b) for b in bit_widths]import matplotlib.pyplot as plt
use_book_style()
plt.rcParams["figure.figsize"] = (6.0, 4.0)
plt.figure()
ax = plt.gca()
ax.plot(bit_widths, errors, "o-", markersize=5.5, linewidth=1.1)
use_book_log_scale(ax, axis="y")
plt.xlabel("Bit Width")
plt.ylabel("Mean Squared Error (log scale)")
plt.title("Quantization Error vs Bit Width")
plt.xticks(bit_widths)
polish_axes(ax)
# Add annotations
for bw, err in zip(bit_widths, errors):
plt.annotate(
f"{err:.2e}",
(bw, err),
textcoords="offset points",
xytext=(0, 10),
ha="center",
fontsize=8,
color=PALETTE["ink"],
bbox={
"boxstyle": "round,pad=0.18",
"facecolor": PALETTE["paper"],
"edgecolor": "none",
"alpha": 0.92,
},
)
plt.show()
Error drops roughly by a factor of 4 for each additional bit. This follows from quantization theory: each extra bit doubles the number of representable levels, halving the quantization step size, which reduces mean squared error by a factor of 4. On the log-scale plot, this appears as a straight line with a slope of 4x per bit. This 4x-per-bit rule is a useful mental model for reasoning about quantization tradeoffs: the difference between 8-bit and 4-bit quantization in error terms is approximately , a factor of 256 in mean squared error. This is a large change, which explains why aggressive 4-bit quantization requires much more sophisticated techniques than simple 8-bit quantization.