Weight Quantization Basics › Per-Tensor vs Per-Channel Quantization

Beyond the symmetric/asymmetric choice, we must decide the granularity at which to compute quantization parameters. Should a single scale cover an entire weight matrix, or should different parts of the matrix have different scales? This decision affects both quantization accuracy and storage overhead. It also touches on a basic property of neural networks: different parts of a model have very different weight distributions, and treating them identically is often suboptimal.

Think of it like calibrating a thermometer for use in different environments. A single calibration might work reasonably well across the range from arctic to tropical temperatures. But if you need maximum accuracy in both environments separately, you calibrate one thermometer for cold conditions and another for hot conditions. The cost is two calibrations instead of one, but the accuracy improvement can be dramatic at the extremes. Quantization granularity works the same way.

The simplest approach computes one scale (and possibly one zero-point) for an entire tensor. All elements share the same quantization mapping, which means every value in the tensor is quantized using identical parameters regardless of where it appears in the matrix.

This is memory-efficient: a weight matrix with millions of elements needs only one or two extra parameters for quantization metadata. The overhead is essentially zero, making per-tensor quantization attractive for memory-constrained environments. However, if different regions of the tensor have very different value ranges, per-tensor quantization performs poorly. Outliers force a large scale, reducing precision for the majority of values.

Consider a weight matrix where most values fall in $[-0.1, 0.1]$ but a few outliers reach $\pm 1.0$. Per-tensor symmetric 8-bit quantization uses $s = 1.0/127 \approx 0.00787$. Values of 0.05 quantize to $\text{round}(0.05/0.00787) = 6$ and dequantize to approximately 0.047. The relative error is 6%. Meanwhile, the outlier at 1.0 maps perfectly to 127 with negligible error. The common-case values suffer large relative errors to accommodate rare outliers.

This phenomenon illustrates a basic tradeoff in quantization: we must balance precision across the entire value distribution. When a single scale must serve all values, extreme values dictate the scale, and typical values pay the precision penalty. The few outliers hold the precision of the entire tensor hostage. This problem becomes severe at 4-bit widths, where the precision budget is so tight that misallocating it to accommodate outliers can cause perceptible quality degradation.

Per-channel quantization addresses this limitation by computing separate scales for each output channel of a weight tensor. For a linear layer with weight matrix $W \in \mathbb{R}^{d_\text{out} \times d_\text{in}}$, where $d_\text{out}$ is the output dimension and $d_\text{in}$ is the input dimension, we compute $d_\text{out}$ different scales, one per row.

This allows each output neuron's weights to use the full integer range based on that neuron's specific value distribution. If neuron 0 has weights in $[-0.1, 0.1]$ and neuron 1 has weights in $[-1.0, 1.0]$, they get scales of approximately 0.00079 and 0.00787 respectively. Each achieves good precision for its own distribution, without being penalized by the other's characteristics. The scale mismatch that causes problems in per-tensor quantization disappears when each channel has its own scale.

The cost is storing $d_\text{out}$ scale values instead of one. For typical transformer dimensions, this is negligible: a 4096-dimensional layer adds 4096 floats (16 KB at FP16) to store scales, while the weight matrix itself contains $4096 \times 4096 = 16\text{M}$ elements (8 MB at 4-bit). The overhead is well under 1%. This small cost buys significant accuracy improvements, particularly for weights with heterogeneous distributions across channels.

Per-channel quantization also integrates cleanly with existing hardware kernels. When dequantizing before or after a matrix multiplication, we can apply the per-channel scales as a vector scaling operation, which most hardware supports efficiently. The scale for each output channel is applied to the corresponding row of the weight matrix during dequantization, requiring one scaling operation per row rather than one per element.

An intermediate approach divides each channel into groups of fixed size (commonly 32, 64, or 128 elements) and computes scales per group. This handles within-channel variation while keeping overhead manageable. The group size represents a tunable parameter that balances precision against storage.

The motivation for per-group quantization becomes clear when you examine how modern LLM weight matrices distribute their values. Within a single output channel, values are not uniformly distributed: some sections of the input dimension may correspond to certain semantic features, and the weights attending to those features may have systematically different magnitudes. A single scale per channel treats all sections identically, potentially misallocating precision within the channel.

For a $4096 \times 4096$ matrix with group size 128, we need $4096 \times (4096/128) = 4096 \times 32 = 131\text{K}$ scale values. At FP16, this adds 262 KB of overhead, roughly 3% of the 4-bit weight size. The finer granularity better handles within-channel distribution variations without significantly inflating storage.

Per-group quantization has become standard in aggressive 4-bit quantization schemes like GPTQ and AWQ, which we will cover in upcoming chapters. The additional precision from per-group scales often makes the difference between acceptable and unacceptable accuracy loss when pushing to very low bit widths. When you have only 16 possible values to represent a continuous distribution, every bit of precision matters, and per-group quantization provides a reliable way to recover precision lost to value distribution variations within a channel.