INT8 Quantization › Per-Tensor vs Per-Channel Quantization

The granularity at which we compute scale factors materially impacts quantization quality. This design choice is a trade-off between simplicity and accuracy, with different granularities being appropriate for different situations. Understanding the trade-off is needed for deploying quantized models effectively, because a choice that works perfectly for one layer might fail badly for another.

Per-tensor quantization uses a single scale factor for an entire weight matrix or activation tensor. Think of it as fitting one set of clothing to an entire crowd: some people will be well-fitted, but anyone materially different from the average will have a poor fit. This approach is simple and introduces minimal overhead, but a single outlier anywhere in the tensor degrades precision everywhere. Imagine a weight matrix where one row has unusually large values: that single row would force a large scale factor that reduces precision for all other rows, potentially turning what should be fine-grained distinctions into coarse approximations.

Per-channel quantization computes separate scale factors for each output channel of a weight matrix. This is like fitting each person with their own set of clothing: everyone gets a good fit, at the cost of measuring each person individually. For a linear layer with weight matrix $\mathbf{W} \in \mathbb{R}^{d_{out} \times d_{in}}$, we compute $d_{out}$ different scale factors, one for each row (corresponding to each output channel):

$$s_j = \frac{\max(|\mathbf{W}_{j,:}|)}{127}$$

where:

• $s_j$: the scale factor for the $j$-th output channel (row)
• $\mathbf{W}_{j,:}$: the vector of weights in the $j$-th output channel (row $j$ of the weight matrix)
• $\max(|\mathbf{W}_{j,:}|)$: the maximum absolute value in that channel's weights
• $127$: the maximum value for a signed 8-bit integer

This allows each output channel to use its full INT8 range, accommodating the fact that different channels often have different magnitude distributions. Consider why this happens: during training, different output features may learn patterns of varying intensity. A channel that detects subtle statistical patterns might have small weights, while a channel that looks for strong, distinctive features might have large weights. Per-channel quantization respects these differences by giving each channel its own appropriately-sized quantization grid.

The overhead is storing $d_{out}$ scale factors instead of one, which is negligible compared to the weight tensor itself. For a layer with 4096 output channels and 4096 input features, we store 4096 scale factors (one per channel) versus over 16 million quantized weights. The scale factors add less than 0.1% to the storage requirements while potentially sharply improving accuracy.

Per-channel quantization for weights combined with per-tensor quantization for activations has emerged as the standard approach. It offers good accuracy with manageable complexity. The asymmetry makes sense: weights are static and can be analyzed offline to compute optimal per-channel scales, while activations vary with each input and would require expensive runtime scale computation for per-channel treatment. Performing per-channel scale computation on activations at every forward pass would add latency that could negate the inference speedup from quantization, defeating the purpose.