INT8 Quantization › Smooth Quantization

The key insight behind smooth quantization is that while activations are hard to quantize (dynamic, contain outliers), weights are easy to quantize (static, relatively uniform). We can mathematically migrate the quantization difficulty from activations to weights through a channel-wise scaling transformation, making the problem tractable.

The intuition is straightforward. If one matrix is "spiky" with outliers and another is "smooth" and well-behaved, we can transfer some of the spikiness from the first to the second. As long as we do this in a mathematically consistent way that preserves the final computation, we have not changed the model's behavior, only redistributed the quantization difficulty to where it is easier to handle. Think of smooth quantization as a seesaw: we load more weight (quantization difficulty) onto the weight side, which can bear it, and unload the activation side, which was struggling under the strain.

The insight that makes this possible is that the product $\mathbf{XW}$ is unchanged if we multiply $\mathbf{X}$ by some diagonal matrix $\mathbf{D}$ and also multiply $\mathbf{W}$ by $\mathbf{D}^{-1}$. These operations are inverses and cancel out. But while they cancel mathematically, they can sharply change the magnitude distributions of both matrices, making each individually much easier to quantize. The question is just how to choose $\mathbf{D}$ to optimally redistribute the difficulty.

Consider a linear layer computing $\mathbf{Y} = \mathbf{X}\mathbf{W}$, where $\mathbf{X} \in \mathbb{R}^{n \times d_{in}}$ is the activation input and $\mathbf{W} \in \mathbb{R}^{d_{in} \times d_{out}}$ is the weight matrix. We introduce a diagonal matrix $\mathbf{S} = \text{diag}(s_1, s_2, \ldots, s_{d_{in}})$ with positive scaling factors, then insert $\mathbf{S}^{-1}\mathbf{S} = \mathbf{I}$ between the two matrices:

$$\begin{aligned} \mathbf{Y} &= \mathbf{X}\mathbf{W} \\ &= (\mathbf{X}\mathbf{S}^{-1})(\mathbf{S}\mathbf{W}) \quad \text{(insert identity } \mathbf{S}^{-1}\mathbf{S} = \mathbf{I} \text{)} \\ &= \hat{\mathbf{X}}\hat{\mathbf{W}} \quad \text{(regroup terms)} \end{aligned}$$

where:

• $\mathbf{Y}$: the output of the linear layer, unchanged by this transformation
• $\mathbf{X}$: the input activation tensor with outlier channels
• $\mathbf{W}$: the weight matrix with relatively uniform distribution
• $\mathbf{S}$: the diagonal smoothing matrix containing scaling factors $s_j$ for each input dimension
• $\hat{\mathbf{X}} = \mathbf{X}\mathbf{S}^{-1}$: the smoothed activations, where each column $j$ is divided by $s_j$
• $\hat{\mathbf{W}} = \mathbf{S}\mathbf{W}$: the adjusted weights, where each row $j$ is multiplied by $s_j$

This transformation is mathematically exact since $\mathbf{S}^{-1}\mathbf{S} = \mathbf{I}$. We have not changed the computation, just how we factor it. The key mathematical property we exploit is that multiplying by a matrix and then its inverse produces the identity, so inserting $\mathbf{S}^{-1}\mathbf{S}$ between $\mathbf{X}$ and $\mathbf{W}$ changes nothing about the final result. We are free to group these factors however we like, and we choose to group $\mathbf{S}^{-1}$ with the activations and $\mathbf{S}$ with the weights.

The transformation affects the data as follows: smoothed activations $\hat{\mathbf{X}} = \mathbf{X}\mathbf{S}^{-1}$ divide each column $j$ of $\mathbf{X}$ by $s_j$, which shrinks outlier channels by choosing large $s_j$ values. Meanwhile, adjusted weights $\hat{\mathbf{W}} = \mathbf{S}\mathbf{W}$ multiply each row $j$ of $\mathbf{W}$ by the same $s_j$, increasing the magnitude of those rows to compensate. If channel $j$ has outlier activations, we choose a large $s_j$. This shrinks the activations (making them easier to quantize) while enlarging the corresponding weights (which are already easy to quantize). We are transferring the outlier problem to a domain where it is more manageable.

How do we pick the $s_j$ values? If we make $s_j$ too large, we might shrink the activations so much that we lose precision in other ways, or we might inflate the weights to the point where they become hard to quantize. The optimal choice balances these competing concerns, and the SmoothQuant paper proposes a principled way to reach this balance.

The SmoothQuant paper proposes balancing the quantization difficulty between activations and weights using a migration strength parameter $\alpha \in [0, 1]$:

$$s_j = \frac{\max(|\mathbf{X}_{:,j}|)^\alpha}{\max(|\mathbf{W}_{j,:}|)^{1-\alpha}}$$

where:

• $s_j$: the smoothing factor for the $j$-th input channel
• $\max(|\mathbf{X}_{:,j}|)$: the maximum absolute value in the $j$-th column of activations, measured across a calibration dataset
• $\max(|\mathbf{W}_{j,:}|)$: the maximum absolute value in the $j$-th row of weights
• $\alpha$: the migration strength hyperparameter controlling how much difficulty shifts from activations to weights

This formula balances two competing goals. When $\alpha = 1$, we have $s_j = \max(|\mathbf{X}_{:,j}|)$, which fully smooths the activations by dividing each channel by its maximum. This makes the activations perfectly balanced but potentially makes weights hard to quantize. When $\alpha = 0$, we have $s_j = 1/\max(|\mathbf{W}_{j,:}|)$, which leaves activations unchanged. The intermediate value $\alpha = 0.5$ gives the geometric mean, which ensures that after transformation, the maximum activation and maximum weight for each channel have roughly equal magnitude, spreading the quantization difficulty evenly.

To understand this formula intuitively, consider what "equal difficulty" means. After applying the transformation, the smoothed activation maximum for channel $j$ is $\max(|\mathbf{X}_{:,j}|)/s_j$, and the adjusted weight maximum is $\max(|\mathbf{W}_{j,:}|) \cdot s_j$. For these to be equal:

$$\frac{\max(|\mathbf{X}_{:,j}|)}{s_j} = \max(|\mathbf{W}_{j,:}|) \cdot s_j$$

Solving for $s_j$:

$$s_j^2 = \frac{\max(|\mathbf{X}_{:,j}|)}{\max(|\mathbf{W}_{j,:}|)} \implies s_j = \sqrt{\frac{\max(|\mathbf{X}_{:,j}|)}{\max(|\mathbf{W}_{j,:}|)}}$$

This is exactly the $\alpha = 0.5$ case of the SmoothQuant formula, confirming that $\alpha = 0.5$ is the "equal difficulty" point. Values above 0.5 shift more difficulty to the weights, values below 0.5 leave more with the activations.

In practice, $\alpha = 0.5$ works well for most models, achieving roughly equal quantization ranges for both activations and weights after smoothing. However, some models may benefit from different values. Models with particularly severe activation outliers might use $\alpha > 0.5$ to more aggressively smooth the activations, while models with already-difficult-to-quantize weights might use $\alpha < 0.5$ to avoid making the weights worse.

A necessary practical advantage of smooth quantization is that the transformation can be computed entirely offline during a calibration phase. This means inference has no additional runtime cost compared to standard quantized inference. The calibration procedure works as follows:

First, run a small calibration dataset (typically a few hundred representative samples) through the model to collect activation statistics. During this calibration forward pass, we record the maximum absolute value seen in each channel across all calibration samples. Second, compute $\max(|\mathbf{X}_{:,j}|)$ for each channel using these statistics, capturing how large the outliers tend to be. Third, calculate smoothing factors $s_j$ for each layer using the formula above. Fourth, apply the weight transformation $\hat{\mathbf{W}} = \mathbf{S}\mathbf{W}$ and store the modified weights. Fifth, and most cleverly, fuse $\mathbf{S}^{-1}$ into the preceding layer's weights or normalization parameters.

Step 5 is important and deserves careful explanation. Rather than explicitly dividing activations by $\mathbf{S}$ at runtime, we absorb this division into the bias of the previous layer or the scale/shift parameters of the preceding LayerNorm. This means inference has zero additional cost compared to standard quantized inference. The activation values arrive at the quantization step already "smoothed" because the previous layer was modified to produce smoothed outputs.

The fusion works because transformer architectures typically have a LayerNorm immediately before each linear layer. LayerNorm applies a learned scale ($\gamma$) and shift ($\beta$) to each channel. We can incorporate the smoothing factor directly into these existing parameters. If the original LayerNorm scale for channel $j$ was $\gamma_j$ and shift was $\beta_j$, we replace them with $\gamma_j/s_j$ and $\beta_j/s_j$. The activations coming out of the modified LayerNorm are already "smoothed," and we never perform the division at runtime. This is mathematically equivalent to explicitly dividing at every forward pass but requires zero additional computation.

The offline nature of smooth quantization also means it is compatible with any existing INT8 quantization infrastructure. You do not need to modify the inference engine or write new kernels. Simply transform the weights and LayerNorm parameters before quantizing, then proceed with standard absmax or per-channel quantization. The resulting model can be deployed on any hardware that supports INT8 matrix multiplication.