INT8 Quantization: Absmax

In the previous chapter on weight quantization basics, we explored why representing neural network parameters with fewer bits can sharply reduce memory requirements and accelerate inference. We saw how the core ideas of scale factors and zero-points let us map floating-point values onto discrete integer grids. We now turn to the most widely deployed quantization format in production systems: 8-bit integers (INT8). This precision point is a sweet spot that the machine learning community discovered through years of experimentation: memory savings are substantial (2x compression from FP16, 4x from FP32), specialized hardware acceleration is widely available, and accuracy degradation remains manageable for most applications. Understanding INT8 quantization in depth means understanding every major LLM inference library used today, from bitsandbytes to TensorRT-LLM.

INT8 quantization is about storing numbers more compactly and running them faster. Modern GPUs and specialized AI accelerators include dedicated INT8 matrix multiplication units that can reach 2-4x higher throughput than their floating-point counterparts. NVIDIA's Tensor Cores, for example, perform INT8 operations at twice the rate of FP16 operations. Quantizing both weights and activations to INT8 therefore reduces memory use and can accelerate computation. The improvement comes from two compounding effects: smaller data types require less memory bandwidth to load, and dedicated integer arithmetic units can perform more operations per clock cycle than their floating-point equivalents. On memory-bandwidth-limited tasks like large-batch LLM inference, these effects can substantially increase throughput.

The challenge is that this sounds easier than it is. Squeezing the continuous range of floating-point values into just 256 discrete integers inevitably introduces quantization error. Every value you store must round to one of those 256 representable points, and the difference between the original and rounded value accumulates across millions of operations. For small models and simple tasks, this noise is negligible. For large language models with billions of parameters and complex dependencies between layers, naive INT8 quantization can cause large accuracy degradation. The field had to develop principled techniques for minimizing this error before INT8 LLM inference became practical.

The journey to effective INT8 quantization for LLMs involved a surprising discovery: the problem was not the weights but the activations. Weights are static tensors we can analyze carefully offline, adjusting our quantization scheme to fit their distribution. Activations are dynamic, varying with every input, and in large language models they develop pathological outliers that resist standard quantization approaches. A small number of hidden dimensions produce values sharply larger than all other dimensions, which makes it impossible to choose a quantization scale that works well for both the outliers and the typical values. This observation, and the elegant mathematical fix for it, forms the intellectual core of this chapter.

We will explore the mathematics of range mapping, the practical absmax quantization scheme that forms the foundation, the granularity choices between per-tensor and per-channel quantization, the emergent outlier problem in large language models, and the smooth quantization technique that makes INT8 viable even for models with hundreds of billions of parameters. By the end you will understand how to apply these techniques, why each design choice was made, and what happens when things go wrong.

An 8-bit signed integer can stand for values from -128 to 127, giving us exactly 256 distinct values to work with. Think of this as having 256 labeled slots on a number line, and every floating-point weight or activation value must be assigned to the nearest slot. When we quantize a floating-point tensor, we create a mapping (technically a codebook) that associates each of these 256 integers with a specific floating-point value. The integers become the compact representation we store and compute with, and the codebook lets us convert back to floating-point whenever we need the full precision.

To understand this mapping intuitively, imagine you have a number line representing all possible floating-point values a weight might take. Quantization places 256 evenly-spaced "buckets" along this line, and every floating-point value must be assigned to its nearest bucket. The original value is then represented by that bucket's integer label, and reconstruction involves converting the label back to the bucket's center position. The spacing between these buckets, and where they are positioned along the number line, completely determines the quality of our approximation. Place them well and every value lands close to its bucket center. Place them poorly and some values are forced into buckets far from where they belong.

The basic challenge is choosing how to position this grid. Consider a weight tensor with values ranging from -0.5 to 1.2. We need to decide what floating-point value should -128 stand for, what floating-point value should 127 stand for, and how to handle values outside our chosen range (clipping them to the boundary). These choices define our quantization scheme and directly impact the reconstruction error. A poorly chosen grid might waste many of its 256 integers representing values that never occur in the tensor, while cramming the observed values into too few buckets. The goal is to align our quantization grid as closely as possible with the distribution of values we need to stand for.

The key insight is that the quality of quantization depends almost entirely on how well the grid matches the distribution of values in the tensor. If our weights happen to be uniformly distributed between -1 and 1, a grid spanning that range is nearly optimal. But if most weights cluster near zero with a long tail of larger values, we need to make a choice: use a wide grid that accommodates the tail but has poor resolution near zero, or use a narrow grid that captures the bulk well but clips the tail. Neither choice is free, and the right answer depends on which values matter more for the model's accuracy.

The simplest and most common approach to INT8 quantization is symmetric quantization, where we center the quantization grid at zero. The "absmax" method finds the maximum absolute value in the tensor and maps it to the largest representable integer. This approach derives its name from the key statistic it computes: the absolute maximum, which determines the extent of our quantization range in both positive and negative directions simultaneously.

Think of absmax quantization as stretching or compressing a rubber band. Your floating-point values are marked on the band, and you resize it until the most extreme mark lines up with the INT8 boundary at -128 or 127. The scale factor tells you how much you stretched or compressed. Every other value gets rescaled by the same factor, preserving relative distances. The reconstruction step is just reversing this scaling operation. The elegance of this approach is that a single number, the scale factor, captures everything you need to know about the quantization.

The intuition behind absmax quantization is straightforward. We want to ensure that no value in our tensor falls outside the representable range, which would force us to clip it and introduce potentially large errors. By finding the largest magnitude value in the tensor and designing our grid to just barely accommodate it, we guarantee that every original value can be represented without clipping. At the same time, we make the grid as fine as possible given this constraint, since using a larger range than necessary would waste precision on integers that never get used.

Given a tensor $\mathbf{x}$ with floating-point values, absmax quantization works as follows. First we compute the scale factor from the tensor's extreme value, then we apply rounding to obtain integer codes:

where:

• $\mathbf{x}$: the input floating-point tensor we want to quantize
• $\alpha$: the maximum absolute value in the input tensor, called the "absmax"
• $s$: the scale factor that determines how much floating-point distance each integer step is
• $127$: the maximum representable value in a signed 8-bit integer (we reserve -128 as an "overflow" value and map the positive side to $[0, 127]$)
• $\mathbf{x}_q$: the resulting tensor of quantized 8-bit integers, each in $[-128, 127]$

Let's trace through this process step by step. First, we scan the entire tensor to find $\alpha$, the largest magnitude value regardless of sign. This becomes our reference point for setting the scale. Next, we compute the scale factor $s$ by dividing $\alpha$ by 127, which tells us how much "floating-point distance" each integer step is. If $\alpha = 0.635$, then $s \approx 0.005$, meaning each integer step covers about 0.005 in floating-point space. Finally, we quantize each value by dividing it by the scale factor and rounding to the nearest integer. This rounding step is where the actual information loss occurs, as we snap each continuous value to its nearest discrete representation.

To recover an approximation of the original values, we simply multiply by the scale:

where:

• $\hat{\mathbf{x}}$: the reconstructed floating-point tensor, an approximation of the original $\mathbf{x}$
• $s$: the scale factor used during quantization (must be stored alongside the quantized tensor)
• $\mathbf{x}_q$: the quantized integer tensor we stored

The hat notation indicates this is a reconstruction, not the exact original values. The difference $\mathbf{x} - \hat{\mathbf{x}}$ is the quantization error. This error arises because the rounding operation discards fractional information. When we divided by $s$ and rounded, we lost the remainder, and multiplying the rounded result back by $s$ cannot restore what was lost. The beauty of this scheme is that the error is bounded: no reconstructed value can differ from its original by more than half a scale step, or $s/2$. For a tensor with absmax of 1.0, the scale is $1.0/127 \approx 0.0079$, so the maximum error is about 0.004. This is roughly 0.4% of the full range, which is small enough to be acceptable for most neural network weights.

Centering the grid at zero has a important computational advantage: the integer zero maps exactly to floating-point zero. This property might seem like a minor mathematical nicety, but it has significant practical implications for neural network inference.

This matters because many neural network activations are zero (from ReLU, dropout, and padding), zero-valued weights do not contribute to computations, and preserving exact zeros avoids accumulating unnecessary rounding errors. Consider what happens during a matrix multiplication when many elements are zero. If zero maps exactly to zero, these elements contribute nothing to the output, exactly as they should. But if zero were represented by some non-zero integer due to an offset, every "zero" element would contribute a small spurious value to the computation. Over millions of operations, these spurious contributions could accumulate into real errors.

Symmetric quantization also simplifies the mathematics during matrix multiplication. When computing $\mathbf{Y} = \mathbf{X}\mathbf{W}$ with quantized values, we only need to track scale factors, not additional offset terms. This simplification translates directly into faster inference, since the dequantization step requires only a single multiplication rather than a multiplication followed by an addition. When both scales are scalar constants (as they are for per-tensor quantization), this final rescaling is in effect free compared to the matrix multiplication itself.

Absmax quantization clips all values to the range $[-\alpha, \alpha]$. If $\alpha$ is dominated by a few outlier values, the quantization grid becomes coarse for the majority of values. This creates a basic tension in quantization design: we want to accommodate outliers to avoid clipping errors, but we also want a fine grid to minimize rounding errors for typical values.

Consider a tensor where 99% of values lie between -0.1 and 0.1, but one outlier is 5.0. Using absmax:

This scale means values around 0.1 get quantized to just $\text{round}(0.1/0.0394) = 3$. The reconstruction is $3 \times 0.0394 = 0.118$, introducing significant relative error for these common values. To understand the magnitude of this problem, note that a typical value of 0.1 experiences an 18% relative error, while the outlier value of 5.0 would be reconstructed nearly perfectly. We have sacrificed precision where it matters most, on the bulk of our values, to perfectly preserve a single outlier.

One solution is to clip outliers: choose $\alpha$ smaller than the true maximum, accepting that extreme values will be clipped. This trades outlier accuracy for better precision on the bulk of values. For example, if we set $\alpha = 0.5$ instead of 5.0, our scale becomes $s = 0.5/127 \approx 0.00394$, and a value of 0.1 now quantizes to integer 25 with reconstruction $25 \times 0.00394 = 0.0985$, a much smaller relative error. The outlier would be clipped to 127, reconstructing to 0.5 instead of 5.0, but this might be an acceptable trade-off depending on the model's sensitivity to those extreme values.

Finding the optimal clipping threshold is the basis for calibration-based quantization methods. These methods analyze the value distribution across representative data to find the clipping point that minimizes total reconstruction error, balancing the clipping errors from outliers against the rounding errors for typical values. Calibration is often as important as the quantization scheme itself, and models that fail to quantize well with default settings often succeed after proper calibration.

When tensor values are not centered around zero, symmetric quantization wastes representable range. ReLU activations, for example, are always non-negative. Mapping the range $[0, 3.5]$ symmetrically would waste half our integers representing negative values that never occur. This inefficiency becomes severe when the actual data distribution is heavily skewed or shifted away from zero. With 256 integers available and only half the range needed, symmetric quantization effectively reduces your precision from 8 bits to 7 bits for these tensors.

Asymmetric quantization solves this by introducing a zero-point offset $z$ that shifts the quantization grid to the data's range. Think of it as sliding our 256-slot number line left or right until it lines up with the data distribution, rather than always keeping it centered at zero. The grid spacing remains uniform, but we can position it anywhere. This flexibility comes at a cost: more complex arithmetic during inference, as we now have two parameters (scale and zero-point) to track rather than one.

The asymmetric quantization formulas show how to derive both parameters from the data range:

where:

• $\mathbf{x}$: the input floating-point tensor
• $s$: the scale factor, computed from the full data range (minimum to maximum)
• $255$: the number of intervals in an 8-bit range ($2^8 - 1 = 255$ intervals for 256 values)
• $z$: the zero-point integer that shifts the grid to align with the data distribution
• $\mathbf{x}_q$: the resulting quantized tensor with values in $[-128, 127]$

The key difference from symmetric quantization lies in how we compute the scale and the introduction of the zero-point. Instead of basing the scale on the maximum absolute value, we use the full range from minimum to maximum. This keeps all 256 integers can potentially be used. The zero-point $z$ is an integer that shifts the quantization grid so that the minimum value maps to -128 and the maximum to 127. This shift allows us to "slide" our quantization grid along the number line to wherever the actual data lies.

Dequantization reverses this process by first undoing the zero-point shift and then applying the scale:

where:

• $\hat{\mathbf{x}}$: the reconstructed floating-point values
• $s$: the scale factor
• $\mathbf{x}_q$: the quantized integer values
• $z$: the zero-point offset, which is subtracted before scaling to recover the original range

While asymmetric quantization can stand for arbitrary ranges more efficiently, it complicates the mathematics during inference. Matrix multiplications now involve additional terms from the zero-point offsets. When we expand the matrix multiplication with asymmetric quantization, cross-terms involving the zero-points appear, requiring additional computation and memory access. For weights, which are typically centered around zero due to common initialization and regularization practices, symmetric quantization remains the standard choice. The computational overhead of asymmetric quantization is generally reserved for activations where the efficiency gains from better range utilization outweigh the additional complexity.

The granularity at which we compute scale factors materially impacts quantization quality. This design choice is a trade-off between simplicity and accuracy, with different granularities being appropriate for different situations. Understanding the trade-off is needed for deploying quantized models effectively, because a choice that works perfectly for one layer might fail badly for another.

Per-tensor quantization uses a single scale factor for an entire weight matrix or activation tensor. Think of it as fitting one set of clothing to an entire crowd: some people will be well-fitted, but anyone materially different from the average will have a poor fit. This approach is simple and introduces minimal overhead, but a single outlier anywhere in the tensor degrades precision everywhere. Imagine a weight matrix where one row has unusually large values: that single row would force a large scale factor that reduces precision for all other rows, potentially turning what should be fine-grained distinctions into coarse approximations.

Per-channel quantization computes separate scale factors for each output channel of a weight matrix. This is like fitting each person with their own set of clothing: everyone gets a good fit, at the cost of measuring each person individually. For a linear layer with weight matrix $\mathbf{W} \in \mathbb{R}^{d_{out} \times d_{in}}$, we compute $d_{out}$ different scale factors, one for each row (corresponding to each output channel):

where:

• $s_j$: the scale factor for the $j$-th output channel (row)
• $\mathbf{W}_{j,:}$: the vector of weights in the $j$-th output channel (row $j$ of the weight matrix)
• $\max(|\mathbf{W}_{j,:}|)$: the maximum absolute value in that channel's weights
• $127$: the maximum value for a signed 8-bit integer

This allows each output channel to use its full INT8 range, accommodating the fact that different channels often have different magnitude distributions. Consider why this happens: during training, different output features may learn patterns of varying intensity. A channel that detects subtle statistical patterns might have small weights, while a channel that looks for strong, distinctive features might have large weights. Per-channel quantization respects these differences by giving each channel its own appropriately-sized quantization grid.

The overhead is storing $d_{out}$ scale factors instead of one, which is negligible compared to the weight tensor itself. For a layer with 4096 output channels and 4096 input features, we store 4096 scale factors (one per channel) versus over 16 million quantized weights. The scale factors add less than 0.1% to the storage requirements while potentially sharply improving accuracy.

Per-channel quantization for weights combined with per-tensor quantization for activations has emerged as the standard approach. It offers good accuracy with manageable complexity. The asymmetry makes sense: weights are static and can be analyzed offline to compute optimal per-channel scales, while activations vary with each input and would require expensive runtime scale computation for per-channel treatment. Performing per-channel scale computation on activations at every forward pass would add latency that could negate the inference speedup from quantization, defeating the purpose.

To build concrete intuition, let's trace through a complete quantization example with specific numbers. This will make the abstractions tangible and reveal where errors accumulate.

Suppose we have a small weight tensor (one row of a larger matrix):

Step 1: Compute the absmax. We scan the entire vector and find:

Step 2: Compute the scale factor. Dividing by 127 to cover the full INT8 range:

Step 3: Quantize each element. Divide by $s$ and round to the nearest integer:

The quantized tensor is: $\mathbf{w}_q = [95, -26, 127, -76, 6, -119, 37, -103]$.

Step 4: Reconstruct and compute error. Multiply each integer by $s$:

The maximum error is 0.0037, and the mean absolute error is about 0.0015. Notice that the largest element (1.12) reconstructs perfectly since it was the absmax and maps exactly to 127. The element with value 0.05 (the smallest magnitude) experiences the largest relative error: the rounding from 5.67 to 6 introduces a 5.9% relative error. This pattern is typical: absmax quantization gives best relative precision to large-magnitude values and worst to small ones.

The maximum possible error for any element is $s/2 = 0.008819/2 = 0.00441$, which all our actual errors satisfy. Storing the entire vector requires just 8 bytes (8 integers) plus 4 bytes for the float32 scale factor, totaling 12 bytes instead of the original 32 bytes (8 float32 values). That is 2.67x compression with bounded error.

As language models scale to billions of parameters, a problematic pattern emerges: certain activation dimensions develop extreme outlier values. Research has shown that in models like OPT-175B and BLOOM-176B, a small number of hidden dimensions (sometimes called "large activations" or "emergent features") can have values 10-100x larger than typical activations. This phenomenon was not anticipated by early quantization research conducted on vision models and smaller transformers, and it poses a significant challenge to naive INT8 approaches.

The emergence of these outliers appears to be linked to how transformers process information at scale. Certain hidden dimensions seem to serve as "highways" for important information, developing consistently large activation magnitudes across many inputs. These are not random fluctuations but systematic features of the model's learned representations. The same channels show outlier behavior across different inputs and different layers, suggesting they encode something structurally important about how the model is information. While we are still working to fully understand why this happens, the practical implication is clear: any quantization strategy for large language models must account for these outliers.

The key insight is that outlier channels break the basic assumption behind per-tensor quantization. When we assume all channels have similar magnitude distributions and can share a single scale factor, we implicitly assume that no channel is sharply different from the others. The outlier phenomenon violates this assumption catastrophically. One channel's behavior forces a scale factor that is appropriate for that channel alone and wildly suboptimal for all others.

Consider a hidden state where 99.9% of values lie in $[-1, 1]$, but dimension 1847 consistently produces values around 50. Per-tensor absmax quantization would set $s = 50/127 \approx 0.39$, meaning values of magnitude 1 quantize to just 2-3 integers. The reconstruction error for these common values becomes unacceptable. A value of 0.5, which might be important for the model's computation, would quantize to either 1 or 2, introducing errors of up to 20% in its representation. This is not a minor precision loss that can be ignored. It can cause the model to make qualitatively wrong predictions.

The naive solutions all have significant drawbacks:

• Clipping outliers: Destroys important model information encoded in those dimensions. The outlier channels are apparently important, not artifacts.
• Per-channel activation quantization: Requires computing new scale factors for every token at runtime, adding latency and complexity.
• Mixed precision: Keeping outlier channels in FP16 complicates kernels and reduces throughput, partially defeating the purpose.

Each of these approaches involves either losing accuracy, losing speed, or adding significant implementation complexity. What was needed was a technique that could tame the outlier problem without sacrificing the efficiency benefits of uniform INT8 quantization. Smooth quantization gives exactly this.

The key insight behind smooth quantization is that while activations are hard to quantize (dynamic, contain outliers), weights are easy to quantize (static, relatively uniform). We can mathematically migrate the quantization difficulty from activations to weights through a channel-wise scaling transformation, making the problem tractable.

The intuition is straightforward. If one matrix is "spiky" with outliers and another is "smooth" and well-behaved, we can transfer some of the spikiness from the first to the second. As long as we do this in a mathematically consistent way that preserves the final computation, we have not changed the model's behavior, only redistributed the quantization difficulty to where it is easier to handle. Think of smooth quantization as a seesaw: we load more weight (quantization difficulty) onto the weight side, which can bear it, and unload the activation side, which was struggling under the strain.

The insight that makes this possible is that the product $\mathbf{XW}$ is unchanged if we multiply $\mathbf{X}$ by some diagonal matrix $\mathbf{D}$ and also multiply $\mathbf{W}$ by $\mathbf{D}^{-1}$. These operations are inverses and cancel out. But while they cancel mathematically, they can sharply change the magnitude distributions of both matrices, making each individually much easier to quantize. The question is just how to choose $\mathbf{D}$ to optimally redistribute the difficulty.

Consider a linear layer computing $\mathbf{Y} = \mathbf{X}\mathbf{W}$, where $\mathbf{X} \in \mathbb{R}^{n \times d_{in}}$ is the activation input and $\mathbf{W} \in \mathbb{R}^{d_{in} \times d_{out}}$ is the weight matrix. We introduce a diagonal matrix $\mathbf{S} = \text{diag}(s_1, s_2, \ldots, s_{d_{in}})$ with positive scaling factors, then insert $\mathbf{S}^{-1}\mathbf{S} = \mathbf{I}$ between the two matrices:

where:

• $\mathbf{Y}$: the output of the linear layer, unchanged by this transformation
• $\mathbf{X}$: the input activation tensor with outlier channels
• $\mathbf{W}$: the weight matrix with relatively uniform distribution
• $\mathbf{S}$: the diagonal smoothing matrix containing scaling factors $s_j$ for each input dimension
• $\hat{\mathbf{X}} = \mathbf{X}\mathbf{S}^{-1}$: the smoothed activations, where each column $j$ is divided by $s_j$
• $\hat{\mathbf{W}} = \mathbf{S}\mathbf{W}$: the adjusted weights, where each row $j$ is multiplied by $s_j$

This transformation is mathematically exact since $\mathbf{S}^{-1}\mathbf{S} = \mathbf{I}$. We have not changed the computation, just how we factor it. The key mathematical property we exploit is that multiplying by a matrix and then its inverse produces the identity, so inserting $\mathbf{S}^{-1}\mathbf{S}$ between $\mathbf{X}$ and $\mathbf{W}$ changes nothing about the final result. We are free to group these factors however we like, and we choose to group $\mathbf{S}^{-1}$ with the activations and $\mathbf{S}$ with the weights.

The transformation affects the data as follows: smoothed activations $\hat{\mathbf{X}} = \mathbf{X}\mathbf{S}^{-1}$ divide each column $j$ of $\mathbf{X}$ by $s_j$, which shrinks outlier channels by choosing large $s_j$ values. Meanwhile, adjusted weights $\hat{\mathbf{W}} = \mathbf{S}\mathbf{W}$ multiply each row $j$ of $\mathbf{W}$ by the same $s_j$, increasing the magnitude of those rows to compensate. If channel $j$ has outlier activations, we choose a large $s_j$. This shrinks the activations (making them easier to quantize) while enlarging the corresponding weights (which are already easy to quantize). We are transferring the outlier problem to a domain where it is more manageable.

How do we pick the $s_j$ values? If we make $s_j$ too large, we might shrink the activations so much that we lose precision in other ways, or we might inflate the weights to the point where they become hard to quantize. The optimal choice balances these competing concerns, and the SmoothQuant paper proposes a principled way to reach this balance.

The SmoothQuant paper proposes balancing the quantization difficulty between activations and weights using a migration strength parameter $\alpha \in [0, 1]$:

where:

• $s_j$: the smoothing factor for the $j$-th input channel
• $\max(|\mathbf{X}_{:,j}|)$: the maximum absolute value in the $j$-th column of activations, measured across a calibration dataset
• $\max(|\mathbf{W}_{j,:}|)$: the maximum absolute value in the $j$-th row of weights
• $\alpha$: the migration strength hyperparameter controlling how much difficulty shifts from activations to weights

This formula balances two competing goals. When $\alpha = 1$, we have $s_j = \max(|\mathbf{X}_{:,j}|)$, which fully smooths the activations by dividing each channel by its maximum. This makes the activations perfectly balanced but potentially makes weights hard to quantize. When $\alpha = 0$, we have $s_j = 1/\max(|\mathbf{W}_{j,:}|)$, which leaves activations unchanged. The intermediate value $\alpha = 0.5$ gives the geometric mean, which ensures that after transformation, the maximum activation and maximum weight for each channel have roughly equal magnitude, spreading the quantization difficulty evenly.

To understand this formula intuitively, consider what "equal difficulty" means. After applying the transformation, the smoothed activation maximum for channel $j$ is $\max(|\mathbf{X}_{:,j}|)/s_j$, and the adjusted weight maximum is $\max(|\mathbf{W}_{j,:}|) \cdot s_j$. For these to be equal:

Solving for $s_j$:

This is exactly the $\alpha = 0.5$ case of the SmoothQuant formula, confirming that $\alpha = 0.5$ is the "equal difficulty" point. Values above 0.5 shift more difficulty to the weights, values below 0.5 leave more with the activations.

In practice, $\alpha = 0.5$ works well for most models, achieving roughly equal quantization ranges for both activations and weights after smoothing. However, some models may benefit from different values. Models with particularly severe activation outliers might use $\alpha > 0.5$ to more aggressively smooth the activations, while models with already-difficult-to-quantize weights might use $\alpha < 0.5$ to avoid making the weights worse.

A necessary practical advantage of smooth quantization is that the transformation can be computed entirely offline during a calibration phase. This means inference has no additional runtime cost compared to standard quantized inference. The calibration procedure works as follows:

First, run a small calibration dataset (typically a few hundred representative samples) through the model to collect activation statistics. During this calibration forward pass, we record the maximum absolute value seen in each channel across all calibration samples. Second, compute $\max(|\mathbf{X}_{:,j}|)$ for each channel using these statistics, capturing how large the outliers tend to be. Third, calculate smoothing factors $s_j$ for each layer using the formula above. Fourth, apply the weight transformation $\hat{\mathbf{W}} = \mathbf{S}\mathbf{W}$ and store the modified weights. Fifth, and most cleverly, fuse $\mathbf{S}^{-1}$ into the preceding layer's weights or normalization parameters.

Step 5 is important and deserves careful explanation. Rather than explicitly dividing activations by $\mathbf{S}$ at runtime, we absorb this division into the bias of the previous layer or the scale/shift parameters of the preceding LayerNorm. This means inference has zero additional cost compared to standard quantized inference. The activation values arrive at the quantization step already "smoothed" because the previous layer was modified to produce smoothed outputs.

The fusion works because transformer architectures typically have a LayerNorm immediately before each linear layer. LayerNorm applies a learned scale ($\gamma$) and shift ($\beta$) to each channel. We can incorporate the smoothing factor directly into these existing parameters. If the original LayerNorm scale for channel $j$ was $\gamma_j$ and shift was $\beta_j$, we replace them with $\gamma_j/s_j$ and $\beta_j/s_j$. The activations coming out of the modified LayerNorm are already "smoothed," and we never perform the division at runtime. This is mathematically equivalent to explicitly dividing at every forward pass but requires zero additional computation.

The offline nature of smooth quantization also means it is compatible with any existing INT8 quantization infrastructure. You do not need to modify the inference engine or write new kernels. Simply transform the weights and LayerNorm parameters before quantizing, then proceed with standard absmax or per-channel quantization. The resulting model can be deployed on any hardware that supports INT8 matrix multiplication.

Understanding how quantized operations execute helps clarify why INT8 gives speedup. The key mechanism is that we perform the heavy matrix multiplication entirely in integer arithmetic, then apply scale factors to recover floating-point values for the final output. This two-stage approach exploits the fact that integer operations are cheaper than floating-point operations, especially for the large matrix multiplications that dominate transformer inference.

When both weights and activations are in INT8, the matrix multiplication proceeds as follows. We multiply the integer tensors together, accumulating into 32-bit integers to prevent overflow, and then convert and rescale the result:

where:

• $\mathbf{Y}_{\text{int32}}$: the intermediate result matrix, accumulated in 32-bit integers
• $\mathbf{X}_{\text{int8}}$: the quantized input activations in 8-bit integer format
• $\mathbf{W}_{\text{int8}}$: the quantized weights in 8-bit integer format

The integer multiplication produces INT32 results to avoid overflow from accumulating many INT8 products. Each INT8 times INT8 multiplication produces a result up to $127 \times 127 = 16129$, which fits in INT16. However, a matrix multiplication accumulates thousands or millions of such products, and these sums can easily exceed the INT16 range. INT32 gives enough headroom to accumulate the typical number of products found in neural network layers without overflow. The hardware handles this automatically: dedicated INT8 multiply-accumulate (MAC) units produce 32-bit partial sums that can be safely added together.

After the integer matrix multiplication, we convert back to floating-point and apply the scale factors to recover approximate floating-point outputs:

where:

• $\mathbf{Y}_{\text{fp}}$: the final floating-point output tensor
• $s_X$: the quantization scale factor for the activations (scalar for per-tensor, vector for per-channel)
• $s_W$: the quantization scale factor for the weights (scalar or per-channel vector)
• $\mathbf{Y}_{\text{int32}}$: the integer output from the matrix multiplication, converted to float before scaling

The scale factors $s_X$ and $s_W$ are multiplied together before being applied, since the quantization error from both sources combines multiplicatively. Since scale factors are scalar (or per-channel), this final rescaling is cheap compared to the matrix multiplication itself. The computational cost of the rescaling step is $O(n \cdot m)$ for an output matrix of size $n \times m$, while the matrix multiplication itself is $O(n \cdot m \cdot k)$. Since $k$ is typically large (thousands of features), the rescaling overhead is less than 0.1% of the total computation.

Hardware INT8 units execute this pattern efficiently because of several compounding advantages. INT8 multiply-accumulate operations use simpler circuits than floating-point, with no need to handle exponent alignment, mantissa normalization, or special cases like NaN and infinity. Smaller operands allow higher parallelism in the same silicon area: the same chip real estate that fits one FP32 multiplier can fit four INT8 multipliers. Memory bandwidth is halved compared to FP16, reducing the primary bottleneck for large models where the GPU must stream weights from memory for every token.

Let's implement INT8 quantization from scratch to solidify these concepts. The implementation will proceed from basic absmax quantization through per-channel quantization, then show the outlier problem and show how smooth quantization resolves it.

We'll start with the basic absmax quantization and dequantization operations. The function returns both the integer tensor and the scale factor, since we need both to reconstruct the original values:

The maximum error is bounded by half the quantization step size ($s/2$), as values are rounded to the nearest integer. For this tensor with a scale around 0.01, our maximum possible error is about 0.005. The element with the largest magnitude (which determined the absmax) reconstructs with zero error, as expected from the theory.

Let's visualize how quantization error varies across different value ranges. This reveals the characteristic "sawtooth" pattern that emerges from uniform rounding:

The error follows a uniform distribution between $-s/2$ and $+s/2$, exactly what we expect from rounding to the nearest integer on a uniform grid. This uniformity has an important implication: the expected value of the quantization error is zero, meaning quantization introduces noise but not systematic bias. This property helps explain why quantized models often perform nearly as well as full-precision models despite the apparent information loss.

Now let's implement per-channel quantization for weight matrices. This extends the basic absmax approach by computing a separate scale for each output channel:

Let's simulate the outlier pattern seen in large language models. We inject a single high-magnitude channel into an otherwise normal activation tensor and observe the damage to quantization quality:

Now let's implement the smooth quantization transformation. The core logic fits in two functions: one that computes the smoothing scales from statistics, and one that applies the transformation to the tensors:

Let's put everything together into a complete quantized linear layer. This class combines per-channel weight quantization with optional smooth quantization to handle outlier activations:

The key parameters for the INT8 quantization implementation are:

• alpha: The migration strength in smooth quantization, typically 0.5. Controls how much quantization difficulty is shifted from activations to weights. Higher values (closer to 1.0) shift more difficulty to the weights; lower values leave more with the activations.
• scale: The step size of the quantization grid. Determined by the maximum absolute value (absmax) in the tensor or channel. Smaller scales give finer precision but only stand for values in a smaller range.
• smooth_scales: The channel-wise scaling factors derived from calibration data to balance activation and weight magnitudes. These are computed once and stored as part of the model.
• Calibration dataset size: Typically 256 to 512 samples suffice for collecting reliable activation statistics. Too few samples may miss worst-case outlier channels.

The accuracy impact of INT8 quantization varies materially between model architectures and parameter counts. It also depends on the task. Understanding these patterns helps you set realistic expectations and choose the right approach for your deployment scenario.

Models under 1 billion parameters typically quantize to INT8 with minimal degradation, usually less than 1% accuracy loss on standard benchmarks. The weight distributions are well-behaved, and activation outliers are rare or mild. For these models, simple per-channel weight quantization with per-tensor activation quantization usually suffices without any smoothing.

Models from 1 billion to 10 billion parameters show increasing sensitivity. Without techniques like smooth quantization, accuracy can degrade 2-5% on certain tasks. Perplexity increases are noticeable but often acceptable for deployment. At this scale, smooth quantization or careful calibration-based clipping is typically necessary to maintain acceptable quality.

Models above 10 billion parameters exhibit the emergent outlier features described earlier, making naive INT8 quantization fail noticeably. Smooth quantization or similar techniques become needed at this scale, and even with best practices, a small accuracy gap compared to FP16 usually remains. The LLM.int8() technique, which keeps outlier channels in FP16 while quantizing the rest to INT8, was the original solution to this problem and remains useful when smooth quantization is insufficient.

Task sensitivity also matters. Simple classification tasks are reliable to quantization because the decision boundary does not require high numerical precision. Open-ended generation shows more degradation as errors compound across hundreds or thousands of autoregressive steps. Mathematical reasoning and code generation are particularly sensitive because small numerical errors can cascade into completely wrong answers.

The calibration dataset is an often-overlooked factor in INT8 accuracy. The statistics used to set scale factors and smooth quantization parameters must come from data representative of the deployment distribution. A model calibrated on English text may produce suboptimal quantization parameters for code completion or multilingual inputs. Choosing calibration data carefully, and potentially using task-specific calibration, can materially improve quantized model quality.

INT8 quantization is a practical and widely-deployed optimization, but it comes with important limitations that you should understand before deploying in production. The limitations are not reasons to avoid quantization, but they are constraints that must be accounted for in system design.

The most basic limitation is that some model information is permanently lost during quantization. While smooth quantization and careful calibration minimize this loss, they cannot eliminate it entirely. For safety-necessary applications or tasks requiring the highest possible accuracy, the small degradation from INT8 may be unacceptable. This is why many production systems maintain FP16 or FP32 models for evaluation and fine-tuning while deploying INT8 for inference. The irreversibility is by design: if we could recover the original values losslessly from INT8, we would not have achieved any compression. The compression and the information loss are two sides of the same coin.

Calibration data sensitivity presents another practical challenge. Smooth quantization computes channel-wise statistics from a calibration dataset, and if this data does not stand for the actual deployment distribution, the smoothing factors may be suboptimal. A model calibrated on English text may perform poorly when quantized for code completion or multilingual inputs. In production, this means you need calibration pipelines that are maintained alongside the model and updated when the deployment distribution shifts. It also means that models with broad deployment contexts (such as general-purpose chatbots serving diverse user queries) may need larger and more diverse calibration sets to reach reliable quantization quality.

Hardware support, while widespread, is not universal. INT8 acceleration requires specific hardware features such as NVIDIA's Tensor Cores or ARM's NEON instructions, and the speedup varies by platform and use case. On CPUs without INT8 vectorization, quantized inference may be slower than FP32 due to the quantization and dequantization overhead. On GPUs, the speedup is most pronounced for large matrix multiplications (typical of large-batch inference) but may be minimal for small-batch or latency-sensitive inference where compute is not the bottleneck. Always benchmark on your target hardware before committing to a quantization strategy.

Attention layers present a special challenge that pure weight quantization does not solve. The key-value cache that accumulates during autoregressive generation stores activation values, not weights. As sequences grow longer, the KV cache grows proportionally and can become the dominant memory consumer. Quantizing the KV cache requires activation quantization, which is harder than weight quantization because activation distributions vary with the input. Dedicated techniques for KV cache quantization are an active area of development and are addressed in later chapters.

Finally, INT8 may not give sufficient compression for edge deployment scenarios. With 8 bits per weight, a 7B parameter model still requires 7 GB of storage, which exceeds the capabilities of most mobile devices and many edge accelerators. For mobile or embedded applications, the more aggressive INT4 quantization techniques we will explore in the next chapter become necessary, accepting greater accuracy trade-offs in exchange for smaller model sizes that fit in constrained hardware.

INT8 quantization gives a practical path to halving model memory and accelerating inference through hardware INT8 units. The full picture requires understanding several interconnected concepts, each building on the others:

• Absmax quantization maps floating-point values to INT8 using a single scale factor derived from the maximum absolute value, centering the quantization grid at zero to preserve exact zeros
• Asymmetric quantization introduces a zero-point offset to shift the grid for data distributions that are not centered at zero, improving efficiency for activations like ReLU outputs
• Per-channel quantization applies separate scales to each output channel, sharply reducing error when channels have different magnitude distributions while adding negligible storage overhead
• The outlier problem emerges in large language models where a small fraction of activation dimensions have values 10-100x larger than typical, making per-tensor quantization fail catastrophically for these models
• Smooth quantization solves the outlier problem by migrating quantization difficulty from activations to weights through a mathematically equivalent transformation, computable entirely offline and fusible into existing LayerNorm parameters at zero runtime cost
• The smoothing factor $s_j$ balances activation and weight ranges using a migration strength $\alpha$, with $\alpha = 0.5$ giving a balanced default that works well for most models

INT8 quantization with smooth quantization reaches near-FP16 accuracy on models up to hundreds of billions of parameters while giving 2x memory reduction and significant throughput improvements on compatible hardware. The techniques described here form the foundation for the inference infrastructure behind most large language model deployments today. The upcoming chapters on INT4 quantization, GPTQ, and AWQ explore techniques that push quantization to just 4 bits per weight, accepting somewhat more accuracy degradation in exchange for even greater memory efficiency.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about INT8 quantization techniques.