Weight Quantization Basics › The Quantization Mapping

With the quantization formula established, we face a practical question: how do we choose the scale and zero-point? These parameters must be selected carefully because they determine both the range of representable values and the precision within that range. The intuition is straightforward: we want to map the full range of actual tensor values onto the full range of available integers, using every bit of precision the integer format provides.

Think of it as a translation and scaling problem. You have a thermometer calibrated from $-0.5$ to $0.3$ degrees, and you want to map it onto a ruler with integer markings from $-128$ to $127$. If you scale correctly, the coldest temperature on the thermometer maps to $-128$ and the warmest maps to $127$, and every mark on the ruler corresponds to a well-defined temperature. If you scale poorly, the thermometer might only use a portion of the ruler's range, leaving most of the precision wasted.

We derive the quantization parameters by establishing a linear mapping where the real minimum $x_\text{min}$ maps to the integer minimum $q_\text{min}$ and $x_\text{max}$ maps to $q_\text{max}$. This ensures that the smallest and largest values in our tensor precisely hit the boundaries of the integer range, leaving no representable integers unused.

$$\begin{aligned} x_\text{min} &= s(q_\text{min} - z) \\ x_\text{max} &= s(q_\text{max} - z) \end{aligned}$$

where:

• $x_\text{min}, x_\text{max}$: the minimum and maximum values in the real (floating-point) tensor
• $q_\text{min}, q_\text{max}$: the minimum and maximum values in the target integer range (e.g., -128 and 127 for signed 8-bit)
• $s$: the scale factor to be determined
• $z$: the zero-point to be determined

These two equations represent the boundary conditions of our linear mapping. The first equation says that when we dequantize the minimum integer value, we should recover the minimum real value. The second says the same for the maximum. Together, they give us two equations with two unknowns, $s$ and $z$, which we can solve algebraically.

Subtracting the first equation from the second eliminates $z$, which is the key insight that makes this system tractable. The zero-point cancels because we are computing the difference between two dequantization equations, and the shift cancels out in the difference:

$$\begin{aligned} x_\text{max} - x_\text{min} &= s(q_\text{max} - z) - s(q_\text{min} - z) \\ &= s \cdot q_\text{max} - s \cdot z - s \cdot q_\text{min} + s \cdot z && \text{(distribute } s \text{)} \\ &= s \cdot q_\text{max} - s \cdot q_\text{min} && \text{(cancel } s \cdot z \text{)} \\ &= s(q_\text{max} - q_\text{min}) && \text{(factor out } s \text{)} \end{aligned}$$

Solving for $s$ yields the scale factor:

$$s = \frac{x_\text{max} - x_\text{min}}{q_\text{max} - q_\text{min}}$$

where:

• $s$: the scale factor representing the step size (real value per integer increment)
• $x_\text{max} - x_\text{min}$: the total range of the continuous values
• $q_\text{max} - q_\text{min}$: the total number of discrete steps available

This formula has a straightforward interpretation: the scale is simply the ratio of the real range to the integer range. If your real values span 1.0 units and you have 256 integers available, each integer step represents $1.0/256 \approx 0.0039$ real units. The scale tells you exactly how much precision you have: smaller scales mean finer distinctions between values, while larger scales mean coarser quantization. A large scale arises when the real range is wide relative to the integer range, and it forces many real values to map to the same integer.

Substituting $s$ back into the first equation allows us to solve for the zero-point $z$:

$$z = q_\text{min} - \frac{x_\text{min}}{s}$$

where:

• $z$: the zero-point (in real arithmetic before rounding)
• $x_\text{min} / s$: the minimum real value scaled to the integer domain

The zero-point equation determines where real zero falls in the integer range. If the real values are centered around zero, the zero-point will be near the middle of the integer range. If the real values are all positive, the zero-point will be at or near the minimum integer value. Geometrically, the zero-point specifies how many integer steps separate the integer origin from the point where real zero would land.

Since $z$ must be an integer, we round the result:

$$z = q_\text{min} - \text{round}\left(\frac{x_\text{min}}{s}\right)$$

where:

• $s$: the computed scale factor
• $z$: the computed zero-point
• $x_\text{min}$: the minimum value in the floating-point tensor
• $q_\text{min}$: the minimum value of the target integer range

The rounding of the zero-point introduces a small complication: it means that real zero may not map exactly to integer $z$, and the boundary conditions we started with may not be satisfied exactly. In practice, this slight imprecision is negligible compared to the rounding errors inherent in quantization itself. The zero-point rounding shifts the mapping by at most half an integer step, which is the same order as the rounding error affecting every value in the tensor.

For 8-bit unsigned integers, $q_\text{min} = 0$ and $q_\text{max} = 255$, giving $q_\text{max} - q_\text{min} = 255$ discrete steps. For 8-bit signed integers, $q_\text{min} = -128$ and $q_\text{max} = 127$, giving the same 255 steps (note: not 256, because the range of $n$-bit integers contains $2^n$ values but only $2^n - 1$ steps between adjacent values). This means signed and unsigned 8-bit formats have the same quantization resolution; they differ only in where the range is centered.

Every quantization operation introduces error because we are fundamentally discarding information. When we round a continuous value to the nearest integer, any fractional part of the scaled value is lost. Understanding the nature and magnitude of this error helps us make informed decisions about quantization parameters and bit widths.

The rounding operation introduces error that depends on where the original value falls relative to the quantization grid. Think of placing a set of points on a number line and then rounding each point to the nearest tick mark on a ruler. Points that fall very close to a tick mark move only slightly and incur small errors. Points that fall exactly halfway between two tick marks must be rounded to one or the other, incurring the maximum possible error of half a step.

For uniformly distributed values, the maximum error from rounding is $s/2$. This occurs when a value falls exactly halfway between two adjacent quantization levels. On average, the rounding error follows a uniform distribution between $-s/2$ and $s/2$, with mean zero and variance:

$$\sigma^2_\text{round} = \frac{s^2}{12}$$

where:

• $\sigma^2_\text{round}$: the variance of quantization rounding error
• $s$: the scale (quantization step size)

This means smaller scales produce smaller errors but can only represent smaller ranges. There is a direct tradeoff: cutting the scale in half cuts the step size in half, which cuts the variance by a factor of four, but it also halves the representable range. If the true value lies outside the representable range, we must clip it to the nearest boundary value, potentially causing much larger errors for outliers.

The mean squared quantization error for a tensor depends on both the value distribution and the choice of $s$ and $z$. Values near the center of the distribution incur only rounding error, bounded by $s^2/12$. Values at the extremes are handled well when the scale is chosen to just accommodate them. But values beyond the representable range are clipped, and the error for a clipped value can be arbitrarily large. An outlier at $10x_\text{max}$ gets clipped to $x_\text{max}$, incurring an error of $9x_\text{max}$. This observation motivates many of the advanced quantization techniques we will explore in later chapters, which focus on handling outliers gracefully rather than letting them to corrupt the quantization of typical values.

The theoretical relationship between bit width $b$ and quantization error is elegant. Each additional bit doubles the number of representable levels, which halves the step size $s$, which reduces the variance of the rounding error by a factor of four. In other words, quantization error decreases by roughly $6$ dB (a factor of 4 in variance, or 2 in amplitude) for each additional bit. This is why plots of quantization error versus bit width on a log scale show nearly straight lines with a slope of 4x per bit.