Mixed Precision Training › Floating Point Formats

Before you can understand mixed precision, you need a clear picture of how computers represent real numbers. The standard format used in most numerical computing is IEEE 754 floating point, a specification that governs how real numbers are approximated using a fixed number of bits. Understanding this representation is not just academic. The failure modes of mixed precision training, from gradient underflow to overflow-induced NaN values, all trace directly back to properties of this format.

IEEE 754 was standardized in 1985, emerging from a decade-long effort to bring consistency to floating point arithmetic across different computer architectures. Before IEEE 754, each hardware vendor used its own representation, making numerical code unreliable and non-portable. The standard specified the bit layout of numbers along with the behavior of arithmetic operations, rounding rules, and special values like infinity and NaN. For over thirty years, virtually every programming language and every CPU conformed to this standard, making FP32 (single precision) the de facto format for scientific computing.

A floating point number is stored as three fields: a sign bit, an exponent, and a fraction (also called the mantissa or significand). The value represented is:

where:

• $\text{sign}$: 0 for positive, 1 for negative
• $\text{exponent}$: raw exponent bits interpreted as an unsigned integer, minus a bias that centers the range
• $\text{bias}$: a constant that shifts the exponent so both very small and very large numbers can be represented
• $\text{fraction}$: the fractional part of the significand; the leading 1 is implied (for normalized numbers)

The bias plays a subtle but role. Without it, the exponent field would represent only non-negative powers of 2, limiting representation to large numbers. By subtracting the bias from the stored exponent, you can represent numbers much smaller than 1. For FP32, the bias is 127, so a stored exponent of 0 represents $2^{0-127} = 2^{-127}$, and a stored exponent of 254 represents $2^{254-127} = 2^{127}$. The stored exponent 255 is reserved for special values (Inf and NaN), and 0 is reserved for subnormals and zero.

The leading-1 convention deserves explicit mention. In the formula above, the fraction contributes values from 0.0 to just under 1.0, and the term $(1 + \text{fraction})$ means the significand always lies in the interval $[1.0, 2.0)$. The leading 1 is implicit: it is not stored in the bits, but assumed. This is called a normalized number. By always storing values in this normalized form, you get one extra bit of precision for free. The special case of a stored exponent of 0 represents subnormal numbers, where the leading 1 is not assumed, allowing very small numbers close to zero to be represented with gradually decreasing precision.

The three standard formats used in deep learning are:

Each format makes a different tradeoff between bit budget and numerical properties. FP32 is the general-purpose workhorse with high precision and wide range. FP16 compresses to 16 bits by narrowing both the exponent and fraction. BF16 also uses 16 bits but makes a different tradeoff: it preserves the full 8-bit exponent from FP32, sacrificing even more fraction bits than FP16 to achieve this.

The exponent bits control the range of representable values, meaning the span from the smallest to the largest representable number. The fraction bits control precision, meaning how finely you can distinguish two numbers that are close together.

For FP32 with 8 exponent bits and a bias of 127, the maximum exponent is 127, so the largest representable value is approximately $2^{127} \approx 3.4 \times 10^{38}$. For FP16 with only 5 exponent bits and a bias of 15, the maximum exponent is 15, giving a maximum value of $2^{15} = 65504$. That is a dramatically narrower range.

The smallest normal FP16 value (the smallest number that can be represented with full precision) is approximately $6.1 \times 10^{-5}$. Values smaller than this are handled by subnormal numbers, which sacrifice precision to extend the range further downward. The smallest subnormal FP16 value is about $5.96 \times 10^{-8}$. Subnormals are important because gradient values during deep learning training often live in this range, and their representation becomes progressively less accurate as values approach zero.

To understand subnormals concretely: when the exponent field contains all zeros, the number is subnormal. The formula changes to $(-1)^{\text{sign}} \times 2^{1-\text{bias}} \times (0 + \text{fraction})$, meaning the leading 1 is dropped and replaced with a leading 0. This allows the representation of values smaller than the smallest normal number, but at reduced precision. A subnormal FP16 value near $10^{-7}$ might have only 3 or 4 bits of true precision rather than the full 10, since the leading bits of the fraction are zeros that carry no information.

Precision is governed by the fraction bits. FP32 provides roughly 7 decimal digits of precision. FP16 provides roughly 3 decimal digits. This means that if you have a value like 0.001234 in FP32 and store it in FP16, it may round to 0.001234 or 0.001235 or some other nearby value, with the rounding error on the order of $10^{-6}$ relative to the magnitude.

This rounding error is called quantization error, and it is the root cause of training instability when using FP16 naively. The gradient values that flow back through a deep network can be extremely small, sometimes well below $10^{-5}$, and they can overflow FP16's narrow range when they are large. Both cases cause training to diverge. The tension between range and precision is precisely what the different 16-bit formats resolve in opposite directions.

Let us trace through a concrete scenario to make these abstractions tangible. Suppose a weight in the embedding layer of a transformer has a value of $w = 0.73211432$ in FP32. When stored in FP16, this rounds to $0.73193359$, an error of about $0.00018$, which is well within FP16's approximately 3 decimal digits of precision near values around 1.

Now consider the gradient for that weight at some training step: $g = 0.00003142$. In FP32, this is represented with full precision. In FP16, the nearest representable value is approximately $0.00003141$, still acceptable. But suppose the learning rate is $10^{-4}$ and the gradient update is $\eta \cdot g = 10^{-4} \times 0.00003142 = 3.142 \times 10^{-9}$. The weight value $w = 0.73$ has a unit in the last place (ULP) in FP16 of roughly $5 \times 10^{-5}$. A gradient update of $3 \times 10^{-9}$ is about 16,000 times smaller than the ULP. In FP16, this update vanishes entirely: the weight does not change. In FP32, the ULP near 0.73 is about $6 \times 10^{-8}$, so the update of $3 \times 10^{-9}$ is small but representable. After enough steps, those tiny updates compound into meaningful weight changes.

This is not a contrived edge case. During the later stages of training, when a model is near convergence, many gradient updates are precisely of this small magnitude. The model has learned the large-scale structure and is fine-tuning minor adjustments. In FP32, these fine adjustments accumulate correctly over thousands of steps. In FP16, they disappear. The model appears to have converged, but it has stalled at a solution that is slightly worse than the FP32 equivalent.

Consider what happens during a gradient update. The optimizer computes a gradient value, multiplies it by the learning rate (often $10^{-4}$ or smaller), and adds the result to a weight parameter. If the weight is around 0.5 and the gradient update is $10^{-7}$, then in FP32 the operation is:

This small difference accumulates over millions of steps and represents a real learning signal. In FP16, the value 0.5 has a unit in the last place (ULP) of approximately $4.8 \times 10^{-5}$. A gradient update of $10^{-7}$ is far smaller than the ULP, so it gets rounded away completely. The weight does not update. This phenomenon is called vanishing updates, and it is one of the main failure modes of FP16 training.

The vanishing update problem is especially insidious because it is silent. When gradients overflow to NaN, training collapses visibly and immediately. When updates vanish, training simply makes no progress in certain parameters. If you are not monitoring gradient magnitudes closely, you may train for thousands of steps without realizing that a large fraction of your model's weights have stopped updating entirely.

Overflow is the other failure mode. Gradients can grow large, especially early in training or during instability. If a gradient value exceeds 65504 (the FP16 maximum), it becomes infinity (Inf) or not-a-number (NaN), and the entire training run collapses. Overflow propagates: a single NaN in one gradient contaminates the entire optimizer step, then the updated weights, then future forward passes. A training run that hits overflow is essentially unrecoverable without reverting to a checkpoint.

The following visualization shows how loss scaling shifts the gradient distribution relative to FP16's representable range. Without scaling, many values fall below the FP16 normal threshold and lose precision; the smaller tail below the subnormal floor becomes zero. A scale factor of $2^{15}$ shifts the distribution by $\log_{10}(32768) \approx 4.5$ decades, placing almost all values in the normal range while exposing a small upper tail to overflow risk.