Mixed Precision Training › BF16: A Better Design for Deep Learning

BF16 (Brain Float 16) is a 16-bit floating point format developed at Google Brain specifically for deep learning. It uses 1 sign bit, 8 exponent bits, and 7 fraction bits. The key insight: BF16 has the same number of exponent bits as FP32.

Understanding why Google Brain chose to allocate bits this way requires stepping back and asking which property of floating point is most critical for stability during training. The answer, as the FP16 challenges above illustrated, is range, not precision. Overflow to Inf or NaN corrupts training irreversibly. Reduced precision in the gradient, by contrast, simply adds noise to each update, and neural network optimization is already highly tolerant of noisy gradients. The entire rationale for stochastic gradient descent is that you can make progress using noisy, batched estimates of the true gradient. The same tolerance extends to numerical noise from reduced floating point precision. By preserving the FP32 exponent, BF16 eliminates the most dangerous failure mode while accepting a modest precision reduction.

The origins of BF16 trace to Google's TPU development. When designing the second generation of TPUs around 2018, the Google Brain team needed a format that could run efficiently on custom hardware while supporting the large models they were training. FP16's overflow problems made it problematic at the scale of their workloads. Rather than building complex loss scaling infrastructure into the TPU stack, they designed a new format that simply did not have the overflow problem. BF16 was the result, and it was later adopted by NVIDIA for their Ampere GPUs (A100, released in 2020) and beyond.

The 8-bit exponent gives BF16 the same numerical range as FP32: values from approximately $10^{-38}$ to $10^{38}$. Overflow is essentially impossible in BF16 during normal training, because any gradient that would overflow FP32 would overflow BF16 as well, and training in FP32 would also be broken. The fact that FP32 training works for a given model architecture and hyperparameters is a guarantee that BF16 will not encounter overflow problems.

This eliminates the need for loss scaling entirely. You can train in BF16 without tracking overflow, without skipping corrupted steps, and without the overhead of checking gradients for Inf/NaN values. The training loop becomes simpler, the failure modes are fewer, and the behavior is more predictable.

The cost is lower precision in the fraction: 7 bits versus 10 bits in FP16. BF16 provides only about 2-3 decimal digits of precision, compared to FP16's roughly 3. In practice, this reduced precision is rarely a problem for gradient values, because gradients carry directional information more than they carry precise magnitudes. What matters is that the optimizer step moves weights in roughly the right direction, not that it does so with 10 significant bits of accuracy. The noise introduced by 7-bit mantissa precision is similar in character to the noise introduced by using a random mini-batch instead of the full dataset: the optimizer is stochastic anyway, and another source of small noise does not materially change the optimization trajectory.

There is a deeper reason for this tolerance. The gradient at any given step is a stochastic estimate of the true gradient because it is computed on a finite batch of examples. This stochasticity is not a bug; it is essential for escaping sharp minima and finding flat, generalizable solutions. Adding a small amount of quantization noise on top of this existing stochasticity is unlikely to change training dynamics significantly. The quantization noise from BF16's 7-bit mantissa is typically smaller than the variance from batch sampling, so it is "absorbed" into the existing noise floor.

The choice between BF16 and FP16 depends primarily on the hardware and the training stability requirements.

BF16 advantages:

• No loss scaling required, simplifying the training loop
• No overflow risk (same range as FP32)
• More numerically stable with large models and high learning rates
• First-class support on TPUs (BF16 was designed for Google's hardware)
• Simpler debugging: NaN values in BF16 training always indicate an underlying numerical problem rather than an FP16 range issue

FP16 advantages:

• Higher fraction precision (10 bits vs 7 bits)
• Broader hardware support: available on older GPUs (Pascal, Volta) that lack BF16
• Slightly better accuracy for models that are precision-sensitive

Hardware availability:

• BF16 requires Ampere GPUs (A100, A30, RTX 30 series) or newer on NVIDIA hardware
• TPUs support BF16 natively and have done so since the early TPU generations
• Intel's newer Xeon processors and Habana Gaudi accelerators also support BF16
• H100 and H200 GPUs support both BF16 and FP8 (an even more compressed format for inference)

In the transformer training community, BF16 has become the default for large model training on modern hardware. The simplicity of eliminating loss scaling, combined with consistent numerical stability, outweighs the small precision reduction. Major models including PaLM, Gemini, LLaMA, and most recent large-scale training runs use BF16 as their primary training precision.

It is worth briefly noting the direction the field is heading beyond BF16. NVIDIA's H100 GPU introduced support for FP8 (8-bit floating point), which comes in two variants: E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 exponent bits, 2 mantissa bits). FP8 offers another 2x reduction in memory and bandwidth compared to BF16, enabling even larger batch sizes and faster training. FP8 training requires even more careful management of numerical range than FP16, with per-tensor scaling factors rather than a single global loss scale. It is primarily used in production training pipelines at large organizations and is not yet as broadly accessible as BF16. The conceptual structure, however, is the same: trade precision for range where possible, and keep critical accumulators in higher precision.