GPU Architecture › Tensor Cores

CUDA cores perform standard scalar floating-point operations: one multiply or one add per core per cycle. Tensor Cores are an entirely different type of compute unit specifically designed for matrix multiply-accumulate operations. They deliver dramatically higher throughput for this operation by computing an entire small matrix product in a single warp-level instruction, rather than requiring individual scalar operations for each element.

The introduction of Tensor Cores in NVIDIA Volta (2017) was as important to LLM training as the transformer architecture itself. Without Tensor Cores, the compute throughput for BF16 matrix multiplications would be limited to FP32 CUDA core rates. With Tensor Cores, throughput increases by 7-15x, directly enabling the training of models at the scale we see today.

A Tensor Core computes the following operation on small tiles of matrices:

$$\mathbf{D} = \mathbf{A} \cdot \mathbf{B} + \mathbf{C}$$

where:

• $\mathbf{A}$: a small input matrix tile (e.g., $16 \times 16$) in lower precision (FP16 or BF16)
• $\mathbf{B}$: another input matrix tile of shape $16 \times 16$ in lower precision
• $\mathbf{C}$: an accumulator matrix tile of shape $16 \times 16$ in higher precision (FP32)
• $\mathbf{D}$: the output matrix tile, also accumulated and stored in FP32

The Tensor Core computes this fused multiply-add at the level of a warp in a single instruction. On H100, each SM has 4 Tensor Core units, and they collectively deliver 494 TFLOP/s per SM for dense FP16/BF16 operations (with sparsity acceleration reaching 989 TFLOP/s). Compare this to the 67 TFLOP/s for FP32 CUDA cores across the entire chip. This roughly 7x speedup comes from hardware specialization: instead of flexible scalar arithmetic units, Tensor Cores are fixed-function circuits that implement the matrix multiply-accumulate fused operation in far fewer clock cycles than scalar cores would require.

The operation $\mathbf{D} = \mathbf{A} \cdot \mathbf{B} + \mathbf{C}$ has a specific structure that Tensor Core hardware exploits. For a $16 \times 16$ tile, computing $\mathbf{A} \cdot \mathbf{B}$ requires $16 \times 16 \times 16 = 4096$ multiply-add operations (each producing 2 FLOPs). The hardware performs all of these simultaneously in a pipelined fashion. The key insight is that this specific operation (small-tile matrix multiply-accumulate) is so common in deep learning that dedicating special hardware to it is extraordinarily cost-effective.

Large matrix multiplications (like multiplying a $4096 \times 4096$ weight matrix against a batch of activations) are computed by tiling the large operation into many small $16 \times 16$ Tensor Core operations, which are then accumulated. cuBLAS and PyTorch's autograd engine handle this tiling automatically.

The accumulation in FP32 while computing in FP16 or BF16 is central to mixed-precision training stability. Let us understand why this matters.

FP16 has a 5-bit exponent and 10-bit mantissa, covering values from roughly $6 \times 10^{-8}$ to $65504$. FP32 has an 8-bit exponent and 23-bit mantissa, covering values from roughly $1.2 \times 10^{-38}$ to $3.4 \times 10^{38}$. When accumulating thousands of products during a dot product, rounding errors accumulate. If each product is slightly off and the accumulation happens in FP16, rounding errors compound and can corrupt the result. Accumulating in FP32 gives 13 additional bits of mantissa precision for the running sum, making the accumulated result accurate even when each individual product is computed in FP16.

BF16 (Brain Float 16) uses a different bit layout: an 8-bit exponent and 7-bit mantissa. Its exponent matches FP32 exactly, giving it the same dynamic range. The trade-off is reduced mantissa precision (7 bits versus 23 for FP32). For gradient computations, where the scale of gradients varies enormously but their exact mantissa values matter less than their rough magnitude, BF16 is more numerically stable than FP16. FP16 frequently requires loss scaling (multiplying the loss by a large constant to shift gradient magnitudes into the representable range) while BF16 training typically runs without loss scaling at all. This practical advantage has made BF16 the default precision for modern LLM training.

TF32 (TensorFloat-32) is an Ampere innovation that connects FP32 and BF16. It uses FP32's exponent (8 bits) and a 10-bit mantissa (more than BF16's 7 bits), fitting into a 19-bit format internally. TF32 operations accept FP32 inputs, perform the computation in TF32 precision internally on Tensor Cores, and return FP32 outputs. No code changes are required; PyTorch and CUDA libraries automatically use TF32 when operating on FP32 tensors, giving approximately 10x speedup over FP32 CUDA cores with minimal accuracy impact for most training scenarios.

NVIDIA has iterated rapidly on Tensor Core design, adding new precisions and substantially increasing throughput with each generation:

The addition of INT8 in Turing opened inference-time quantization to Tensor Core acceleration. The addition of BF16 in Ampere was the critical training enabler. FP8, introduced in Hopper, is still maturing from a training stability perspective (it requires very careful scaling) but its 2x throughput advantage over FP16/BF16 makes it increasingly attractive for pre-training at massive scale.

H100 Tensor Cores support fine-grained structured sparsity, specifically the 2:4 sparsity pattern where exactly 2 of every 4 consecutive values in a matrix must be zero. When this pattern is satisfied, the hardware can skip zero-multiplicand pairs and effectively double the throughput, explaining the difference between 494 TFLOP/s (dense) and 989 TFLOP/s (sparse with 2:4 pattern) in H100 specifications.

Structured pruning techniques can impose this pattern during or after training while recovering most of the original model accuracy. The process involves identifying the two largest-magnitude weights in each group of 4, zeroing the smaller two, and then fine-tuning to recover accuracy. The resulting model has exactly 50% of its weight parameters set to zero in the required pattern, enabling the full hardware sparsity speedup without any software overhead.