GPU Architecture › Summary

GPU architecture enables modern deep learning through massive parallelism and specialized hardware. The hardware constraints and capabilities you have seen in this chapter directly motivate the training techniques covered in subsequent chapters.

Key takeaways from this chapter:

• The GPU memory hierarchy moves from fast, small registers through shared memory and L1 cache to L2 cache and finally to global HBM memory. Bandwidth decreases by roughly one order of magnitude at each level. Effective kernel design maximizes data reuse in the fast on-chip memory levels to avoid bandwidth bottlenecks at the slower off-chip level.

• CUDA cores execute one multiply-add per cycle on a single pair of scalar values. They are organized into streaming multiprocessors, which execute groups of 32 threads (warps) in lockstep under the SIMT model. Warp divergence and low occupancy are the primary sources of underutilization, and both stem from not designing code around the 32-thread warp granularity.

• Tensor Cores accelerate the matrix multiply-accumulate operation $\mathbf{D} = \mathbf{A} \cdot \mathbf{B} + \mathbf{C}$ at the level of 16x16 tiles in a single warp instruction. FP16 and BF16 Tensor Cores on H100 deliver 989 TFLOP/s in dense mode, roughly 15x the FP32 CUDA core rate. Mixed-precision training uses Tensor Cores for the matrix multiplications (in FP16/BF16) while accumulating in FP32 to preserve numerical stability.

• The roofline model provides a principled framework for understanding whether an operation is compute-bound or memory-bound. For H100, the ridge point is approximately 295 FLOP/byte. Large matrix multiplications are compute-bound at roughly 2000 FLOP/byte. Elementwise operations like ReLU are memory-bound at 0.5 FLOP/byte. Kernel fusion is the primary tool for improving efficiency of memory-bound operations.

• Key GPU specifications for training include memory capacity (how large a model fits), memory bandwidth (throughput for memory-bound operations), peak FP16 throughput (throughput for compute-bound matrix operations), NVLink bandwidth (multi-GPU communication speed for tensor parallelism), and TDP (power draw and cooling requirements).

The subsequent chapters on training infrastructure build directly on these hardware foundations. Distributed training strategies are designed around GPU memory limits and inter-GPU communication bandwidth. Mixed-precision training exploits the Tensor Core capabilities described here. Gradient checkpointing addresses the activation memory problem quantified in the code examples. Understanding the hardware makes all of these higher-level decisions legible and gives you the tools to reason about performance from first principles.