GPU Architecture › GPU Architecture: Memory Hierarchy, CUDA and Tensor Cores

Every large language model you have encountered in this book was trained on GPUs. GPT-4 required tens of thousands of them running continuously for months. Llama 3 70B was trained on roughly 15,000 H100 GPUs over weeks. Yet GPU hardware is often treated as a black box: you call model.cuda(), the training loop runs faster, and you move on. Understanding what happens inside a GPU turns you from a user of hardware into someone who can reason about training bottlenecks, memory constraints, and performance optimization.

This chapter covers the hardware foundations that make GPU-accelerated deep learning possible. You will learn how the GPU memory hierarchy determines what fits in fast versus slow memory, how CUDA cores perform the floating-point arithmetic that powers every matrix multiplication, how Tensor Cores achieve peak throughput by specializing in matrix operations, and how to read GPU specification sheets to choose hardware and understand performance claims. These concepts apply directly to every training decision covered in later chapters, from batch size selection to mixed-precision training to distributed strategies that span thousands of devices.

The perspective here is deliberately hardware-first. The techniques you have studied throughout this book, self-attention, layer normalization, feed-forward networks, all of them ultimately execute as sequences of operations on GPU compute units. Understanding the hardware illuminates why certain design choices dominate modern LLM architectures: why large matrix multiplications are preferred over branchy code, why activation functions like GELU are implemented as kernel fusions rather than sequential operations, and why sequence length has historically been the most expensive axis to scale.