GPU Architecture › GPU Memory Hierarchy

Registers are the fastest storage on a GPU. Each CUDA core has its own set of registers, private to the thread executing on it. Register accesses happen in a single clock cycle with no latency penalty whatsoever. Every intermediate value in a computation lives in registers while it is being computed: the partial sum of a dot product, a temporary index variable, the output of an activation function before it is written back to memory, all of these occupy register space during execution.

The catch is capacity. Each streaming multiprocessor (SM, the basic processing unit of a GPU) has a fixed register file, typically 256 KB on modern NVIDIA GPUs. This register budget is shared across all threads running on the SM simultaneously. If a kernel uses many registers per thread, fewer threads can run concurrently, reducing what is called occupancy, which in turn reduces the GPU's ability to hide memory latency by switching between threads.

Register spilling occurs when a kernel needs more registers than are available. The compiler detects this at compile time and moves excess data to local memory, which is physically located in the slow global memory rather than on-chip. Spilling turns single-cycle register accesses into expensive global memory loads that take hundreds of cycles to complete. This is one of the most common sources of unexpected performance degradation in complex kernels. When you profile a kernel and find that its memory traffic is much higher than you expect from the algorithm, register spilling is often the culprit.

The relationship between register use and occupancy is quantitative. Suppose an SM can support 2048 threads and has a 256 KB register file. If each thread uses 32 registers of 4 bytes each, that is 128 bytes per thread, and 2048 threads require 256 KB total, filling the register file exactly at maximum occupancy. If each thread uses 64 registers, only 1024 threads can run, halving occupancy. GPU profiling tools report register usage per kernel, and understanding this number helps you predict occupancy.

One level below registers is the shared memory and L1 cache, which share the same physical SRAM but can be configured to partition the space differently between the two functions. On NVIDIA Ampere and Hopper GPUs, each SM has up to 228 KB of configurable L1/shared memory, plus an additional 32 KB of fixed L1 cache.

Shared memory is explicitly programmer-managed. A CUDA kernel can allocate a portion of shared memory at launch time and use it as a fast scratchpad that all threads within the same thread block can access. This makes shared memory ideal for two things: communication between threads in the same block, and reusing data that multiple threads need. The classic use case is matrix multiplication tiling: a block of threads cooperatively loads a small tile of the input matrices from slow global memory into shared memory. Each thread reads the tile multiple times to compute multiple output elements. The expensive global memory load happens once, but the data is used many times. This is the fundamental principle behind hand-optimized GEMM kernels and libraries like cuBLAS.

L1 cache is hardware-managed and operates transparently. Like a CPU cache, it stores recently accessed global memory data and satisfies future accesses to the same addresses without going back to global memory. Unlike shared memory, you do not program it explicitly. Instead, you design access patterns that are likely to benefit from caching: reading nearby addresses (spatial locality) so that data loaded for one thread is reused by neighboring threads, and reusing the same addresses over time (temporal locality) so that cached data does not get evicted before it is needed again.

The bandwidth difference between shared memory and global memory on the H100 is roughly 10x per SM: approximately 33 TB/s versus 3.35 TB/s. This asymmetry is why kernel authors invest significant effort in tiling strategies. An operation that would otherwise read the same data from global memory multiple times can be redesigned to read it once into shared memory and reuse it in the fast tier.

The L2 cache sits between L1 and global memory and is shared across all SMs on the chip. This chip-level sharing makes L2 useful in a different way from L1. When multiple SMs happen to need the same data, like a weight matrix that is broadcast across a batch, L2 can serve all of them without each SM independently fetching from global memory. On the H100, the L2 is 50 MB, large enough to hold a meaningful fraction of the weights for small models. The H100 L2 bandwidth is approximately 12 TB/s total across the chip.

L2 acts as the first line of defense against repeated global memory accesses that cannot be optimized through shared memory. Even without explicit tiling strategies in the kernel code, hardware-managed L2 caching provides automatic benefit for workloads with moderate data reuse. The key limitation is that 50 MB, while substantial, is dwarfed by the billions of parameters in a modern LLM. Most training workloads see very low L2 hit rates for weight accesses because the weights are too large to fit and access patterns are too scattered.

Global memory is what practitioners mean when they say "GPU memory" in everyday conversation. On modern training GPUs, global memory uses High Bandwidth Memory (HBM), a 3D-stacked DRAM technology that achieves far higher bandwidth than conventional GDDR memory by stacking multiple memory dies vertically and connecting them with thousands of short, wide interconnects.

The H100 SXM has 80 GB of HBM3 with 3.35 TB/s bandwidth. The A100 SXM has 80 GB of HBM2e with 2 TB/s bandwidth. Compare this to a typical CPU's DDR5 RAM at roughly 100 GB/s: GPU global memory is 20-30x faster for sequential reads. This impressive bandwidth is what makes operations like attention over moderate sequence lengths feasible even though they access the full $n \times n$ attention matrix.

Despite this impressive bandwidth, global memory is still the slowest level in the hierarchy by a large margin. The ratio of compute throughput to memory bandwidth determines whether a kernel is compute-bound or memory-bound. When compute throughput is much higher than what memory can feed, the GPU cores sit idle waiting for data. This ratio is called arithmetic intensity, and it is a central concept in GPU performance analysis.

Global memory capacity determines which models fit on a single GPU. A parameter in FP32 occupies 4 bytes; in BF16, 2 bytes. But training requires far more memory than just the model parameters. The memory budget for training includes four distinct components:

• Parameters: 2 bytes per parameter in BF16, or 4 bytes in FP32
• Gradients: Same size as the parameters; each parameter accumulates a gradient during backpropagation
• Optimizer state: Adam stores two additional tensors per parameter (the first-moment estimate $m$ and the second-moment estimate $v$), each in FP32, totaling 8 bytes per parameter
• Activations: The outputs of each layer stored for use during the backward pass; size depends on batch size, sequence length, and model dimensions

For a 7 billion parameter model trained with Adam in mixed precision (parameters and gradients in BF16, optimizer state in FP32), the minimum memory requirement for weights, gradients, and optimizer state is:

$$\text{Memory}_{\text{model}} \approx N_{\text{params}} \times (2 + 2 + 8) \text{ bytes}$$ $$\text{Memory}_{\text{model}} \approx 7 \times 10^9 \times 12 = 84 \text{ GB}$$

where the 2-byte term covers BF16 parameters, the second 2-byte term covers BF16 gradients, and the 8-byte term covers FP32 Adam state. This already exceeds the 80 GB capacity of a single H100, before adding any activations. Including activations for a batch of sequences makes the situation considerably worse. This memory pressure is the primary reason that distributed training exists: not to train faster, but to train at all.

For inference, the picture is more favorable. You need only the parameters, not gradients or optimizer state. A 7B BF16 model requires 14 GB for inference, fitting comfortably on a single H100. This is why inference can sometimes run on much smaller hardware than training requires.