GPU Architecture › CUDA Cores and the SM Architecture

The computational heart of an NVIDIA GPU is the Streaming Multiprocessor (SM). Understanding the SM architecture explains how a GPU executes thousands of operations in parallel and why certain programming patterns are efficient while others are not.

An SM is a self-contained processing unit with its own compute cores, register file, shared memory, instruction schedulers, and warp dispatchers. A modern GPU has many SMs: the H100 SXM has 132, the A100 has 108, the RTX 4090 has 128. When you launch a CUDA kernel from PyTorch (which happens automatically whenever you call an operation on a CUDA tensor), its thread blocks are distributed across available SMs. Each block runs entirely on one SM, but multiple blocks can run on the same SM simultaneously if resources permit.

Each SM contains multiple types of compute cores, each specialized for a different class of operations:

• CUDA cores (FP32 units): For single-precision floating-point arithmetic. These are the general-purpose workers.
• FP64 units: For double-precision arithmetic. Datacenter GPUs include substantial FP64 capacity; consumer GPUs drastically reduce it since most deep learning does not need it.
• INT32 units: For integer arithmetic, used in index calculations and certain quantized inference workloads.
• Tensor Cores: For mixed-precision matrix multiply-accumulate operations, covered extensively in the next section.
• Special Function Units (SFUs): For transcendental functions like sine, cosine, and reciprocal square root that appear in activation functions and attention computations.

On the H100 SM, there are 128 CUDA cores (FP32 units). Across 132 SMs, this gives 16,896 CUDA cores total. Each core can perform one multiply-add operation (two floating-point operations) per clock cycle. At 1.98 GHz boost clock, the peak FP32 CUDA core throughput is roughly 67 TFLOP/s, a figure that matches NVIDIA's official specification.

The GPU does not execute individual threads independently. Threads are grouped into warps of 32 threads, and all threads in a warp execute the same instruction simultaneously on different data. This execution model is called SIMT: Single Instruction, Multiple Threads.

SIMT is both the source of the GPU's efficiency and its most important constraint. The efficiency comes from instruction fetch and decode overhead being amortized across 32 threads: fetching one instruction and dispatching 32 operations is 32x more efficient than fetching 32 separate instructions for 32 separate CPU cores. The constraint is that all 32 threads in a warp must execute the same instruction at the same time.

SIMT means that branches are handled differently on GPUs than on CPUs. When threads in a warp take different branches (for example, half the threads satisfy an if condition and half do not), the warp serializes: it executes the true branch with the non-participating threads masked off (they do no work but consume time), then executes the false branch with the other threads masked off. This is called warp divergence. In the worst case, when all 32 threads take different paths, throughput drops to 1/32 of maximum.

Well-optimized GPU code minimizes warp divergence by ensuring threads in the same warp follow the same control flow. Transformer inference with padding tokens is one place where divergence can appear: if threads in the same warp process tokens with different padding masks, their execution paths diverge. Careful batching strategies group sequences of similar length to reduce this effect.

A fundamental principle of GPU performance is that latency is hidden through concurrency rather than eliminated through caching. When a warp issues a memory load that takes 400-800 cycles to complete (a typical round trip to global HBM), the SM does not stall. Instead, it immediately switches to another warp that is ready to execute its current instruction. If enough warps are available, the SM never sits idle: while one warp waits for memory, dozens of others continue computing.

Occupancy measures the ratio of active warps to the maximum number of warps an SM can support. The H100 SM can support up to 64 active warps. Higher occupancy generally means better latency hiding because more warps are available to fill the gaps left by stalled warps. However, occupancy is constrained by three resources:

• Register file: More registers per thread means fewer threads fit in the fixed register budget per SM, reducing occupancy
• Shared memory: More shared memory allocated per thread block means fewer blocks can run simultaneously on the SM
• Thread count: Each block can have at most 1024 threads on modern GPUs, and each SM can host multiple blocks simultaneously if resources permit

The relationship between occupancy and performance is not simple. Maximum occupancy does not always mean maximum performance. Once enough warps exist to hide the dominant latency source, adding more warps provides diminishing returns. A kernel with 50% occupancy might achieve 95% of maximum throughput if the arithmetic intensity is high enough that the compute units stay busy. Understanding your kernel's bottleneck (memory latency, memory bandwidth, or compute throughput) determines how aggressively to optimize for occupancy.

A useful mental model: think of warps as workers at a factory with multiple machines. If the machines have long setup times (memory latency), you need many workers so that while one worker waits for a machine to finish setup, others are using different machines. But if the machines run fast (high arithmetic intensity), you only need enough workers to keep all machines busy at any moment. Packing in extra workers beyond that threshold just means they compete for the same limited machines.

When PyTorch launches a CUDA kernel, the entire computation is organized as a grid of thread blocks. Each block contains a fixed number of threads (up to 1024) that share access to the same shared memory and can synchronize with each other. The blocks in the grid execute independently and can be scheduled across SMs in any order.

This two-level structure (grid of blocks, blocks of threads) is not arbitrary. It reflects the two levels of parallelism available in the hardware: SM-level parallelism (multiple SMs running different blocks simultaneously) and intra-SM parallelism (multiple warps within a block running on the SM's compute units). Efficient GPU code exploits both levels.

For matrix multiplication, the canonical thread block structure assigns each block responsibility for computing one tile of the output matrix. Threads within the block cooperate to load tiles of the input matrices into shared memory, then each thread computes one or more elements of the output tile using data from shared memory. This structure naturally maps the tiling algorithm onto the two-level hardware parallelism.