GPU Architecture › GPU Specifications and Reading Spec Sheets

When choosing hardware for training or evaluating a new GPU model, several key specifications determine suitability. Understanding what each number means lets you cut through marketing language and make informed hardware decisions.

Memory capacity (GB) sets a hard limit on model size. For training with Adam in mixed precision, you need approximately 12-16 bytes per parameter for weights, gradients, and optimizer state, plus activation memory that scales with batch size and sequence length. For BF16 inference, you need 2 bytes per parameter. These numbers directly determine whether a model fits on one GPU or requires multi-GPU configurations.

Memory bandwidth (TB/s) determines the throughput of memory-bound operations. Elementwise activations, layer normalization, embedding lookups, and softmax are all memory-bound: the GPU can process the data faster than it can load it from global memory. For these operations, doubling memory bandwidth halves execution time. This explains why the H100's 3.35 TB/s bandwidth, 67% higher than the A100's 2.0 TB/s, is particularly valuable for inference workloads that are dominated by memory-bound operations with small batch sizes.

Peak FP16/BF16 throughput (TFLOP/s) represents the maximum floating-point operations per second using Tensor Cores. This is the dominant metric for training throughput. Always compare dense figures (without structured sparsity) when making like-for-like comparisons between GPUs, since sparse throughput requires specific weight patterns that may not apply to your workload.

Peak FP32 throughput (TFLOP/s) matters for operations that require FP32: optimizer updates in Adam (first and second moment estimates), gradient accumulation across microbatches, and certain normalization operations. In mixed-precision training, roughly half the compute time involves FP16/BF16 matrix multiplications and half involves FP32 overhead operations.

NVLink bandwidth (GB/s) governs GPU-to-GPU communication in multi-GPU configurations. NVLink connects GPUs directly without routing through the CPU, enabling bandwidths that are 5-10x higher than PCIe. The H100 provides 900 GB/s bidirectional NVLink bandwidth per GPU. High NVLink bandwidth is critical for tensor parallelism, where model layers are split across GPUs and intermediate activations must be communicated after every operation. For data parallelism with gradient synchronization, lower inter-GPU bandwidth is more tolerable since communication occurs less frequently.

TDP (Thermal Design Power, Watts) sets the maximum sustained power draw. A single rack of 8 H100 SXM GPUs draws approximately 5.6 kW from the GPUs alone, plus another 2-3 kW for networking and compute overhead. Large training clusters draw megawatts. TDP determines the total cost of ownership beyond hardware price, including power infrastructure, cooling systems, and electricity costs. For on-premises deployments, power delivery constraints often limit GPU density before space constraints do.

The roofline model provides a principled framework for understanding whether a given operation is compute-bound or memory-bound, and what the maximum achievable performance is in each regime. It takes its name from its characteristic shape on a log-log plot: a sloped line rising from the left (the memory-bound regime) and a flat ceiling on the right (the compute-bound regime). The ridge point where the two lines meet represents the minimum arithmetic intensity required to be compute-bound.

For the H100 SXM with FP16 Tensor Cores, the ridge point is:

$$\text{Ridge Point} = \frac{\text{Peak FP16 Throughput}}{\text{Memory Bandwidth}} = \frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295 \text{ FLOP/byte}$$

This means an operation must perform at least 295 floating-point operations per byte it reads from global memory to be compute-bound rather than memory-bound. Large matrix multiplications easily exceed this threshold. Multiplying two $4096 \times 4096$ FP16 matrices reads approximately 64 MB of data but performs $2 \times 4096^3 \approx 137$ billion FLOPs, giving:

$$\text{Arithmetic Intensity}_{\text{MatMul}} = \frac{137 \times 10^9}{64 \times 10^6} \approx 2140 \text{ FLOP/byte}$$

This is 7x above the ridge point: the computation is deeply compute-bound. Adding more memory bandwidth would not speed it up; only faster Tensor Cores would.

Contrast this with an elementwise operation like ReLU applied to a $4096 \times 4096$ FP16 matrix. It reads 32 MB of data and performs about 16 million FLOPs (one comparison and one maximum operation per element), giving:

$$\text{Arithmetic Intensity}_{\text{ReLU}} = \frac{16 \times 10^6}{32 \times 10^6} = 0.5 \text{ FLOP/byte}$$

This is 590x below the ridge point. The operation is entirely memory-bound. The Tensor Cores are irrelevant: ReLU cannot use them since it is not a matrix multiply-accumulate. More Tensor Core throughput would do nothing to speed up ReLU; only higher memory bandwidth would help.

This analysis has direct implications for architecture design. Operations in the memory-bound regime can often be fused together: instead of reading data from HBM, applying ReLU, writing back, then reading again to apply dropout, you can fuse these into a single kernel that reads once and applies all transformations before writing back. This is why kernel fusion libraries like FlashAttention and Triton-based implementations of transformer operations matter so much for real-world training throughput.

The specifications table reveals structural differences between consumer and datacenter GPUs that go beyond the raw numbers.

Consumer GPUs like the RTX 4090 and 3090 are designed for gaming, content creation, and prosumer workloads. They have good FP16 Tensor Core throughput (165 TFLOP/s for the 4090) and modest memory (24 GB of GDDR6X), but they lack NVLink, have reduced FP64 throughput, and use consumer-grade thermal solutions. For single-GPU fine-tuning or inference on models up to about 10 billion parameters, they offer excellent performance per dollar. The RTX 4090 can run 7B models in BF16 and even quantized 13B models with tools like llama.cpp.

Datacenter GPUs (A100, H100, H200) are designed for multi-GPU scale-out. Their higher TDP (700W for H100 versus 450W for the 4090) is accompanied by higher-capacity thermal management, error-correcting memory (ECC), and high-speed NVLink interconnects. ECC memory is important for long training runs: a single bit error in a weight or gradient without correction can corrupt an entire training run, potentially wasting weeks of compute. NVLink enables the tensor parallelism strategies needed to train models that do not fit in a single GPU's memory.

The H200 represents a significant leap in memory capacity: 141 GB of HBM3e with 4.8 TB/s bandwidth. This expanded memory makes it possible to fit larger models on a single GPU, reducing the need for certain multi-GPU communication patterns and simplifying deployment of very large models at inference time.