GPU Architecture › Limitations and Practical Implications

GPU hardware delivers remarkable performance for deep learning, but several practical constraints shape how models are trained and deployed.

Memory capacity is the binding constraint for training large models. The gap between what fits on a single GPU and what state-of-the-art models require has grown wider with each generation of models. A single H100 with 80 GB cannot hold GPT-4-class models even for inference, let alone training. This has driven the development of distributed training strategies: tensor parallelism (splitting individual weight matrices across GPUs, requiring GPU-to-GPU communication after every matrix multiplication), pipeline parallelism (assigning different layers to different GPUs, requiring inter-layer activation transfers), and data parallelism (running multiple replicas of the full model, requiring gradient synchronization). Each strategy trades communication overhead for memory capacity, and the NVLink bandwidth specification becomes as important as compute throughput when selecting hardware for large-scale training. The chapters on distributed training cover these strategies in detail.

The memory bandwidth ceiling affects more operations than most practitioners realize. The dominance of Tensor Cores for matrix multiplication has created a situation where compute capacity for matrix operations vastly exceeds what memory can feed for non-matrix operations. Normalization layers, activation functions, embedding lookups, and attention score computation for long sequences are all memory-bound on current hardware. Kernel fusion, which combines multiple memory-bound operations into a single GPU kernel that reads data once and applies all transformations before writing back, is one of the primary techniques for recovering efficiency. Libraries like FlashAttention, Unsloth, and Triton-based kernel implementations are largely applications of this principle. The arithmetic intensity of self-attention with full materialization of the attention matrix is low enough that it is memory-bound even at moderate sequence lengths, which is the primary motivation for FlashAttention's tiling approach.

Precision options are changing rapidly, and the right choice depends on the workload. BF16 has become the standard for pre-training due to its stability and good memory efficiency. FP8 offers another 2x reduction in memory and bandwidth usage but introduces quantization challenges: gradients computed in FP8 have very limited precision, requiring careful per-tensor or per-block scaling factors. INT8 quantization has become standard for inference but requires post-training calibration to set quantization thresholds. Understanding which parts of training require FP32 (optimizer state, loss scaling, gradient accumulation) versus which can safely use lower precision (forward activations, weight storage) is an increasingly important skill for efficient large-scale training.

Power and cooling are increasingly the limiting factor at scale. A single H100 SXM GPU draws up to 700W. A rack of 8 H100 GPUs draws 5.6 kW just from the GPUs themselves, before accounting for networking, servers, and cooling infrastructure. Large training clusters draw tens of megawatts. The practical constraints of power delivery and thermal management often limit cluster density more than the cost of the GPUs themselves. This explains the industry investment in liquid cooling systems and purpose-built AI data centers with power densities that conventional data center infrastructure cannot support. It also explains why total cost of ownership (hardware plus power plus cooling over the lifetime of the equipment) can be 3-5x the hardware purchase price, shifting economic analysis away from simple compute-per-dollar comparisons.

Consumer GPUs present real tradeoffs for research workloads. The RTX 4090 and similar consumer cards deliver impressive single-GPU performance at a fraction of H100 prices. For research on single-GPU fine-tuning, inference optimization, and experiments on models up to approximately 7B parameters, they provide strong value. However, they lack NVLink connectivity, making multi-GPU scaling less efficient. Their FP64 throughput is dramatically reduced (1/64 of FP32 on the RTX 4090, compared to 1/2 on the A100) and is irrelevant for most training but matters for some scientific computing. Their 24 GB of memory is sufficient for inference on quantized 13B models but limits training to small models or requires aggressive gradient checkpointing and small batch sizes. For researchers running large-scale experiments or training models from scratch, the features and memory capacity of datacenter GPUs quickly justify their cost.