Memory Management › Limitations and Practical Considerations

Memory estimation formulas are approximations, not guarantees. Real GPU memory usage includes framework overhead, CUDA kernel buffers, temporary tensors created during forward pass operations, and memory fragmentation that prevents large contiguous allocations even when total free memory appears sufficient. The estimator built here provides a useful floor estimate; actual usage often runs 10-20% higher due to these factors. When planning a training run, always target 80-85% GPU utilization in your memory budget rather than 100%, to leave room for fragmentation and framework overhead.

The interaction between memory and compute efficiency is also important to understand. Many memory reduction techniques trade memory for compute. Gradient checkpointing adds 33% compute overhead. CPU offloading adds PCIe bandwidth latency. Mixed precision improves both memory and compute throughput on hardware with tensor core support, making it the only "free" memory reduction. Understanding these tradeoffs lets you make rational choices: if your training is already compute-bound (GPU utilization near 100%), adding gradient checkpointing will slow training noticeably. If you are memory-bound and the GPU is sitting idle between steps, the checkpointing compute overhead costs you nothing.

Memory management in practice also interacts heavily with the distributed training strategies covered in subsequent chapters. Tensor parallelism splits individual weight matrices across GPUs, reducing per-device parameter memory but requiring high-bandwidth communication for every matrix multiplication. Pipeline parallelism assigns different layers to different GPUs, reducing parameter memory per device but requiring careful management of activation buffers at pipeline stage boundaries. Fully Sharded Data Parallelism (FSDP) shards parameters, gradients, and optimizer states across all GPUs, making the effective per-device memory proportional to $1/N$ of the full model, where $N$ is the GPU count.

The "14x rule" for static memory deserves a practical caveat: it assumes the full training stack is running. Fine-tuning with LoRA or other parameter-efficient methods dramatically changes the breakdown. In LoRA fine-tuning, only the adapter parameters receive gradients and optimizer states, while the base model parameters are frozen. For a 7B model fine-tuned with LoRA at rank 16, the adapter might have fewer than 10M trainable parameters out of 7B total. The gradient and optimizer state costs apply only to those 10M parameters, while the base model parameters require only inference memory ($2P$ bytes for FP16). This can reduce the effective multiplier from 14x to something closer to 2-3x of the base model size, making fine-tuning of large models feasible on single GPUs that could not possibly run full training.

Understanding these tradeoffs is not merely academic. Production training runs at scale involve budgeting across hardware configurations, choosing between larger batches with gradient checkpointing versus smaller batches without it, and deciding which distributed strategy minimizes communication overhead while keeping per-device memory within budget. The memory breakdown covered in this chapter is the foundation for all of those decisions.