Memory Management › Memory Management

Every large language model training run is ultimately constrained by one resource before compute, before data, even before time: GPU memory. A model that fits in memory trains; one that does not, crashes. Understanding exactly what occupies GPU memory, how much each component costs, and how to diagnose failures when things go wrong separates practitioners who can scale models from those who hit walls.

This chapter breaks down the four major memory consumers in transformer training: model parameters, gradients, optimizer states, and activations. For each category, you will learn how much memory it requires, why that amount is unavoidable, and what tradeoffs appear when you try to reduce it. You will then build a working memory estimator that predicts whether a model fits in a given GPU budget before you ever start training, and finally work through systematic OOM (out-of-memory) debugging strategies to recover from crashes when estimation alone is not enough.

The concepts here connect directly to the distributed training strategies covered in the next chapters. Tensor parallelism, pipeline parallelism, and gradient checkpointing all exist primarily to manage memory. Without understanding what they are managing, you cannot reason about when to use them or what tradeoffs they impose.