Gradient Accumulation › Gradient Accumulation

Modern language models are voracious consumers of memory. A single forward pass through a large transformer can require tens of gigabytes of GPU memory just to store the activations needed for backpropagation, and that figure grows linearly with batch size. Yet the batch size you use during training is not a mere implementation detail. As we explored in the Large Batch Training chapter, the effective batch size directly shapes how gradients estimate the true loss gradient, how well learning rate scaling rules apply, and how quickly the model converges. When your GPU can only fit 4 samples at a time but your training recipe calls for an effective batch size of 512, hardware limits conflict with optimization requirements.

Gradient accumulation resolves this conflict with an elegant observation: the gradient of a loss computed over a large batch is equal to the average of the gradients computed over smaller sub-batches. Instead of requiring all samples to be processed simultaneously, you can process them sequentially and sum up their gradients before applying a single weight update. The effect is mathematically equivalent to using the full large batch, but the memory footprint at any given moment reflects only the small sub-batch currently being processed.

This technique has become a standard component of the training toolkit for large language models. When training GPT-style models on single nodes, when fine-tuning large models with limited GPU memory, or when experimenting on consumer hardware, gradient accumulation provides the bridge between what your hardware can hold and what your optimizer needs to work effectively.