Gradient Accumulation › Throughput and Wall-Clock Time

Gradient accumulation trades sequential computation for memory efficiency. Understanding the throughput implications helps you decide whether accumulation is the right tool or whether another memory-saving technique would be more efficient for your workload.

In terms of raw floating-point operations, gradient accumulation is exactly as expensive as processing the full effective batch in a single forward-backward pass. The total number of multiply-accumulate operations performed across all $K$ micro-batches is identical to what a single large batch would require. This means that if your GPU were perfectly utilized at every batch size, the wall-clock time per optimizer step would not change at all.

In practice, GPU utilization is not constant across batch sizes. GPUs achieve their peak throughput when the workload is large enough to saturate all compute units and hide memory access latency. Very small micro-batches can leave compute units idle while waiting for data to transfer from global memory, resulting in lower effective throughput. For this reason, processing 8 micro-batches of size 4 can be slower in wall-clock time than processing one batch of size 32, even though the arithmetic work is identical.

The impact depends heavily on the model architecture and the hardware. For large models with high parameter counts relative to batch size, the model weights themselves dominate memory access time, and the overhead from sequential micro-batch processing is small. For smaller models where activations are proportionally larger, the inefficiency is more pronounced.

A useful way to think about accumulation overhead is in terms of kernel launch efficiency. Each forward and backward pass through a model involves hundreds to thousands of individual GPU kernel launches. Each kernel has a fixed overhead for scheduling and dispatch, regardless of the batch size. With $K$ accumulation steps, you incur $K$ times as many kernel launches per optimizer step. For large models where each kernel processes a substantial amount of data, this overhead is negligible. For very small models on fast GPUs, it can become noticeable.

The other throughput consideration is memory allocation and deallocation. Each micro-batch allocates activation tensors in the forward pass and frees them during or after the backward pass. The allocator must satisfy these requests efficiently. PyTorch's CUDA memory allocator uses a caching strategy that largely eliminates allocation overhead after the first accumulation cycle, so this is rarely a bottleneck in practice.