Gradient Accumulation › Summary

Gradient accumulation splits a logical large batch into $K$ smaller micro-batches, runs forward and backward passes on each without updating weights, and applies a single optimizer step after all micro-batches are processed. The key results are:

• The accumulated gradient equals the full-batch gradient when each micro-batch loss is divided by $K$ before backpropagation.
• Activation memory is bounded by the micro-batch size, not the effective batch size.
• Parameters, gradients, and optimizer states are unaffected by accumulation.
• Gradient clipping must be applied after the full accumulation cycle, not inside the loop.
• Batch normalization statistics are inconsistent across micro-batches; layer normalization is unaffected.
• Distributed training requires the no_sync() pattern to avoid redundant all-reduce calls during accumulation.
• The technique combines naturally with activation checkpointing, mixed precision training, and ZeRO optimizer sharding.
• Throughput is best preserved by using the largest micro-batch size that fits in memory, then setting $K$ to reach the target effective batch size.

The effective batch size $B_{\text{eff}} = K \times B_{\text{micro}}$ is the quantity that governs optimization behavior. Gradient accumulation gives you control over this quantity independently of GPU memory, making it an essential tool for matching your training recipe to your hardware.