Gradient Accumulation › Accumulation Steps: How Many?

The primary driver for choosing $K$ is the target effective batch size from your training recipe. As discussed in the Large Batch Training chapter, large language models typically train with effective batch sizes ranging from 256 to 4 million tokens per step, depending on the model size and phase of training. If your target effective batch size is fixed by empirical best practices (such as the linear scaling rule for learning rates), then $K$ is determined by:

$$K = \left\lceil \frac{B_{\text{eff}}}{B_{\text{micro}}} \right\rceil$$

where:

• $\lceil \cdot \rceil$: the ceiling function, rounding up to the nearest integer
• $B_{\text{eff}}$: the target effective batch size from your training recipe
• $B_{\text{micro}}$: the maximum micro-batch size that fits in GPU memory

A secondary consideration is the computational overhead of accumulation. Each micro-batch requires a complete forward and backward pass. With $K = 8$ instead of $K = 1$, you perform eight forward-backward passes per optimizer step rather than one. On a single GPU, this overhead is negligible in terms of floating-point operations (you perform exactly the same number of multiplications and additions), but it can affect memory bandwidth utilization and synchronization overhead in distributed settings.

A practical way to find the maximum micro-batch size is to start with a small batch (for instance, 1 or 2 samples) and double it until you hit an out-of-memory error. The last successful size is your $B_{\text{micro}}$. Then set $K = B_{\text{eff}} / B_{\text{micro}}$, rounding up.

Increasing $K$ is not always beneficial beyond a certain point. Once the effective batch size exceeds the critical batch size for a given task, adding more samples per update produces diminishing returns in gradient quality while increasing the number of steps needed to reach the same number of epochs. The Large Batch Training chapter covers this trade-off in detail, including the Chinchilla and GPT-3 effective batch size recipes. The key point here is that gradient accumulation is a tool for reaching a target effective batch size, not an excuse to make that target arbitrarily large.

A useful mental model is that gradient quality saturates. With very few samples (say $B_{\text{micro}} = 1$), the gradient from a single sample can point in nearly any direction because it reflects only one data point's contribution. As batch size grows, the gradient direction becomes more stable and more representative of the true loss landscape. But beyond a certain size, adding samples barely changes the direction. Gradient accumulation lets you climb out of the noisy, small-batch regime without requiring hardware that can hold the entire logical batch simultaneously.

To see this saturation quantitatively, consider the variance of the stochastic gradient estimator. If each sample contributes a gradient with variance $\sigma^2$, the variance of the mean over $B$ independent samples is $\sigma^2 / B$. Doubling $B$ from 64 to 128 halves the variance, a clear benefit. Doubling $B$ from 8192 to 16384 again halves the variance, but the remaining variance is already small. The signal-to-noise ratio of the gradient direction improves much faster at small batch sizes than at large ones. The concept of the critical batch size formalizes where this diminishing return becomes severe enough to matter in practice.