Gradient Accumulation › The Mathematical Foundation

The insight behind gradient accumulation predates deep learning. It is a direct consequence of the linearity of differentiation: the gradient of a sum is the sum of the gradients, and the gradient of a mean is the mean of the gradients. This property means you can split a batch into any number of smaller chunks, compute gradients independently on each chunk, add the results together, and obtain the same answer as if you had processed the entire batch at once.

To see why this works, consider a dataset $\mathcal{D}$ and a model with parameters $\theta$. The standard training objective is to minimize the empirical loss over a batch $B$:

$$\mathcal{L}(\theta; B) = \frac{1}{|B|} \sum_{i \in B} \ell(\theta; x_i, y_i)$$

where:

• $\ell(\theta; x_i, y_i)$: the per-sample loss for input $x_i$ and label $y_i$
• $|B|$: the number of samples in the batch
• $\theta$: the model parameters

The gradient of this loss is:

$$\nabla_\theta \mathcal{L}(\theta; B) = \frac{1}{|B|} \sum_{i \in B} \nabla_\theta \ell(\theta; x_i, y_i)$$

Now suppose we partition $B$ into $K$ non-overlapping micro-batches $B_1, B_2, \ldots, B_K$ of equal size $m = |B|/K$. The gradient of the loss on micro-batch $B_k$ is:

$$\nabla_\theta \mathcal{L}(\theta; B_k) = \frac{1}{m} \sum_{i \in B_k} \nabla_\theta \ell(\theta; x_i, y_i)$$

If we compute the average of these per-micro-batch gradients:

$$\frac{1}{K} \sum_{k=1}^{K} \nabla_\theta \mathcal{L}(\theta; B_k) = \frac{1}{K} \sum_{k=1}^{K} \frac{1}{m} \sum_{i \in B_k} \nabla_\theta \ell(\theta; x_i, y_i)$$

Since the micro-batches partition $B$ and each has size $m = |B|/K$:

$$= \frac{1}{K \cdot m} \sum_{i \in B} \nabla_\theta \ell(\theta; x_i, y_i) = \frac{1}{|B|} \sum_{i \in B} \nabla_\theta \ell(\theta; x_i, y_i) = \nabla_\theta \mathcal{L}(\theta; B)$$

The average of the micro-batch gradients equals the full-batch gradient exactly. This identity is not an approximation or a statistical trick. It is an algebraic consequence of the distributive property of addition over summation.

Deep learning frameworks make this easy to exploit. In PyTorch, each call to .backward() adds gradients into the parameter's .grad attribute rather than overwriting it. Gradient accumulation is simply using this additive behavior intentionally: you call .backward() multiple times without calling optimizer.zero_grad() between calls, allowing gradients to accumulate. The only thing you need to get right is the scaling factor, which ensures the accumulated sum equals the mean rather than the sum of the micro-batch gradients.