Backpropagation › Backpropagation Through Time: Gradient Accumulation

When a variable participates in multiple computations, its gradient accumulates. This is the gradient accumulation rule: the gradient of the loss with respect to any variable equals the sum of gradients from all downstream uses of that variable.

Consider a weight $w$ used in two computations: $z_1 = w \cdot x_1$ and $z_2 = w \cdot x_2$, with loss $\mathcal{L} = f(z_1, z_2)$. The total gradient is:

This addition arises directly from the multivariate chain rule: $w$ has two "paths" to the loss, and the total effect is the sum of both paths' contributions.

This principle applies beyond mini-batch training. It is also the key concept behind backpropagation through time (BPTT) in recurrent neural networks, where the same weight matrix is used at each time step. The gradient of the loss with respect to the shared weight is the sum of gradients from every time step where that weight was used. This is why BPTT in deep recurrent networks also suffers from vanishing gradients: the gradient from a token 50 time steps ago must flow through 50 weight matrices, and the product of their individual contributions tends to vanish. LSTMs and GRUs were specifically designed to mitigate this by adding gating mechanisms that create shorter gradient paths.

In practice, deep learning frameworks handle gradient accumulation automatically. PyTorch's .grad attribute accumulates gradients from all backward passes through a tensor, which is why optimizer.zero_grad() must be called before each training step. Failing to zero gradients leads to gradients from multiple batches summing together, a common source of bugs that produces unexpectedly fast apparent convergence followed by instability.