Backpropagation › Common Backpropagation Bugs

Even with a correct conceptual understanding, backprop implementations often have subtle bugs. Here are the most common ones:

Forgetting to zero gradients. In PyTorch, calling loss.backward() accumulates gradients into .grad. If you call backward twice without zeroing, the gradients from both passes sum together. Always call optimizer.zero_grad() before each backward pass. This is perhaps the most common beginner mistake in PyTorch, and it can produce training curves that initially look plausible (the loss decreases) but eventually diverge as the accumulated gradients from many mini-batches grow unboundedly large.

Wrong shapes in matrix multiplications. The gradient of the loss with respect to weights $\mathbf{W}$ has shape equal to $\mathbf{W}$. A common mistake is computing $\frac{\partial \mathcal{L}}{\partial \mathbf{z}} \cdot \mathbf{x}$ (a scalar times vector) instead of the correct outer product $\frac{\partial \mathcal{L}}{\partial \mathbf{z}} \mathbf{x}^T$ (which produces a matrix matching $\mathbf{W}$'s shape). NumPy's broadcasting can silently hide shape errors, returning something that looks plausible but has the wrong semantics.

Missing the activation derivative. When computing the gradient at pre-activation values $\mathbf{z}$, you must multiply by the derivative of the activation function. Missing this step gives gradients for the post-activation values $\mathbf{a}$ instead of $\mathbf{z}$, which is incorrect. This bug is easy to introduce when refactoring code: it is tempting to skip the activation backward when the activation is ReLU (since the derivative is simple), but the elementwise multiply by the ReLU mask must still happen.

Using the same array for input and output. In-place operations can corrupt the stored cache needed for the backward pass. Always use separate arrays for input and output in each layer. A related pitfall is mutating a parameter in the forward pass (e.g., normalizing it) before storing it in the cache, so the backward pass receives the modified value rather than the original.

Summing bias gradient incorrectly in batch mode. With a batch of examples, the bias gradient is the sum over examples (not the mean, not a single example's gradient). This is because each example contributes independently to the loss through the bias. If you use the mean instead of the sum, your bias updates will be underscaled by $1/B$, effectively giving the bias a much smaller effective learning rate than the weights.

Not accounting for broadcasting. When a smaller tensor is broadcast to match a larger one during the forward pass, the backward pass must sum over the broadcast dimensions. PyTorch handles this automatically, but manual implementations often forget it. For example, if you add a bias vector of shape $(m,)$ to an activation matrix of shape $(B, m)$, the forward pass broadcasts the bias across the batch dimension. The bias gradient in the backward pass must then sum over the $B$ dimension to produce a gradient of shape $(m,)$ matching the original bias.

Incorrect handling of numerical stability. Operations like log(softmax(x)) should be computed as log_softmax(x) rather than log(softmax(x)) to avoid the intermediate softmax values going to zero. If you compute softmax first and then take the log, tiny probabilities can underflow to zero, and log(0) produces negative infinity. PyTorch provides F.log_softmax and F.cross_entropy precisely to avoid these numerical issues.