Backpropagation › Summary

Backpropagation is the algorithm that makes training neural networks with gradient descent tractable:

• Computational graphs decompose neural network computations into simple operations, each with a known local gradient. The graph structure makes the chain rule applicable layer by layer.
• The chain rule shows how to compute the gradient of a composite function by multiplying local gradients from output to input. The multiplicative structure is both the source of backprop's power (gradients compose cleanly) and its vulnerability (vanishing and exploding gradients).
• The backward pass traverses the computational graph in reverse, computing gradients for all parameters in time proportional to a single forward pass, roughly 2x the cost of the forward pass.
• Jacobian matrices formalize gradient flow through vector operations. In practice, backprop never forms full Jacobians but computes vector-Jacobian products directly, which is much more efficient.
• Gradient flow through activations depends on the activation: sigmoid and tanh saturate (vanishing gradients), while ReLU passes gradients through unchanged for positive inputs. Modern architectures use ReLU or GELU activations and add residual connections to keep gradients flowing through many layers.
• Numerical gradient checking verifies implementations by comparing analytical gradients against finite-difference approximations. Relative errors below $10^{-5}$ indicate a correct implementation.
• Common bugs include forgetting to zero gradients, wrong shapes in matrix products, missing activation derivatives, incorrect bias gradient summation over batches, and numerical instability from computing log-softmax naively.
• PyTorch autograd implements the same VJP-based algorithm, building a dynamic computation graph as code runs and traversing it in reverse on the .backward() call. The dynamic graph enables full Python control flow in models.
• Gradient accumulation allows simulating large batch sizes by accumulating gradients across multiple smaller passes before calling optimizer.step().
• Architectural solutions to vanishing gradients include residual connections (which create direct gradient highways), careful weight initialization (Xavier for tanh, He for ReLU), and normalization layers (batch normalization, layer normalization) that keep activations in healthy ranges.