Backpropagation › Backpropagation

How does a neural network learn? In the previous chapters, we built multilayer perceptrons that transform inputs through layers of weights and activations, and we defined loss functions that measure prediction quality. But we haven't answered the fundamental question: how do we update the millions of parameters in a neural network to reduce the loss?

The answer is backpropagation, short for "backward propagation of errors." This algorithm computes the gradient of the loss function with respect to every weight in the network, telling us exactly how to adjust each parameter to improve predictions. Backpropagation is the most important algorithm in deep learning. Without it, training neural networks would be computationally intractable.

This chapter builds backpropagation from the ground up. We start with computational graphs that visualize how operations compose, then derive the chain rule that makes gradient computation tractable. You'll trace through forward and backward passes step by step, understand gradient flow through activation functions, examine Jacobian matrices for vector operations, and implement backpropagation from scratch. We'll also cover numerical gradient checking to verify implementations, common backprop bugs, and how PyTorch autograd works internally. By the end, you'll see exactly how modern deep learning frameworks compute gradients automatically.