Backpropagation (1986) › What is Backpropagation?

Backpropagation solved the credit assignment problem with a surprisingly elegant idea: work backward through the network, calculating exactly how much each weight contributed to the error. The insight was that you could trace the error signal backward from the output layer through each hidden layer, decomposing the total error into contributions from individual connections. At each layer, you could determine precisely how changing a particular weight would affect the final output.

Technically speaking, backpropagation is an algorithm that efficiently computes the gradients of the loss function with respect to each weight in a neural network. The loss function measures how wrong the network's predictions are, and the gradient tells you how to adjust each weight to reduce that wrongness. In more practical terms, backpropagation figures out which weights to adjust, by how much, and in which direction to reduce errors. It transforms the opaque, seemingly intractable problem of training deep networks into a systematic, computational process that a computer can execute efficiently.

The mathematical trick that makes this possible is the chain rule from calculus, a principle you might remember from introductory calculus courses. The chain rule provides a way to decompose complicated derivatives into simpler pieces that can be computed step by step. In the context of neural networks, it lets you efficiently compute how each weight, no matter how deep in the network, affects the final output. You start with the error at the output layer and systematically work backward, using the chain rule to propagate that error signal through each layer. A calculation that might appear to require exponentially many operations can be done in linear time, making deep networks practical to train. This computational efficiency made training deep networks feasible.