Backpropagation (1986) › How Backpropagation Works

Let's make this concrete by walking through a complete example. We'll examine how backpropagation trains a sentiment analysis system using a simple three-layer neural network. This example will illustrate both the forward pass, where the network makes a prediction, and the backward pass, where it learns from its mistakes.

Say we feed the sentence "I love this movie" into our network. The network processes this input through three stages, transforming the words into a final prediction.

First, the input layer converts the sentence into numerical form. Each word gets represented as a word embedding, a vector of numbers that captures something about the word's meaning. The words "I", "love", "this", and "movie" each become vectors of perhaps 50 or 100 numbers. These vectors might encode semantic properties like whether a word is positive or negative, whether it relates to emotions, whether it's a noun or verb, and so on.

Next, the hidden layer takes these numerical representations and computes weighted combinations of them. Each neuron in this layer combines the input values using learned weights, adds them up, and then applies an activation function. The activation function introduces non-linearity, allowing the network to capture complex patterns that simple linear combinations couldn't represent. You might think of this layer as detecting higher-level features in the text, like whether emotional language is present or whether the sentence structure suggests a positive or negative tone.

Finally, the output layer produces a single number between 0 and 1, which we interpret as a probability. A value near 0 means "definitely negative sentiment," while a value near 1 means "definitely positive sentiment." For our sentence "I love this movie," we'd hope the network outputs something close to 1. But imagine it's still early in training, and the weights aren't well tuned yet. The network might output 0.3, incorrectly classifying this positive review as negative.

The network predicted 0.3, but the review is positive, which we represent as 1.0. Something has clearly gone wrong, and this is where backpropagation does its work. The algorithm proceeds through three systematic steps to fix the problem.

First, we compute the error using a loss function. A common choice is squared error, which in this case gives us Loss = (1.0 - 0.3)² = 0.49. This is a substantial error. The loss function quantifies just how wrong the prediction was, giving us a single number that represents the network's failure. Different loss functions emphasize different aspects of being wrong. Squared error, for example, penalizes large errors much more heavily than small ones.

Next, we calculate gradients by working backward through the network. This is the heart of backpropagation and where the chain rule earns its keep. Starting from the output layer, we ask: how much did the output neuron's weights contribute to this error? We can calculate this directly because we know the output and the loss. Then we move to the hidden layer and ask: how much did each of these neurons' weights contribute to the output layer's error? Using the chain rule, we can decompose the overall error into contributions from each layer, then from each neuron, and finally from each individual weight. The result is a gradient for every single weight in the network, telling us exactly how much that particular weight contributed to the prediction being wrong.

Finally, we update the weights. For each weight, we nudge it in the direction that reduces the error. The size of the nudge is proportional to the gradient, so weights that contributed more to the error get adjusted more. We also multiply by a learning rate, a small number that controls how aggressive our updates are. After this update, if we were to run the same input through the network again, it would predict something slightly closer to 1.0, perhaps 0.35 instead of 0.3. Repeat this process thousands or millions of times with different examples, and the network gradually learns to make accurate predictions.

For those interested in the mathematical details, let's look at what's happening under the hood. If you prefer to skip the equations and move on to the broader implications, feel free to jump to the next section. The high-level understanding you've gained so far is sufficient for following the rest of the article.

Backpropagation uses the chain rule from calculus to efficiently compute how each weight affects the final loss. For any weight $w_{ij}$ connecting neuron $i$ to neuron $j$, we need to calculate the gradient:

$$\frac{\partial L}{\partial w_{ij}} = \frac{\partial L}{\partial a_j} \cdot \frac{\partial a_j}{\partial z_j} \cdot \frac{\partial z_j}{\partial w_{ij}}$$

Here, $L$ is the loss function, $a_j$ is the activation of neuron $j$ (the output after applying the activation function), and $z_j$ is the weighted input to neuron $j$ (before the activation function). This formula breaks down the complex question "how does this weight affect the loss?" into three simpler questions we can answer sequentially: How does the loss change with the activation? How does the activation change with the weighted input? And how does the weighted input change with the weight? Each of these partial derivatives is straightforward to compute, and multiplying them together gives us the gradient we need.

For the output layer, calculating how the loss changes with the activation is relatively straightforward. If we're using squared error loss, the derivative is:

$$\frac{\partial L}{\partial a_j} = 2(a_j - y_j)$$

This simply tells us that the rate of change of the loss is proportional to how far off our prediction $a_j$ is from the true value $y_j$. The further off we are, the steeper the gradient and the larger the correction we'll make.

For hidden layers, things get more interesting. We use the chain rule to propagate gradients backward from the output. The gradient for a hidden layer activation is:

$$\frac{\partial L}{\partial a_i} = \sum_j \frac{\partial L}{\partial a_j} \cdot \frac{\partial a_j}{\partial z_j} \cdot w_{ij}$$

This equation expresses a key insight: the gradient for a hidden neuron depends on the gradients of all the neurons it connects to in the next layer, weighted by the connection strengths. This is what allows error signals to flow backward through the network. Each layer's gradients are computed using the gradients from the layer ahead of it, creating a chain of computations that efficiently calculates all the gradients in a single backward pass through the network.

Once we have computed all the gradients, updating the weights is straightforward. We use gradient descent, one of the fundamental optimization algorithms in machine learning:

$$w_{ij}^{new} = w_{ij}^{old} - \alpha \frac{\partial L}{\partial w_{ij}}$$

The term $\alpha$ is the learning rate, a hyperparameter that controls how big of a step we take in the direction indicated by the gradient. Its value determines whether training converges efficiently. Set it too large and the network makes wild adjustments that can cause training to diverge, with the loss bouncing around or even increasing over time. Set it too small and the network learns painfully slowly, requiring far more training examples and computation time to reach good performance. In practice, researchers often start with a moderate learning rate and decrease it over time, allowing the network to make large adjustments early in training when it's far from a good solution, and smaller, more refined adjustments later when it's getting close.

The negative sign in the update rule is critical. We subtract the gradient because the gradient points in the direction of increasing loss. We want to move in the opposite direction, toward decreasing loss. This is why it's called gradient descent: we're descending down the loss landscape, seeking the lowest point we can find.