Backpropagation › The Gradient Problem

Neural networks have millions of parameters. GPT-3 has 175 billion. How do we figure out which direction to nudge each one to reduce the loss?

Calculus gives us the answer: the gradient. For a function $f(\mathbf{w})$ where $\mathbf{w}$ is a vector of parameters, the gradient $\nabla_\mathbf{w} f$ is a vector that points in the direction of steepest increase of $f$. Each component tells us how sensitive the function is to changes in the corresponding parameter. To minimize the loss, we move opposite to the gradient:

where:

• $\mathbf{w}_{\text{new}}$: the updated parameter vector after one step
• $\mathbf{w}_{\text{old}}$: the current parameter vector before the update
• $\eta$: the learning rate, a small positive scalar controlling step size
• $\nabla_\mathbf{w} \mathcal{L}$: the gradient of the loss with respect to parameters, where each entry $\frac{\partial \mathcal{L}}{\partial w_i}$ indicates how much the loss increases if we increase $w_i$ by a tiny amount
• $\mathcal{L}$: the loss function, a scalar value measuring prediction error

The subtraction makes intuitive sense: if $\frac{\partial \mathcal{L}}{\partial w_i} > 0$, increasing $w_i$ would increase the loss, so we decrease it instead. This is gradient descent, simple in concept, but the challenge is computing $\nabla_\mathbf{w} \mathcal{L}$ when the loss depends on parameters through many nested operations.

Consider a 3-layer network. The loss depends on the output, which depends on the third layer activations, which depend on the second layer activations, which depend on the first layer activations, which depend on the input and first layer weights. This chain of dependencies seems hopelessly complex to differentiate.

Backpropagation solves this elegantly. By applying the chain rule in reverse order, from output back to input, we compute all gradients in time proportional to a single forward pass. This efficiency is what makes training deep networks practical.

The story of backpropagation stretches back to the 1960s and 1970s, though its impact on neural networks only became clear in the 1980s. The basic idea of differentiating through a chain of composed functions was understood in the automatic differentiation community, but applying it to neural networks required several conceptual steps.

The paper that really crystallized the algorithm for the machine learning community was Rumelhart, Hinton, and Williams' 1986 paper "Learning representations by back-propagating errors" in Nature. This paper demonstrated that backpropagation could efficiently train multilayer networks to learn useful internal representations, which was not obvious at the time. Before this, the dominant view in AI was that useful internal representations needed to be hand-designed by experts.

The key insight of the 1986 paper was simple but powerful: hidden units in a network could learn to represent features of the input that were useful for the task, and backpropagation would discover what those features should be automatically. This was a striking departure from prior approaches that either used only one-layer models or required laborious feature engineering.

Despite this theoretical breakthrough, neural networks fell out of fashion in the 1990s for two reasons. First, training was slow on the hardware available then. Second, networks large enough to solve interesting problems ran into the vanishing gradient problem, making deep networks hard to train in practice. Other methods like support vector machines and random forests worked better for many problems. The deep learning renaissance beginning around 2006, with better hardware (GPUs), better initialization, and better architectures (like ReLU activations and skip connections), finally made the potential suggested by the 1986 paper practical.