Backpropagation (1986) › The Limitations

Backpropagation wasn't a silver bullet that solved all problems in neural network training. While it made deep learning possible, it also revealed new challenges and limitations. Some of these were inherent to the algorithm itself, while others emerged only as researchers tried to push networks deeper and tackle more complex problems. Many of these challenges remain active areas of research today, nearly four decades after backpropagation's introduction.

The first major problem showed up when researchers tried to build truly deep networks with many layers. They discovered that gradients could become exponentially smaller as they propagated backward through the layers. Each layer multiplies the gradient by various weights and derivatives, and when these numbers are small (less than one), repeated multiplication causes the gradient to shrink rapidly. By the time the error signal reaches the early layers, it might be so tiny as to be effectively zero.

This meant that early layers in deep networks learned extremely slowly or stopped learning entirely. The network could adjust its final layers reasonably well, but the early layers responsible for fundamental features remained stuck with nearly random weights. This severely limited the practical depth of networks for many years, creating a frustrating situation where deeper architectures seemed theoretically better but couldn't be trained effectively. The problem was eventually addressed through innovations like better activation functions (ReLU instead of sigmoid), careful weight initialization schemes, and architectural changes like residual connections, but it remains a consideration in network design today.

Then there's the problem of getting stuck in suboptimal solutions. The loss landscape, which you can visualize as a hilly terrain where you're trying to find the lowest valley, contains many local minima. These are points where the gradient is zero, so gradient descent stops, but they're not the best possible solution, just better than their immediate surroundings. Imagine standing in a small depression on the side of a mountain. You're at a local low point, but the actual valley floor is much further down. Gradient descent can settle into these mediocre solutions instead of finding the global minimum.

This led to inconsistent and frustrating results, especially in the early days. Train the same network architecture twice with different random starting weights, and you might get wildly different performance. One run might find a great solution, while another got stuck in a poor local minimum. This made research difficult because you couldn't be sure whether a new idea helped or whether you just got lucky with the initialization.

Interestingly, this problem turned out to be less severe than initially feared, at least for large networks. Modern research suggests that many local minima in high-dimensional spaces perform nearly as well as the global minimum. Additionally, techniques like momentum-based optimization, which we'll discuss in later chapters, help the optimizer escape shallow local minima. Still, the stochastic and somewhat unpredictable nature of neural network training traces back to this fundamental issue.

Backpropagation, while efficient compared to alternatives, is still computationally expensive in absolute terms. You need to compute gradients for every weight in every layer, and as networks grew larger, this became a serious bottleneck. A modern large language model might have billions or even trillions of parameters, and backpropagation must compute a gradient for each one, for each training example.

Early researchers were severely constrained by computational limitations. They were stuck with relatively small models, not because they didn't want bigger ones or because small models were sufficient, but because they simply couldn't afford to train larger networks. The computers of the late 1980s and early 1990s were orders of magnitude slower than today's hardware. Training a moderately sized network on a reasonable dataset might take days or weeks. Every experiment required careful consideration of computational budgets, and researchers had to make difficult tradeoffs between network size, dataset size, and the number of experiments they could run.

This computational barrier limited the practical application of neural networks for many years. While there had been earlier AI winters in the 1970s and late 1980s, neural networks continued to make progress through the 1990s and early 2000s, though progress was slower than it would later become. It was only with the advent of GPU acceleration in the late 2000s, particularly demonstrated by breakthroughs like AlexNet in 2012, that training truly large and deep networks became practical. The algorithmic breakthrough of backpropagation had arrived decades before the hardware could fully exploit it.

Finally, neural networks trained with backpropagation had a troubling tendency to memorize their training data rather than learning general patterns. They'd achieve excellent performance on the training set but fail miserably when confronted with new, unseen examples. This overfitting problem is a fundamental challenge in machine learning, but it became particularly acute with neural networks because of their enormous capacity to memorize.

Think of it like a student who memorizes every practice problem verbatim but doesn't understand the underlying concepts. When exam day arrives with slightly different problems, they're lost. Neural networks can do something similar, learning to recognize specific training examples perfectly without extracting the generalizable patterns that would let them handle new data.

The problem gets worse with larger networks and smaller datasets. A network with millions of parameters trained on just thousands of examples has more than enough capacity to simply memorize every training example. Yet we need large networks to learn complex patterns, and we can't always get massive datasets. This creates a difficult tension.

The solution involved developing sophisticated regularization techniques. Dropout randomly turns off neurons during training, preventing the network from relying too heavily on any particular path through the network. Weight decay penalizes large weights, encouraging the network to use simpler patterns. Early stopping halts training before the network has fully memorized the training set. Batch normalization, skip connections, and data augmentation all help in various ways. But getting these techniques right required expertise and extensive experimentation. Even today, preventing overfitting while maintaining the network's ability to learn complex patterns remains a delicate balancing act.