Residual Connections (2015) › The Problem

The problem of training very deep networks had been recognized for years, but its severity became clear as researchers attempted to scale convolutional networks to greater depths. Early deep networks like AlexNet had eight layers, and by 2014, networks like VGG had grown to 19 layers and GoogLeNet used a complex architecture with 22 layers. Researchers attempted to go even deeper, expecting that additional depth would capture more abstract and sophisticated features. However, experiments revealed a surprising and frustrating result: adding more layers often led to higher training error, not lower.

This observation was puzzling because in theory, a deeper network should always be able to match or exceed the performance of a shallower network. A deeper network could simply set its additional layers to perform identity transformations, copying inputs to outputs, and then continue learning from there. If a network with $n$ layers could achieve a certain error rate, a network with $n + k$ layers should be able to achieve at least that same error rate by learning the identity function in the extra $k$ layers and then performing the same computation as the original $n$-layer network. Yet in practice, this wasn't happening.

The root cause was the vanishing gradient problem. When training deep networks using backpropagation, gradients must flow from the output layer back to the input layer. In a network with many layers, each layer multiplies the gradient by the weight matrix during backpropagation. If the weights are small or the activations are in saturation regions, these multiplications cause gradients to shrink exponentially as they propagate backward. By the time gradients reach the early layers, they may be so small that they provide no useful learning signal, effectively freezing those layers early in training.

Batch normalization, introduced in 2015, helped address this problem by normalizing activations and providing more stable gradients. However, even with batch normalization, training networks with more than about 20 layers remained challenging. The gradients were still decaying, and the optimization landscape for very deep networks appeared to have many local minima and flat regions that made convergence difficult. Researchers needed a more fundamental solution that would change how information flowed through the network.

Another related problem was the difficulty of learning identity mappings. In theory, a network should be able to learn to pass inputs through unchanged, but in practice, this proved difficult. Each layer applies a nonlinear transformation, and even if the goal is to preserve the input, the network must learn to counteract its own nonlinearities to approximate the identity function. For deep networks, this requires coordinating many layers to collectively produce an identity mapping, which is a complex optimization problem that can be difficult to solve.

The computational cost of training deep networks was also significant. When training failed, it often failed after many hours or days of computation, wasting computational resources. Researchers needed a way to ensure that deeper networks would train successfully and converge to good solutions reliably, not just occasionally. The field needed an architectural innovation that would make depth beneficial rather than problematic.