Residual Connections (2015) › The Solution

Residual connections solve these problems by providing a direct pathway for information to flow through the network. Instead of requiring each layer to learn the complete transformation from input to output, residual connections allow layers to learn the residual, or difference, between input and desired output. This architectural pattern makes it trivial for the network to learn identity mappings: if the residual is zero, the output equals the input.

The core idea is captured in the residual block. A traditional layer computes $y = F(x)$, where $F$ represents the transformation learned by the layer. A residual block computes $y = F(x) + x$, adding the input $x$ directly to the output of the transformation. This simple addition creates a shortcut connection that bypasses the layer, allowing information to flow directly from input to output if the learned transformation $F$ is close to zero.

The mathematical formulation is straightforward. Given an input $x$ and a learned transformation $F(x)$ represented by one or more layers, the output of a residual block is:

$$y = F(x) + x$$

This formulation assumes that the input $x$ and the output of $F(x)$ have the same dimensions. When dimensions don't match, as they might when changing the number of channels in a convolutional network or when downsampling, a linear projection $W_s$ can be applied to the shortcut connection:

$$y = F(x) + W_s x$$

The projection matrix $W_s$ is learned during training, but it's typically implemented as a simple convolutional layer with $1 \times 1$ filters that matches dimensions while preserving spatial information when necessary.

The power of this approach becomes clear when considering what the network needs to learn. In a traditional deep network, a layer must learn the complete transformation from its input to a desired output. In a network with residual connections, a layer only needs to learn the difference, or residual, between input and output. If the desired output is very similar to the input, the layer can learn to produce a small residual, making the learning task easier. If the desired transformation is significant, the layer can learn a large residual. In the extreme case where the identity mapping is optimal, the layer can learn to produce zero, and the shortcut connection handles the identity mapping automatically.

This makes training much more stable. Even if a layer's learned transformation $F(x)$ is initially poor or produces gradients that are too small, the shortcut connection ensures that the input still reaches the output with a strong gradient signal. The network can learn incrementally: early in training, the shortcut connections dominate. This provided a baseline performance. As training progresses, the learned transformations $F(x)$ can refine this baseline by learning corrections that improve performance.

The ResNet architecture applies this residual block pattern throughout a deep convolutional network. The network is organized into stages, each containing multiple residual blocks. Within each block, typically two or three convolutional layers are applied, followed by batch normalization and ReLU activations. The input to the block is added to the output of these transformations, and another ReLU is applied after the addition. This pattern repeats throughout the network, creating a deep architecture where each stage can incrementally refine features while maintaining gradient flow.

The architecture also incorporates downsampling layers that reduce spatial dimensions and increase the number of channels as the network processes higher-level features. At these transition points, the shortcut connections use $1 \times 1$ convolutions with stride 2 to match dimensions. This allows the network to progressively extract features at multiple scales while maintaining the benefits of residual connections throughout.