Residual Connections (2015) › Limitations

While residual connections solved critical problems with training deep networks, they also introduced some limitations and considerations. One issue is that residual connections can create a tendency for networks to rely too heavily on the shortcut path, potentially underutilizing the learned transformations. If the shortcut connection always provides a strong signal, the learned layers might not develop as sophisticated representations as they could in a network without shortcuts. However, in practice, this hasn't proven to be a major problem, as the learned transformations do contribute significantly to performance.

The addition operation in residual connections requires that inputs and outputs have compatible dimensions. While this is straightforward when dimensions match, handling dimension mismatches requires additional layers (the projection matrices $W_s$), which adds parameters and computational cost. In some architectures, particularly those with complex dimension changes, managing these projections can become complex. However, the benefits of residual connections typically outweigh these costs.

Another consideration is that residual connections increase memory usage during training. Because the input must be stored to be added to the output, residual blocks require more memory than traditional layers. This can be a constraint when training very large models or when working with limited computational resources. The memory overhead is usually manageable but becomes more significant as model size increases.

Some research has questioned whether residual connections are always necessary or optimal. In certain settings, particularly with careful initialization and normalization, very deep networks can sometimes be trained without residual connections. However, these cases are exceptions, and residual connections remain the standard approach for training deep networks reliably. The technique provides such significant benefits for training stability and convergence that it's generally worth the small overhead.

Residual connections also don't solve all problems with deep networks. Issues like overfitting, catastrophic forgetting in continual learning scenarios, and computational efficiency remain challenges that residual connections don't directly address. Networks with residual connections can still overfit if not properly regularized, and they still require significant computational resources for training and inference.

The technique works best when the transformations being learned are incremental refinements rather than complete reworkings of the representation. In cases where each layer needs to make substantial changes to the representation, the shortcut connection might interfere with learning. However, for most practical applications, the incremental refinement that residual connections enable is exactly what's needed for effective feature learning.