Residual Connections (2015) › 2015: Residual Connections

By 2015, deep convolutional neural networks had demonstrated remarkable success in computer vision tasks. Networks like VGG and GoogLeNet had achieved impressive results on ImageNet, showing that additional depth could improve learned visual representations. However, researchers were encountering a counterintuitive problem: adding more layers to networks wasn't always beneficial, and in many cases, deeper networks performed worse than their shallower counterparts. This observation challenged the intuition that more depth should always enable more sophisticated feature learning.

Researchers at Microsoft Research Asia, led by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, investigated this phenomenon and discovered that the problem wasn't that deeper networks couldn't learn useful representations, but rather that training them had become increasingly difficult. The vanishing gradient problem, first identified in the context of recurrent networks, was also affecting very deep convolutional networks. As networks grew deeper, gradients would diminish as they propagated backward through many layers, making it difficult or impossible for early layers to learn effectively.

The solution they developed, residual connections, would prove to be one of the most important architectural innovations in deep learning. The idea was elegantly simple: instead of forcing layers to learn the complete transformation from input to output, allow them to learn the residual, or difference, between input and output. This approach enables networks to learn identity mappings when necessary, effectively allowing the network to skip layers that don't contribute to the task. The resulting architecture, ResNet, could be trained with networks over 100 layers deep, achieving dramatic improvements in accuracy while being easier to train than much shallower networks.

The impact of residual connections extended far beyond computer vision. The technique became fundamental to training deep networks across domains, from natural language processing to reinforcement learning. When transformer architectures emerged in 2017, they incorporated residual connections extensively throughout their structure. Modern language models rely on residual connections to enable training of networks with dozens or hundreds of layers. The technique has become so standard that it's now considered an essential component of nearly any deep neural network architecture.