Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Residual Connections › The Residual Connection Solution

Michael BrenndoerferUpdated June 8, 20256 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 103Residual Connections › The Residual Connection Solution

Reading section 3 of 12 · The Residual Connection Solution · 6 minSection 3 of 12 · 6 min

Section 3 of 12 · 6 min read

Residual Connections › The Residual Connection SolutionOne section · Sections in this chapter3 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The degradation problem presents a paradox: deeper networks should be more expressive, yet they perform worse. The solution comes from rethinking what we ask each layer to learn.

Consider a network that already performs well at 20 layers. If we add 10 more layers, the ideal outcome might be for those extra layers to simply pass information through unchanged, effectively becoming identity mappings. But asking a layer to learn $F(x) = x$ through its weights is surprisingly difficult. The layer must coordinate all its parameters to reproduce the input exactly, and any training noise pushes it away from this fragile configuration.

What if, instead of learning the complete transformation, each layer only learned the adjustment to make? This shift in perspective is the key insight behind residual learning.

From Transformations to AdjustmentsLink Copied

In a traditional neural network, each layer maps its input to a completely new representation. Given input $x$ , the layer computes:

y = F(x)

where:

$x$ : the input representation (a vector of activations from the previous layer)
$y$ : the output representation (passed to the next layer)
$F$ : the layer's learned transformation, typically combining linear projections (weights), nonlinearities (like ReLU), and possibly normalization

The layer bears full responsibility for producing $y$ from scratch. Every useful feature in the output must be constructed by the learned function $F$ .

Now consider what happens if the optimal output is simply the input, unchanged. The layer must learn $F(x) = x$ . This is an identity mapping, and while mathematically simple, it requires precise coordination of weights. Any perturbation during training disrupts this delicate balance.

Residual connections flip this relationship. Instead of learning the output directly, the layer learns only the difference between desired output and input:

y = F(x) + x

where:

$F(x)$ : the residual function, representing only the change the layer should make
$x$ : added directly to form the output (the skip or shortcut connection)
$y$ : equals the input plus whatever adjustment $F$ learns

The addition $+x$ is the skip connection or shortcut. It provides a direct path for the input to flow to the output unchanged. The function $F$ now only needs to learn what to add to the input, rather than recreating the input plus modifications.

Residual Learning

A residual block computes $y = F(x) + x$ , where $F(x)$ is the transformation learned by the layer and $x$ is the input. The layer learns the residual $F(x)$ rather than the full mapping. If the optimal transformation is close to identity, the layer only needs to learn small adjustments.

The Identity Mapping AdvantageLink Copied

This reformulation seems minor, just adding a term, but it fundamentally changes what the layer must learn. Consider what each architecture must learn to achieve an identity mapping:

Comparison of identity mapping requirements between traditional and residual architectures.
Architecture	To produce $y = x$ , must learn:	Difficulty
Traditional	$F(x) = x$	Hard: requires precise weight configuration
Residual	$F(x) = 0$	Easy: just set weights to zero

For the traditional layer, learning identity requires that every weight matrix perfectly reproduces the input. For the residual block, learning identity requires only that $F(x) = 0$ , which happens automatically when weights are small or zero. The skip connection handles the identity part for free.

This asymmetry changes the training problem:

Easy initialization: A residual block initialized with small weights is nearly an identity function, passing information through without disruption.
Graceful degradation: If a layer's contribution isn't helpful, it can "turn off" by driving $F(x)$ toward zero. The network doesn't lose information because $x$ still flows through.
Additive refinement: Each layer adds information rather than replacing it. The network builds up its representation incrementally, with each layer contributing what it can.

Demonstrating the DifferenceLink Copied

Let's see this principle in action. We'll create two blocks, one traditional and one residual, and observe how they behave when initialized with small weights.

In[7]:

Code

import torch.nn as nn


class PlainBlock(nn.Module):
    """Traditional block without skip connection."""

    def __init__(self, dim):
        super().__init__()
        self.linear1 = nn.Linear(dim, dim)
        self.relu = nn.ReLU()
        self.linear2 = nn.Linear(dim, dim)

    def forward(self, x):
        out = self.linear1(x)
        out = self.relu(out)
        out = self.linear2(out)
        return out  # Must learn full transformation


class ResidualBlock(nn.Module):
    """Block with skip connection."""

    def __init__(self, dim):
        super().__init__()
        self.linear1 = nn.Linear(dim, dim)
        self.relu = nn.ReLU()
        self.linear2 = nn.Linear(dim, dim)

    def forward(self, x):
        residual = self.linear1(x)
        residual = self.relu(residual)
        residual = self.linear2(residual)
        return x + residual  # Learn only the adjustment

The two classes are nearly identical, differing only in their final line: the plain block returns out while the residual block returns x + residual. This single change, adding the input back, creates the skip connection.

Now we'll test what happens when the output layer is initialized to zero, simulating a layer that hasn't yet learned to contribute:

In[8]:

Code

import torch

# Compare what each block outputs for zero-initialized weights
dim = 64
plain = PlainBlock(dim)
residual = ResidualBlock(dim)

# Initialize second linear layer to zero
nn.init.zeros_(plain.linear2.weight)
nn.init.zeros_(plain.linear2.bias)
nn.init.zeros_(residual.linear2.weight)
nn.init.zeros_(residual.linear2.bias)

# Test with random input
x = torch.randn(1, dim)
plain_out = plain(x)
residual_out = residual(x)

Out[9]:

Console

Behavior with Zero-Initialized Output Layer
--------------------------------------------------
Input norm:                7.9568
Plain block output norm:   0.0000
Residual block output norm: 7.9568

Plain output equals input:    False
Residual output equals input: True

The results reveal the fundamental difference. The plain block, with its output layer zeroed, produces output with near-zero norm. It has destroyed the input signal entirely. The residual block, in contrast, produces output identical to its input. The skip connection preserves information perfectly when the learned function $F$ contributes nothing.

This behavior at initialization cascades through deep networks. Stack 50 plain blocks with zero output layers, and the signal is completely lost. Stack 50 residual blocks with zero output layers, and the input passes through unchanged. As training progresses, each layer can gradually "turn on" its residual contribution without ever disrupting the signal flow.