Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Residual Connections › Residual Connections in Transformers

Michael BrenndoerferUpdated June 8, 20254 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 103Residual Connections › Residual Connections in Transformers

Reading section 5 of 12 · Residual Connections in Transformers · 4 minSection 5 of 12 · 4 min

Section 5 of 12 · 4 min read

Residual Connections › Residual Connections in TransformersOne section · Sections in this chapter5 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Having established why residual connections work in general, we now turn to their specific application in transformers. The transformer architecture relies heavily on residual connections, not as an optional enhancement, but as a fundamental design principle that makes training possible.

A transformer encoder or decoder consists of stacked identical blocks. Each block contains two distinct operations: multi-head self-attention (which lets tokens gather information from each other) and a position-wise feed-forward network (which processes each token independently). Without residual connections, stacking 12, 24, or 96 of these blocks would be impossible to train. With them, modern language models routinely use such depths.

The Transformer Block PatternLink Copied

A standard transformer block processes input $x$ through two sub-layers, each wrapped with its own residual connection. The computation proceeds as:

\begin{aligned} h &= x + \text{MultiHeadAttention}(x) \\ y &= h + \text{FFN}(h) \end{aligned}

where:

$x \in \mathbb{R}^{n \times d}$ : input to the transformer block, with $n$ tokens and $d$ -dimensional embeddings
$\text{MultiHeadAttention}(x)$ : the attention sub-layer output, same shape as $x$
$h \in \mathbb{R}^{n \times d}$ : intermediate representation after the first residual addition
$\text{FFN}(h)$ : the feed-forward sub-layer output, same shape as $h$
$y \in \mathbb{R}^{n \times d}$ : final block output, passed to the next transformer block

Each addition ( $+$ ) represents a skip connection. Notice the interpretive power this structure provides:

Attention as adjustment: The attention sub-layer doesn't produce a completely new representation; it learns what context-aware information to add to each token based on other positions in the sequence.
FFN as refinement: The feed-forward network learns what position-wise transformations to add to the post-attention representation, processing each token independently.
Graceful bypass: If either sub-layer's contribution isn't helpful for a particular input, it can learn to produce near-zero output. The information flows through unchanged via the skip connection.

This framing helps explain what transformers learn: not arbitrary transformations at each layer, but successive refinements to a representation that remains connected to the original input.

In[15]:

Code

class TransformerBlockWithResiduals(nn.Module):
    """
    Simplified transformer block showing residual structure.
    Uses post-norm configuration for clarity.
    """

    def __init__(self, d_model, n_heads, d_ff):
        super().__init__()
        # Multi-head attention
        self.attention = nn.MultiheadAttention(
            d_model, n_heads, batch_first=True
        )
        self.norm1 = nn.LayerNorm(d_model)

        # Feed-forward network
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_ff), nn.ReLU(), nn.Linear(d_ff, d_model)
        )
        self.norm2 = nn.LayerNorm(d_model)

    def forward(self, x):
        # First residual: attention
        attn_output, _ = self.attention(x, x, x)
        x = self.norm1(x + attn_output)  # Residual + LayerNorm

        # Second residual: FFN
        ffn_output = self.ffn(x)
        x = self.norm2(x + ffn_output)  # Residual + LayerNorm

        return x


# Create a transformer block
d_model = 512
n_heads = 8
d_ff = 2048

block = TransformerBlockWithResiduals(d_model, n_heads, d_ff)

Out[16]:

Console

Transformer Block with Residuals
--------------------------------------------------
Input shape:  (4, 16, 512)
Output shape: (4, 16, 512)
Number of parameters: 3,152,384

Components:
  Attention: MultiheadAttention(d_model=512, n_heads=8)
  FFN: Linear(512 -> 2048 -> 512)

The block processes a batch of 4 sequences, each with 16 tokens and 512-dimensional embeddings. The output maintains the same shape, a requirement for residual connections to work since the addition $x + F(x)$ requires matching dimensions. With over 6 million parameters split between attention and FFN, each sub-layer contributes its learned adjustments through the residual pattern x + SubLayer(x).

Why Every Sub-Layer Gets a ResidualLink Copied

You might wonder: why not just one residual connection per transformer block, wrapping both attention and FFN together? The answer relates to optimization stability and gradient flow.

With residual connections around each sub-layer:

Gradients have two "highways" per block: one bypassing attention, one bypassing FFN
Each sub-layer can be trained somewhat independently
The network can easily learn to skip either sub-layer if it's not helpful

With a single residual per block:

Gradients must flow through both attention and FFN or bypass both
Harder to learn that attention is useful but FFN isn't (or vice versa)
Effectively half as many gradient highways

The empirical result is that per-sub-layer residuals train more stably and achieve better final performance, especially in deep networks.