Residual Connections › Limitations and Trade-offs

Residual connections have proven remarkably effective, so much so that they're essentially mandatory for deep networks. But no architectural choice is without trade-offs. Understanding the limitations helps make informed design decisions.

The skip connection requires storing the input $x$ until the residual $F(x)$ is computed, so they can be added. In a single block, this is trivial. In a deep network during training, it accumulates:

• Plain networks: Store only the current layer's activations (can discard earlier layers)
• Residual networks: Store input to each block until the addition completes

For training with large batch sizes and long sequences, this memory overhead becomes significant. A 32-layer transformer processing 2048-token sequences might need to store 32 copies of the activation tensor for the residual additions alone. Techniques like gradient checkpointing help by recomputing some activations during the backward pass instead of storing them, trading compute for memory.

The residual formulation $y = x + F(x)$ imposes a fundamental constraint: $x$ and $F(x)$ must have matching dimensions for the addition to be valid. In transformers, this is straightforward because the hidden dimension stays constant throughout. But in convolutional networks or architectures that change dimensionality, this creates design challenges.

When dimensions must change (e.g., increasing channels, reducing spatial resolution, or transitioning between embedding sizes), you need projection shortcuts:

$$y = W_s x + F(x)$$

where:

• $W_s$: a learned projection matrix of shape $(d_{\text{out}}, d_{\text{in}})$ that maps input dimension $d_{\text{in}}$ to output dimension $d_{\text{out}}$
• $W_s x$: the projected input, now matching the dimension of $F(x)$
• $F(x)$: the residual transformation, which outputs vectors of dimension $d_{\text{out}}$

Projection shortcuts work, but they sacrifice some benefits. The gradient through $W_s x$ is multiplied by $W_s^T$, reintroducing potential gradient scaling issues. The skip path is no longer "free": it adds parameters, computation, and a transformation that can amplify or attenuate gradients. This is why transformer architectures prefer to maintain constant dimensions throughout, using the simpler $y = x + F(x)$ formulation everywhere.

The ease of learning identity mappings, a key feature of residual connections, can become a bug. If the optimization finds a local minimum where $F(x) \approx 0$ for many layers, the network effectively collapses to a much shallower architecture. The layers exist and consume memory, but they contribute nothing.

This "identity plateau" isn't catastrophic (the network still functions), but it fails to utilize the model's full capacity. You've paid for 96 layers but are effectively using 20. Careful initialization, appropriate learning rates, and normalization help avoid this plateau, as does training long enough for gradients to "turn on" dormant layers.

Despite these limitations, residual connections have become indispensable. The benefits (stable gradient flow, trainable depth, additive representation refinement) far outweigh the costs. Any serious deep learning practitioner will encounter residual connections repeatedly; understanding their mechanics is foundational knowledge.