Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Residual Connections › The Deep Network Problem

Michael BrenndoerferUpdated June 8, 20256 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 103Residual Connections › The Deep Network Problem

Reading section 2 of 12 · The Deep Network Problem · 6 minSection 2 of 12 · 6 min

Section 2 of 12 · 6 min read

Residual Connections › The Deep Network ProblemOne section · Sections in this chapter2 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before residual connections, training deep networks was notoriously difficult. The culprit is the chain rule of calculus: during backpropagation, gradients are multiplied layer by layer. With many layers, these products either explode or vanish.

The Depth Dilemma

Deep networks have more representational capacity than shallow ones, but they're harder to optimize. The gradient signal that guides learning must travel through every layer, and each layer can amplify or diminish it. With dozens of layers, even small per-layer effects compound catastrophically.

Consider a network with $L$ layers. During backpropagation, we need to compute how the loss changes with respect to parameters deep in the network. The chain rule tells us to multiply derivatives through each intermediate layer. For parameters $\theta_l$ in layer $l$ , this gradient involves products of Jacobian matrices from all subsequent layers:

\frac{\partial \mathcal{L}}{\partial \theta_l} = \frac{\partial \mathcal{L}}{\partial h_L} \cdot \prod_{k=l}^{L-1} \frac{\partial h_{k+1}}{\partial h_k} \cdot \frac{\partial h_l}{\partial \theta_l}

where:

$\mathcal{L}$ : the loss function we're minimizing (e.g., cross-entropy)
$\theta_l$ : parameters (weights and biases) in layer $l$
$h_k$ : the hidden representation (activation vector) at layer $k$
$h_L$ : the final output of the network
$\frac{\partial h_{k+1}}{\partial h_k}$ : the Jacobian matrix of layer $k$ 's transformation, showing how each output dimension depends on each input dimension
$\prod_{k=l}^{L-1}$ : the product of all Jacobians from layer $l$ to the output layer $L-1$

The critical term is the product of Jacobians. Each Jacobian $\frac{\partial h_{k+1}}{\partial h_k}$ is a matrix, and we multiply $L - l$ of them together. If each Jacobian has spectral norm (largest singular value) slightly greater than 1, the product grows exponentially. If slightly less than 1, it shrinks exponentially. For example, with spectral norm 0.9 across 50 layers, the gradient shrinks by a factor of $0.9^{50} \approx 0.005$ , effectively killing the learning signal.

Keeping gradients stable across 50 or 100 layers requires exquisite balance that's nearly impossible to maintain during training.

In[3]:

Code

import numpy as np


def simulate_gradient_flow(n_layers, jacobian_scale):
    """Simulate gradient magnitude through a deep network."""
    gradient_magnitudes = [1.0]  # Start with unit gradient

    for _ in range(n_layers):
        # Simulate Jacobian multiplication with some noise
        scale = jacobian_scale * (1 + 0.1 * np.random.randn())
        gradient_magnitudes.append(gradient_magnitudes[-1] * scale)

    return gradient_magnitudes


# Compare different Jacobian scales
layers = 50
exploding = simulate_gradient_flow(layers, 1.05)  # Slightly > 1
stable = simulate_gradient_flow(layers, 1.0)  # Exactly 1
vanishing = simulate_gradient_flow(layers, 0.95)  # Slightly < 1

Out[4]:

Visualization

Line plot showing three gradient trajectories diverging exponentially based on Jacobian scale. — Gradient magnitude across network depth for different Jacobian scales. Even small deviations from 1.0 compound exponentially. With scale 1.05, gradients explode to 10× after 50 layers. With scale 0.95, they vanish to 0.1× of their original magnitude.

The simulation reveals the challenge. A mere 5% deviation from perfect gradient preservation compounds across 50 layers to produce 10x amplification or 90% attenuation. Real networks with nonlinearities, varying weight magnitudes, and changing activation patterns face even more severe instabilities. Training such networks requires fighting against mathematics.

Why Adding Layers Can Hurt PerformanceLink Copied

A counterintuitive phenomenon plagued early deep learning: adding more layers to a well-performing network often decreased accuracy. This wasn't just about training difficulty. Even on training data, deeper networks performed worse than shallower ones.

This violates basic intuition. A 56-layer network should be at least as expressive as a 20-layer network: it could theoretically learn to make the extra 36 layers act as identity mappings, reproducing the shallower network's behavior. Yet empirically, the 56-layer network trains to higher error.

In[5]:

Code

# Simulating the degradation problem
# In reality, this was observed on CIFAR-10 and ImageNet with plain networks

network_depths = [20, 32, 44, 56, 110]
# Simulated training errors (based on He et al., 2015 observations)
plain_train_error = [4.5, 5.2, 6.1, 7.5, 8.9]  # Gets worse with depth
plain_test_error = [8.2, 8.8, 9.5, 10.8, 12.1]

Out[6]:

Visualization

Grouped bar chart showing training and test error increasing with network depth from 20 to 110 layers. — The degradation problem in plain networks. Both training and test error increase with depth, indicating an optimization failure rather than overfitting. A 110-layer network performs substantially worse than a 20-layer network even on training data.

The pattern is unmistakable: deeper plain networks achieve higher error on both training and test sets. If this were overfitting, we'd see low training error and high test error. Instead, both increase monotonically with depth. The optimization itself is failing. The deeper networks simply cannot learn the transformations they theoretically can represent.