Residual Connections › The Gradient Highway Interpretation

Residual connections preserve information and enable identity mappings in the forward pass. During backpropagation, skip connections create express lanes for gradients to flow backward.

To understand why this matters, recall the vanishing gradient problem from earlier. In plain networks, gradients must traverse every layer's Jacobian matrix, and even slight attenuation at each layer compounds to near-zero gradients deep in the network. Residual connections fundamentally change this dynamic.

Consider a deep network with $L$ residual blocks. Let $x_0$ be the network input and $x_l$ be the output of block $l$. Each block computes:

$$x_{l+1} = x_l + F_l(x_l)$$

where:

• $x_l$: the input to block $l$ (output of block $l-1$)
• $x_{l+1}$: the output of block $l$
• $F_l$: the residual function for block $l$ (the learned transformation, excluding the skip connection)

Unlike traditional layers that chain together as compositions $F_L(F_{L-1}(\cdots F_1(x_0)))$, residual blocks have an additive structure. We can unroll this recurrence to see the elegant form it takes:

$$\begin{aligned} x_1 &= x_0 + F_0(x_0) \\ x_2 &= x_1 + F_1(x_1) = x_0 + F_0(x_0) + F_1(x_1) \\ &\vdots \\ x_L &= x_0 + \sum_{l=0}^{L-1} F_l(x_l) \end{aligned}$$

This formulation reveals something remarkable: the output $x_L$ is the sum of the original input $x_0$ and all intermediate residuals. The input isn't transformed through a chain of compositions. Instead, it's preserved and augmented. Each layer contributes additively rather than multiplicatively.

Now we trace gradients backward. We want to compute how the loss $\mathcal{L}$ changes with respect to an early representation $x_l$. The chain rule gives:

$$\frac{\partial \mathcal{L}}{\partial x_l} = \frac{\partial \mathcal{L}}{\partial x_L} \cdot \frac{\partial x_L}{\partial x_l}$$

where:

• $\frac{\partial \mathcal{L}}{\partial x_L}$: the gradient at the output layer (computed from the loss)
• $\frac{\partial x_L}{\partial x_l}$: how changes in $x_l$ affect the final output $x_L$

The critical question is: what is $\frac{\partial x_L}{\partial x_l}$? From the unrolled formulation $x_L = x_0 + \sum_{k=0}^{L-1} F_k(x_k)$, we differentiate with respect to $x_l$:

$$\frac{\partial x_L}{\partial x_l} = 1 + \frac{\partial}{\partial x_l} \sum_{k=l}^{L-1} F_k(x_k)$$

where:

• The 1 comes from the direct path: $x_l$ contributes to $x_L$ through the chain of skip connections ($x_l \to x_{l+1} \to \cdots \to x_L$)
• The sum captures indirect contributions through the residual functions $F_k$

This decomposition is the heart of residual learning. The gradient splits into two components:

1. Direct path (the "1"): Gradients flow directly through skip connections, completely bypassing all layer transformations. This path always contributes exactly 1 to the gradient magnitude.

2. Residual path (the sum): Gradients flow through the learned functions $F_k$, just as they would in a plain network. This path can vanish, explode, or behave unpredictably.

Even if the residual path completely vanishes (all $\frac{\partial F_k}{\partial x_l} \to 0$), the gradient still maintains magnitude 1 through the identity path. The skip connection guarantees a floor on gradient magnitude.

Let's simulate this gradient decomposition to see how the two paths behave as we go deeper into the network:

The term "gradient highway" captures this behavior perfectly. Think of a city's road network: local streets pass through many intersections, each one potentially causing delays or blockages. An expressway bypasses these intersections entirely. This provides guaranteed throughput regardless of local traffic conditions. Skip connections serve the same purpose for gradients. They bypass the potentially problematic layer transformations. This provides guaranteed gradient flow to early parameters.