Backpropagation › The Chain Rule: Foundation of Backpropagation

The chain rule from calculus is the mathematical engine powering backpropagation. It tells us how to differentiate composed functions, and since a neural network is one giant composition of functions, the chain rule is precisely what we need.

Let $u = g(x)$ and $y = f(u) = f(g(x))$. The chain rule states:

$$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$$

where:

• $\frac{dy}{dx}$: the total derivative of the final output with respect to the original input
• $\frac{dy}{du}$: the local gradient of the outer function $f$ at the intermediate value $u$
• $\frac{du}{dx}$: the local gradient of the inner function $g$ at $x$

The intuition is multiplicative sensitivity: if $x$ changes by a small amount $\delta x$, it causes $u$ to change by $\frac{du}{dx} \cdot \delta x$, which in turn causes $y$ to change by $\frac{dy}{du} \cdot \frac{du}{dx} \cdot \delta x$. The total sensitivity is the product of each stage's sensitivity.

Think of it as a chain of dominoes. A small push on the first domino ($\delta x$) gets amplified or attenuated at each stage. The chain rule tells us the total amplification from one end of the chain to the other, which is just the product of each stage's individual amplification. This multiplicative structure is central to understanding both why backpropagation works and why deep networks can suffer from vanishing or exploding gradients.

For a chain of three functions $y = f(g(h(x)))$, the chain rule extends naturally:

$$\frac{dy}{dx} = \frac{dy}{df} \cdot \frac{df}{dg} \cdot \frac{dg}{dx}$$

This pattern extends to any depth. For a network with $n$ layers, the gradient of the loss with respect to the first layer's weights involves multiplying $n$ local gradients together. This is why deep networks can suffer from vanishing or exploding gradients: if each local gradient is slightly less than 1, their product shrinks exponentially; if each is slightly greater than 1, the product explodes.

Concretely, suppose we have a 20-layer network and the average gradient magnitude at each layer is 0.9. The gradient at the first layer would be multiplied by $0.9^{20} \approx 0.12$, reducing it to about 12% of the output gradient. That might seem manageable, but with 50 layers and an average gradient of 0.7, we get $0.7^{50} \approx 1.3 \times 10^{-8}$: essentially zero. The first layers of the network would receive essentially no learning signal.

Neural networks work with vectors and matrices, so we need the multivariate form. If $\mathbf{z} = \mathbf{f}(\mathbf{y})$ and $\mathbf{y} = \mathbf{g}(\mathbf{x})$, the chain rule becomes:

$$\frac{\partial z_i}{\partial x_k} = \sum_j \frac{\partial z_i}{\partial y_j} \cdot \frac{\partial y_j}{\partial x_k}$$

where:

• $\frac{\partial z_i}{\partial x_k}$: the partial derivative of the $i$-th output component with respect to the $k$-th input component
• $\frac{\partial z_i}{\partial y_j}$: the partial derivative of $z_i$ with respect to the $j$-th intermediate variable
• $\frac{\partial y_j}{\partial x_k}$: the partial derivative of $y_j$ with respect to the $k$-th input

In matrix form, this is exactly a matrix product: if $\mathbf{J}_f$ and $\mathbf{J}_g$ are the Jacobian matrices of $f$ and $g$ respectively, then the Jacobian of the composition is:

$$\mathbf{J}_{f \circ g} = \mathbf{J}_f \cdot \mathbf{J}_g$$

This matrix multiplication is the vectorized form of the chain rule, and it is what makes backpropagation efficient. The sum over $j$ in the scalar form becomes a dot product, and repeating it for all $(i, k)$ pairs gives a matrix product. Gradient computation can therefore be done with the same fast matrix operations that power the forward pass.