Backpropagation › Forward Pass and Backward Pass

With computational graphs and the chain rule in hand, we can now describe the complete backpropagation algorithm. The algorithm has two phases: a forward pass that computes outputs and stores intermediate values, and a backward pass that computes gradients using those stored values.

During the forward pass, we evaluate the network layer by layer, storing all intermediate values. These stored values are critical because the backward pass needs them to compute local gradients.

For a 2-layer network with input $\mathbf{x}$, weights $\mathbf{W}^{(1)}$, $\mathbf{W}^{(2)}$, biases $\mathbf{b}^{(1)}$, $\mathbf{b}^{(2)}$, and activation $\sigma$, the forward pass computes:

$$\begin{aligned} \mathbf{z}^{(1)} &= \mathbf{W}^{(1)} \mathbf{x} + \mathbf{b}^{(1)} \\ \mathbf{a}^{(1)} &= \sigma(\mathbf{z}^{(1)}) \\ \mathbf{z}^{(2)} &= \mathbf{W}^{(2)} \mathbf{a}^{(1)} + \mathbf{b}^{(2)} \\ \hat{\mathbf{y}} &= \text{softmax}(\mathbf{z}^{(2)}) \\ \mathcal{L} &= -\sum_k y_k \log \hat{y}_k \end{aligned}$$

where:

• $\mathbf{z}^{(l)}$: the pre-activation values at layer $l$ (also called logits)
• $\mathbf{a}^{(l)}$: the post-activation values at layer $l$
• $\hat{\mathbf{y}}$: the predicted probability distribution
• $\mathbf{y}$: the one-hot encoded target
• $\mathcal{L}$: the cross-entropy loss (scalar)

After the forward pass, we have stored $\mathbf{z}^{(1)}, \mathbf{a}^{(1)}, \mathbf{z}^{(2)}, \hat{\mathbf{y}}$. These are the "cache" that the backward pass will consume. The reason we store $\mathbf{z}^{(l)}$ specifically (and not just $\mathbf{a}^{(l)}$) is that the activation derivative $\sigma'(\mathbf{z}^{(l)})$ needs the pre-activation value, not the post-activation value. For sigmoid, though, we can recover $\sigma'(z)$ from the activation value $a$ itself using $\sigma'(z) = a(1-a)$, so storing either is fine.

The backward pass applies the chain rule in reverse, starting from the loss $\mathcal{L}$ and working backward through each operation.

Gradient of loss with respect to output. We start with $\frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(2)}}$. For softmax + cross-entropy, this has the elegant form:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(2)}} = \hat{\mathbf{y}} - \mathbf{y}$$

This says: the gradient at the output layer is simply the difference between prediction and target. Larger errors produce larger gradients. This elegant form is not a coincidence: the cross-entropy loss was designed to pair with softmax precisely because the combination yields this clean gradient formula. This pairing is sometimes called the "softmax-cross-entropy trick," and it avoids numerical instability that would arise from computing the softmax and cross-entropy gradients separately.

Gradient through the second layer. Using the matrix multiply gradient rules:

$$\begin{aligned} \frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(2)}} &= \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(2)}} \cdot (\mathbf{a}^{(1)})^T \\ \frac{\partial \mathcal{L}}{\partial \mathbf{b}^{(2)}} &= \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(2)}} \\ \frac{\partial \mathcal{L}}{\partial \mathbf{a}^{(1)}} &= (\mathbf{W}^{(2)})^T \cdot \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(2)}} \end{aligned}$$

Gradient through the activation function. Passing through $\sigma$:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(1)}} = \frac{\partial \mathcal{L}}{\partial \mathbf{a}^{(1)}} \odot \sigma'(\mathbf{z}^{(1)})$$

where $\sigma'(\mathbf{z}^{(1)})$ is the elementwise derivative of the activation function evaluated at the stored pre-activation values.

Gradient through the first layer:

$$\begin{aligned} \frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(1)}} &= \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(1)}} \cdot \mathbf{x}^T \\ \frac{\partial \mathcal{L}}{\partial \mathbf{b}^{(1)}} &= \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(1)}} \end{aligned}$$

The pattern is clear: at each layer, we multiply the upstream gradient by the transpose of each weight matrix to flow backward through layers, then multiply elementwise by the activation derivative at the nonlinearity, then pass the result further back. The upstream gradient arrives from the right (closer to the loss), we compute local contributions for this layer's parameters, and we send a modified gradient leftward to the previous layer.