Backpropagation › A Worked Example: Two-Layer Network

Let's trace backpropagation through a complete two-layer network on a single training example to make the abstract formulas concrete. Walking through this computation by hand builds the kind of intuition that helps you debug real implementations.

We'll use a tiny network with:

• Input: $\mathbf{x} = [1, 2]^T$ (2 features)
• Layer 1: 3 hidden units with sigmoid activation
• Layer 2: 2 output units with softmax
• Target: class 0, so $\mathbf{y} = [1, 0]^T$

The weights and biases (chosen for illustration):

$$\mathbf{W}^{(1)} = \begin{pmatrix} 0.1 & 0.2 \\ 0.3 & 0.4 \\ 0.5 & 0.6 \end{pmatrix}, \quad \mathbf{b}^{(1)} = \begin{pmatrix} 0 \\ 0 \\ 0 \end{pmatrix}$$ $$\mathbf{W}^{(2)} = \begin{pmatrix} 0.7 & 0.8 & 0.9 \\ 0.1 & 0.2 & 0.3 \end{pmatrix}, \quad \mathbf{b}^{(2)} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}$$

Step 1, compute pre-activations in layer 1:

$$\mathbf{z}^{(1)} = \mathbf{W}^{(1)} \mathbf{x} = \begin{pmatrix} 0.1 \cdot 1 + 0.2 \cdot 2 \\ 0.3 \cdot 1 + 0.4 \cdot 2 \\ 0.5 \cdot 1 + 0.6 \cdot 2 \end{pmatrix} = \begin{pmatrix} 0.5 \\ 1.1 \\ 1.7 \end{pmatrix}$$

Step 2, apply sigmoid activation:

$$\mathbf{a}^{(1)} = \sigma(\mathbf{z}^{(1)}) = \begin{pmatrix} 0.622 \\ 0.750 \\ 0.846 \end{pmatrix}$$

Step 3, compute output logits:

$$\mathbf{z}^{(2)} = \mathbf{W}^{(2)} \mathbf{a}^{(1)} \approx \begin{pmatrix} 2.394 \\ 0.560 \end{pmatrix}$$

Step 4, apply softmax:

$$\hat{\mathbf{y}} = \text{softmax}(\mathbf{z}^{(2)}) \approx \begin{pmatrix} 0.853 \\ 0.147 \end{pmatrix}$$

Step 5, compute cross-entropy loss:

$$\mathcal{L} = -\log(\hat{y}_0) = -\log(0.853) \approx 0.159$$

This is a relatively small loss because the network already predicts the correct class (class 0) with high confidence (85.3%). In the backward pass, we'll see that this small loss produces correspondingly small gradients, meaning the network weights change very little for this example.

Step 1, output gradient. For softmax + cross-entropy:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(2)}} = \hat{\mathbf{y}} - \mathbf{y} = \begin{pmatrix} 0.853 - 1 \\ 0.147 - 0 \end{pmatrix} = \begin{pmatrix} -0.147 \\ 0.147 \end{pmatrix}$$

The network predicted class 0 with 85.3% confidence, and the target is class 0, so the gradient is small. If the prediction were wrong (say, 0.1 confidence on the correct class), the gradient would be $\hat{\mathbf{y}} - \mathbf{y} \approx [-0.9, 0.9]^T$, much larger. This is the self-correcting property of cross-entropy: bigger mistakes generate bigger gradients, which cause bigger parameter updates.

Step 2, gradients for layer 2 weights:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(2)}} = \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(2)}} (\mathbf{a}^{(1)})^T = \begin{pmatrix} -0.147 \\ 0.147 \end{pmatrix} \begin{pmatrix} 0.622 & 0.750 & 0.846 \end{pmatrix}$$ $$= \begin{pmatrix} -0.091 & -0.110 & -0.124 \\ 0.091 & 0.110 & 0.124 \end{pmatrix}$$

Each entry $\frac{\partial \mathcal{L}}{\partial W^{(2)}_{ij}}$ is the product of the upstream gradient at output $i$ and the activation at hidden unit $j$. This outer product structure is fundamental: the weight gradient is proportional to how much that weight "contributed" (via the activation magnitude) multiplied by how much the output was "wrong" (the upstream gradient).

Step 3, propagate gradient back through layer 2:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{a}^{(1)}} = (\mathbf{W}^{(2)})^T \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(2)}} \approx \begin{pmatrix} -0.088 \\ -0.102 \\ -0.117 \end{pmatrix}$$

Step 4, gradient through sigmoid. We need $\sigma'(\mathbf{z}^{(1)}) = \mathbf{a}^{(1)} \odot (1 - \mathbf{a}^{(1)})$:

$$\sigma'(\mathbf{z}^{(1)}) = \begin{pmatrix} 0.622 \cdot 0.378 \\ 0.750 \cdot 0.250 \\ 0.846 \cdot 0.154 \end{pmatrix} = \begin{pmatrix} 0.235 \\ 0.188 \\ 0.130 \end{pmatrix}$$ $$\frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(1)}} = \frac{\partial \mathcal{L}}{\partial \mathbf{a}^{(1)}} \odot \sigma'(\mathbf{z}^{(1)}) \approx \begin{pmatrix} -0.021 \\ -0.019 \\ -0.015 \end{pmatrix}$$

Notice how the sigmoid derivative compressed the gradients further. The activations at this layer are relatively large (0.6 to 0.8), which puts them in the mildly-saturated region of the sigmoid, so the derivatives are between 0.13 and 0.24. The gradient coming out of the activation layer is roughly 3-4x smaller than what came in, which is the vanishing gradient effect in action even on this single-layer example.

Step 5, gradients for layer 1 weights:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(1)}} = \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(1)}} \mathbf{x}^T$$

The worked example shows the key pattern: gradients start at the output, get multiplied by the transpose of each weight matrix to flow backward through layers, and get multiplied elementwise by activation derivatives at each nonlinearity. At each step, the gradient is the "responsibility signal" telling each parameter how much it contributed to the final error.