Backpropagation Through Time › A Worked Example

Derivations and code can make BPTT seem abstract. Working through a tiny numerical example by hand is the fastest way to build intuition about what the equations compute, where the values come from, and why the backward pass has to visit timesteps in reverse order.

Let's trace through BPTT on a minimal example to make the mechanics concrete.

Consider an RNN with hidden dimension $d = 2$, processing a sequence of three steps. The recurrent weight matrix is:

$$\mathbf{W}_{hh} = \begin{pmatrix} 0.5 & 0.1 \\ 0.2 & 0.5 \end{pmatrix}$$

After the forward pass, the hidden states are:

$$\mathbf{h}_0 = \begin{pmatrix} 0 \\ 0 \end{pmatrix}, \quad \mathbf{h}_1 = \begin{pmatrix} 0.3 \\ 0.5 \end{pmatrix}, \quad \mathbf{h}_2 = \begin{pmatrix} 0.4 \\ 0.6 \end{pmatrix}, \quad \mathbf{h}_3 = \begin{pmatrix} 0.5 \\ 0.7 \end{pmatrix}$$

The error signals flowing back from the output layer (local gradients from each timestep's loss) are:

$$\frac{\partial \mathcal{L}_1}{\partial \mathbf{h}_1} = \begin{pmatrix} 0.1 \\ -0.2 \end{pmatrix}, \quad \frac{\partial \mathcal{L}_2}{\partial \mathbf{h}_2} = \begin{pmatrix} 0.3 \\ -0.1 \end{pmatrix}, \quad \frac{\partial \mathcal{L}_3}{\partial \mathbf{h}_3} = \begin{pmatrix} -0.2 \\ 0.4 \end{pmatrix}$$

These local gradients come from the output layer: specifically, they are the derivatives of each timestep's loss $\mathcal{L}_t$ with respect to the hidden state $\mathbf{h}_t$ at that step. In practice, this derivative is computed by backpropagating through the output weight matrix $\mathbf{W}_{hy}$ and then through the softmax (if applicable). For this example, we take them as given to focus on the backward-through-time portion.

Step 1: Initialize $\boldsymbol{\delta}_3$ at the last timestep

The backward pass starts from the final timestep. The initial error signal is simply the local gradient:

$$\boldsymbol{\delta}_3 = \frac{\partial \mathcal{L}_3}{\partial \mathbf{h}_3} = \begin{pmatrix} -0.2 \\ 0.4 \end{pmatrix}$$

The tanh derivative (elementwise) at timestep 3 is:

$$\mathbf{g}_3 = \begin{pmatrix} 1 - 0.5^2 \\ 1 - 0.7^2 \end{pmatrix} = \begin{pmatrix} 1 - 0.25 \\ 1 - 0.49 \end{pmatrix} = \begin{pmatrix} 0.75 \\ 0.51 \end{pmatrix}$$

The modified error (tanh derivative applied before backpropagating) is:

$$\boldsymbol{\delta}_3 \odot \mathbf{g}_3 = \begin{pmatrix} -0.2 \times 0.75 \\ 0.4 \times 0.51 \end{pmatrix} = \begin{pmatrix} -0.150 \\ 0.204 \end{pmatrix}$$

Step 2: Propagate backward to $\boldsymbol{\delta}_2$

The recurrence carries the error signal one step further back:

$$\boldsymbol{\delta}_2 = \frac{\partial \mathcal{L}_2}{\partial \mathbf{h}_2} + \mathbf{W}_{hh}^T (\boldsymbol{\delta}_3 \odot \mathbf{g}_3)$$

Computing the matrix-vector product:

$$\mathbf{W}_{hh}^T (\boldsymbol{\delta}_3 \odot \mathbf{g}_3) = \begin{pmatrix} 0.5 & 0.2 \\ 0.1 & 0.5 \end{pmatrix} \begin{pmatrix} -0.150 \\ 0.204 \end{pmatrix} = \begin{pmatrix} -0.075 + 0.041 \\ -0.015 + 0.102 \end{pmatrix} = \begin{pmatrix} -0.034 \\ 0.087 \end{pmatrix}$$

Note that $\mathbf{W}_{hh}^T$ is $\mathbf{W}_{hh}$ with rows and columns swapped, because we are differentiating $\mathbf{W}_{hh}\mathbf{h}$ with respect to $\mathbf{h}$.

Adding the local gradient:

$$\boldsymbol{\delta}_2 = \begin{pmatrix} 0.3 \\ -0.1 \end{pmatrix} + \begin{pmatrix} -0.034 \\ 0.087 \end{pmatrix} = \begin{pmatrix} 0.266 \\ -0.013 \end{pmatrix}$$

Step 3: Accumulate the gradient contribution to $\mathbf{W}_{hh}$

At each timestep, the contribution to $\frac{\partial \mathcal{L}}{\partial \mathbf{W}_{hh}}$ is an outer product of the modified error signal and the previous hidden state. For timestep 3:

$$\Delta \mathbf{W}_{hh}^{(3)} = (\boldsymbol{\delta}_3 \odot \mathbf{g}_3) \cdot \mathbf{h}_2^T = \begin{pmatrix} -0.150 \\ 0.204 \end{pmatrix} \begin{pmatrix} 0.4 & 0.6 \end{pmatrix} = \begin{pmatrix} -0.060 & -0.090 \\ 0.082 & 0.122 \end{pmatrix}$$

where:

• $(\boldsymbol{\delta}_3 \odot \mathbf{g}_3)$: the $2 \times 1$ modified error vector at time 3
• $\mathbf{h}_2^T$: the $1 \times 2$ previous hidden state as a row vector
• The outer product gives a $2 \times 2$ gradient matrix matching the shape of $\mathbf{W}_{hh}$

The total gradient for $\mathbf{W}_{hh}$ is the sum of these outer products across all three timesteps:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{W}_{hh}} = \Delta \mathbf{W}_{hh}^{(1)} + \Delta \mathbf{W}_{hh}^{(2)} + \Delta \mathbf{W}_{hh}^{(3)}$$

This example illustrates several important properties of BPTT. First, the backward pass must run in strict reverse order: you cannot compute $\boldsymbol{\delta}_2$ until you have computed $\boldsymbol{\delta}_3$. Second, each timestep's gradient contribution is an outer product that has exactly the shape of the weight matrix, which is why the gradient accumulation is just a simple sum. Third, the magnitude of the gradient contribution from each timestep depends both on the local error signal and on how much that signal has been attenuated by the chain of Jacobians from later steps.

It is instructive to think about what the gradient $\frac{\partial \mathcal{L}}{\partial \mathbf{W}_{hh}}$ says. The entry at position $(i, j)$ in this matrix tells you: if you increase the weight connecting hidden dimension $j$ of the previous state to hidden dimension $i$ of the current state, how does the total loss change? A large positive value means increasing that weight increases the loss. A large negative value means increasing it decreases the loss. A near-zero value means that weight has minimal effect on the loss, perhaps because the gradient signal has been attenuated by vanishing gradients along its path.

When the gradient for $\mathbf{W}_{hh}$ is very small (close to the zero matrix), gradient descent will barely update the recurrent weights. This is the vanishing gradient problem: the model cannot effectively learn from gradient signal that traveled a long path through the unrolled network. When the gradient is very large, a single gradient step will make a drastic change to the weights, destabilizing training. The next chapter examines both pathologies in rigorous detail.