Backpropagation Through Time › Unrolling the RNN

To understand BPTT, you first need a clear picture of what "unrolling" means, why it is conceptually useful, and what it costs in terms of computation and memory.

A standard RNN update at timestep $t$ is:

$$\mathbf{h}_t = \tanh(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b}_h)$$

where:

• $\mathbf{h}_t \in \mathbb{R}^d$: the hidden state at timestep $t$, a $d$-dimensional vector that summarizes sequence history up to that point
• $\mathbf{h}_{t-1} \in \mathbb{R}^d$: the hidden state at the previous timestep, carrying information forward
• $\mathbf{W}_{hh} \in \mathbb{R}^{d \times d}$: the recurrent weight matrix, applied to the previous hidden state at every step
• $\mathbf{W}_{xh} \in \mathbb{R}^{d \times k}$: the input weight matrix, mapping inputs to the hidden space
• $\mathbf{x}_t \in \mathbb{R}^k$: the input vector at timestep $t$
• $\mathbf{b}_h \in \mathbb{R}^d$: the bias vector
• $\tanh$: the hyperbolic tangent activation applied elementwise, squashing values to $(-1, 1)$

This equation describes a computation that is deeply self-referential: to compute $\mathbf{h}_t$, you need $\mathbf{h}_{t-1}$. To compute $\mathbf{h}_{t-1}$, you need $\mathbf{h}_{t-2}$. The chain extends all the way back to $\mathbf{h}_0$, the initial hidden state (typically a zero vector). This is why RNNs are powerful: they can, in principle, let information from the very beginning of a sequence influence predictions at the very end.

When you express an RNN as a recurrence relation, you are describing a loop: the same operation executed repeatedly, each time consuming the hidden state from the previous iteration. This compact representation is convenient for describing the model, but it is not the right representation for computing gradients.

Gradient computation requires a directed acyclic graph (DAG). The chain rule can only traverse a graph that does not have cycles, because cycles would create circular dependencies in the gradient equations. An RNN's loop is inherently cyclic: $\mathbf{h}_t$ depends on $\mathbf{h}_{t-1}$ which depends on $\mathbf{h}_{t-2}$, and so on. The way to resolve this is to expand the loop along the time dimension and treat each step as a distinct node in the graph.

Unrolling makes this expansion explicit. Instead of showing the RNN as a loop that re-uses a single node, you draw out a separate copy of the computation for each timestep. For a sequence of length $T$, you get a graph with $T$ hidden state nodes connected in a chain: $\mathbf{h}_1 \to \mathbf{h}_2 \to \cdots \to \mathbf{h}_T$. Each node receives the previous hidden state and the current input and produces the next hidden state. The weight matrices $\mathbf{W}_{hh}$ and $\mathbf{W}_{xh}$ are shared across all copies, meaning they are the same physical parameters, but the intermediate values they produce at each step are all distinct and stored separately.

This is an important subtlety. Unrolling does not create new parameters; it creates new nodes in the computation graph that reference the same parameters. The implication for gradient computation is that each weight participates in $T$ computation paths, and its total gradient must account for all of them.

An output $\hat{\mathbf{y}}_t$ is computed from $\mathbf{h}_t$ at each timestep (or at the final timestep only, depending on the task):

$$\hat{\mathbf{y}}_t = \mathbf{W}_{hy}\mathbf{h}_t + \mathbf{b}_y$$

where:

• $\mathbf{W}_{hy} \in \mathbb{R}^{c \times d}$: the output weight matrix, mapping hidden state to output space (e.g., a vocabulary in language modeling)
• $\mathbf{b}_y \in \mathbb{R}^c$: the output bias
• $c$: the output dimensionality (e.g., vocabulary size)

In many applications, an output is produced at every timestep. Language modeling is the canonical example: at each position in a sentence, the model predicts the next word. In sequence classification, outputs are produced only at the final step. In sequence-to-sequence models, outputs appear only during the decoding phase. BPTT applies in all of these cases, though the gradient structure differs slightly depending on where losses originate.

The total loss over a sequence is the sum of per-timestep losses:

$$\mathcal{L} = \sum_{t=1}^{T} \mathcal{L}_t$$

where:

• $\mathcal{L}$: total sequence loss
• $\mathcal{L}_t$: loss at timestep $t$, measuring prediction error at that step
• $T$: sequence length

The additivity of the loss is important: it means the gradient of the total loss is the sum of gradients from each individual timestep. Each $\mathcal{L}_t$ only directly depends on $\hat{\mathbf{y}}_t$, which depends on $\mathbf{h}_t$. But $\mathbf{h}_t$ is a function of all earlier hidden states and therefore of all earlier inputs and all applications of $\mathbf{W}_{hh}$. So even though $\mathcal{L}_t$ does not directly reference steps before $t$, its gradient with respect to $\mathbf{W}_{hh}$ reaches all the way back through the entire history.