RNN Architecture › Unrolling Through Time

An RNN's recurrent connection forms a loop in the graph, which complicates analysis and differentiation. The standard approach is to unroll (also called unfold) the RNN by creating one copy of the network for each time step, eliminating the cycle. The result is a deep feedforward network where depth equals the number of time steps.

For a sequence of length $T$, the unrolled RNN has $T$ copies of the hidden layer, connected in a chain:

Each hidden state $\mathbf{h}_t$ receives two inputs: $\mathbf{x}_t$ from the current position in the sequence, and $\mathbf{h}_{t-1}$ from the preceding hidden state. The same weight matrices $\mathbf{W}_{xh}$, $\mathbf{W}_{hh}$, and $\mathbf{b}_h$ are used at every copy. This is a key point: the unrolling does not create new parameters. The same transformation is applied at every time step, just to different inputs.

Unrolling has an important implication: an RNN processing a sequence of $T$ steps behaves like a feedforward network of depth $T$. This means the gradient must flow backward through $T$ layers of $\tanh$ activations during training. Consider what happens as the gradient flows from $\mathbf{h}_T$ back to $\mathbf{h}_1$. At each step, the chain rule multiplies by the Jacobian of the hidden state update, which includes $\mathbf{W}_{hh}^T$ and the derivative of $\tanh$. If the spectral radius of $\mathbf{W}_{hh}$ is less than 1, these repeated multiplications shrink the gradient toward zero. If it is greater than 1, they amplify it exponentially. This instability is the root of the vanishing and exploding gradient problems, which the next chapter analyzes in mathematical detail.

In practice, PyTorch and other frameworks perform this unrolling automatically through their dynamic computation graph (autograd) system. You do not manually unroll the loop; you simply call the RNN forward pass for each time step and then call loss.backward(). The framework traces the computation graph and computes gradients through the recurrent chain correctly.

For very long sequences, unrolling the full sequence and backpropagating through all $T$ steps is computationally expensive and memory-intensive. The full unrolled graph requires storing all intermediate hidden states for the backward pass, and for sequences of thousands of tokens this can exhaust GPU memory.

Truncated Backpropagation Through Time (truncated BPTT) addresses this by dividing the sequence into segments and backpropagating through each segment independently. The forward pass still carries the hidden state across segment boundaries (so the model sees information from earlier in the sequence during its forward computation), but the gradient does not flow backward across segment boundaries.

The tradeoff is between computational feasibility and gradient signal quality. With truncation length $k$, the model can only learn dependencies up to $k$ steps in the past from gradient information, even if its forward hidden state in principle carries information from further back. In practice, truncation lengths of 50 to 200 tokens are common, and the model learns to encode important information in the hidden state in ways that remain useful even when gradients are truncated.