Recurrent Neural Networks (1995) › Unfolding Through Time

One of the most illuminating ways to understand how RNNs work is through the concept of unfolding or unrolling the network through time. Although an RNN is defined by a single set of recurrent connections that loop back on themselves, we can visualize its operation over a sequence by imagining that we create a copy of the network for each time step and connect these copies in a chain. This unfolded view makes it clear how information flows through the sequence and how the hidden state is passed from one time step to the next.

In the unfolded representation, each copy of the network corresponds to one time step in the sequence. The input at time $t$ is fed into the network at that time step, and the hidden state from the previous time step $h_{t-1}$ is passed in as well. The network computes a new hidden state $h_t$ and an output $o_t$, and then passes $h_t$ forward to the next time step. This continues until the entire sequence has been processed. Although the unfolded network looks like a deep feedforward network with many layers, every layer shares the same weights. There is only one set of parameters, and they are applied repeatedly at each time step.

The unfolding perspective is more than just a visualization tool. It is also the basis for how RNNs are trained. To train an RNN, we need to compute gradients of the loss function with respect to the network's parameters, so that we can adjust those parameters to improve performance. The standard algorithm for computing gradients in neural networks is backpropagation, which applies the chain rule of calculus to propagate error signals backward through the network. For RNNs, we use a variant called backpropagation through time, or BPTT, which applies backpropagation to the unfolded network. The gradients are computed by flowing backward through the sequence, from the final time step to the first, accumulating contributions to the gradient at each step. This allows the network to learn how to update its weights to make better predictions across the entire sequence.