RNN Architecture › Summary

This chapter covered the core ideas behind recurrent neural networks:

• Recurrent connections introduce memory into the network by passing the hidden state from one time step to the next, enabling context-sensitive processing of sequences.
• The Elman RNN computes $\mathbf{h}_t = \tanh(\mathbf{W}_{hh}\,\mathbf{h}_{t-1} + \mathbf{W}_{xh}\,\mathbf{x}_t + \mathbf{b}_h)$, with $\tanh$ as the squashing nonlinearity and separate weight matrices for the recurrent and input connections.
• Unrolling through time converts the cyclic RNN graph into a directed acyclic graph of depth $T$, making gradient computation possible via Backpropagation Through Time (BPTT). Truncated BPTT segments this unrolling to make training on long sequences tractable.
• Parameter sharing means the same weights apply at every time step. This gives RNNs positional generalization, efficiency with variable-length sequences, and the ability to handle sequences longer than those seen during training.
• Three core configurations: many-to-one for sequence classification (read all, output once), one-to-many for generation (receive a seed, output a sequence), and many-to-many for sequence labeling (output at every step) or encoder-decoder translation.
• Bidirectional RNNs run a forward and a backward RNN over the sequence and concatenate their hidden states, enabling each position to incorporate both left and right context.
• Stacked RNNs build a hierarchy of representations by feeding the hidden states of one RNN layer as input to the next, enabling more abstract sequential patterns to emerge in higher layers.
• Language modeling with an RNN predicts the probability of each next token conditioned on all prior tokens, trained with cross-entropy loss. Perplexity measures how confident the model is on held-out data.
• Teacher forcing provides ground-truth tokens as inputs during training, improving stability and convergence speed at the cost of a train/inference mismatch called exposure bias.
• Hidden state dimensionality is the primary capacity hyperparameter. The quadratic cost of $\mathbf{W}_{hh}$ makes large hidden sizes expensive.
• PyTorch's nn.RNN provides an efficient implementation with batch_first=True making it easy to integrate into standard training loops, with support for stacking (num_layers) and bidirectionality (bidirectional).
• Practical engineering involves handling variable-length sequences with packed tensors, gradient clipping for training stability, and careful design of the stateful hidden state transfer for truncated BPTT.
• Limitations include vanishing gradients (making long-range dependencies hard to learn), the fixed-size memory bottleneck, and the sequential nature of computation (preventing parallelization across time steps). These limitations motivated LSTMs, GRUs, and ultimately the transformer.

The next chapter explores Backpropagation Through Time in detail, analyzing exactly how gradients flow through the unrolled graph and why long sequences pose fundamental challenges for vanilla RNNs.