RNN Architecture › RNN Architecture

How do you build a neural network that reads a sequence? A feedforward network processes each input independently, treating it as if nothing came before or after it. That works fine for classifying a single image, but language does not work that way. The meaning of a word depends on what surrounds it. The next word in a sentence depends on everything preceding it. Time series data, speech, and DNA sequences all share this property: order matters, and history matters.

Recurrent Neural Networks (RNNs) solve this by introducing a mechanism for carrying information forward through time. At each step in the sequence, the network reads the current input and updates a hidden state that summarizes everything it has processed so far. That hidden state is then passed into the next step, creating a feedback loop (the recurrence) that gives the architecture its name.

This chapter builds your understanding of RNN architecture from the ground up. We start with the intuition behind recurrent connections and the role of the hidden state as memory. We then formalize the Elman RNN equations, explain how to unroll an RNN through time into a computational graph, and clarify why parameter sharing across time steps makes the architecture so efficient. You will see how the same architecture adapts to many-to-one classification, one-to-many generation, and many-to-many sequence labeling. We will implement a character-level language model in PyTorch using nn.RNN and explore the teacher forcing training strategy. We will also look at bidirectional RNNs, stacked RNNs, and the practical engineering decisions that arise when deploying RNNs on real tasks. By the end, you will have a complete, working mental model of how RNNs process sequences and why they were the dominant approach to sequence modeling before transformers arrived.