RNN Architecture and Hidden-State Memory

How do you build a neural network that reads a sequence? A feedforward network processes each input independently, treating it as if nothing came before or after it. That works fine for classifying a single image, but language does not work that way. The meaning of a word depends on what surrounds it. The next word in a sentence depends on everything preceding it. Time series data, speech, and DNA sequences all share this property: order matters, and history matters.

Recurrent Neural Networks (RNNs) solve this by introducing a mechanism for carrying information forward through time. At each step in the sequence, the network reads the current input and updates a hidden state that summarizes everything it has processed so far. That hidden state is then passed into the next step, creating a feedback loop (the recurrence) that gives the architecture its name.

This chapter builds your understanding of RNN architecture from the ground up. We start with the intuition behind recurrent connections and the role of the hidden state as memory. We then formalize the Elman RNN equations, explain how to unroll an RNN through time into a computational graph, and clarify why parameter sharing across time steps makes the architecture so efficient. You will see how the same architecture adapts to many-to-one classification, one-to-many generation, and many-to-many sequence labeling. We will implement a character-level language model in PyTorch using nn.RNN and explore the teacher forcing training strategy. We will also look at bidirectional RNNs, stacked RNNs, and the practical engineering decisions that arise when deploying RNNs on real tasks. By the end, you will have a complete, working mental model of how RNNs process sequences and why they were the dominant approach to sequence modeling before transformers arrived.

Part X covered feedforward networks and the building blocks of deep learning. Those architectures have a fixed input size. You feed in a vector, and the network produces an output. To handle multiple inputs, you stack layers, but each forward pass is stateless: no information persists between calls. The network does not remember what it processed one second ago.

Consider trying to predict the next word in a sentence. The word "bank" in "She walked to the river bank" means something completely different from "bank" in "She deposited money at the bank." Context resolves ambiguity. A feedforward network processing only a single-word window would see "bank" in isolation and have no mechanism to access the surrounding context. You could widen the window, feeding in five words at once, but this forces a fixed context length and creates a different input structure for every window size. Worse, it requires a separate set of parameters for every position in the window: the weights for the first word slot are entirely independent of the weights for the fifth word slot. The network cannot share what it learns about word patterns across positions.

This position-specific parameterization means the model must learn the same linguistic patterns independently for every context slot. If it learns that a certain word appearing near the start of a sentence suggests a certain topic, that knowledge does not automatically transfer to predicting topic when that word appears in the middle of the sentence. This is a fundamental mismatch between how language works and how fixed-window feedforward networks represent it.

RNNs avoid this limitation by processing sequences step by step and maintaining a running summary. Rather than requiring a fixed-size window, they thread context through a hidden state that grows richer as more of the sequence is consumed. The network architecture stays the same regardless of sequence length; only the number of recurrent steps changes. A sentence of 5 words and a paragraph of 500 words go through the same model, just for different numbers of steps.

The idea of maintaining state across computations has roots in control theory and classical automata theory. A finite state machine reads a sequence of symbols and transitions between states at each step, with the current state encoding everything relevant from the past. An RNN is a continuous, parameterized, differentiable version of this idea: instead of a discrete state from a finite set, it maintains a continuous vector state learned through gradient descent.

The key innovation in an RNN is the recurrent connection: the hidden state at time step $t$ depends on the current input $\mathbf{x}_t$ and on the hidden state from the previous time step $\mathbf{h}_{t-1}$.

Think of it as reading a book. Each time you finish a sentence, you carry a mental summary forward. That summary informs how you interpret the next sentence. You do not re-read the entire book to understand each new sentence; instead, you rely on the compressed representation in your memory. The hidden state plays exactly this role.

The hidden state is not directly observed or directly supervised. It is an internal representation that the training process shapes to be useful for the downstream prediction task. If the task is language modeling, the hidden state will learn to encode information about grammatical structure, topic, and recent word history. If the task is sentiment classification, the hidden state will accumulate evidence about the emotional tone of the text.

Concretely, the recurrent connection looks like a cycle in the network graph. The output of the hidden layer feeds back into itself at the next time step. This single cycle is what separates an RNN from a feedforward network and what allows it to process sequences of arbitrary length. The cycle is mathematically well-defined because the flow is across time: at step $t$ the network reads $\mathbf{h}_{t-1}$ (already computed) and produces $\mathbf{h}_t$ (the new output). There is no ambiguous circular dependency within a single time step.

The recurrent connection also means the network's behavior is dynamic. For the same current input $\mathbf{x}_t$, the network will produce a different hidden state depending on what it has seen before. The word "not" followed by "happy" should drive a different hidden state than "not" followed by "guilty", and the recurrent weights learn to encode exactly this kind of history-sensitive behavior.

The most basic RNN, introduced by Jeffrey Elman in 1990, computes the hidden state and output at each time step with two equations.

The hidden state update is:

where:

• $\mathbf{h}_t \in \mathbb{R}^{H}$: the hidden state at time step $t$, a vector of dimension $H$ (the hidden size)
• $\mathbf{h}_{t-1} \in \mathbb{R}^{H}$: the hidden state from the previous time step (initialized to a zero vector at $t=0$)
• $\mathbf{x}_t \in \mathbb{R}^{D}$: the input at time step $t$, a vector of dimension $D$ (the input size)
• $\mathbf{W}_{hh} \in \mathbb{R}^{H \times H}$: the recurrent weight matrix connecting the previous hidden state to the current one
• $\mathbf{W}_{xh} \in \mathbb{R}^{H \times D}$: the input weight matrix connecting the current input to the hidden state
• $\mathbf{b}_h \in \mathbb{R}^{H}$: the hidden state bias vector
• $\tanh$: the hyperbolic tangent activation function, applied element-wise to squash the values to $(-1, 1)$

The output at time step $t$ (when producing a prediction at that step) is:

where:

• $\mathbf{y}_t \in \mathbb{R}^{V}$: the output vector at time step $t$ (often called logits before softmax)
• $\mathbf{W}_{hy} \in \mathbb{R}^{V \times H}$: the output weight matrix
• $\mathbf{b}_y \in \mathbb{R}^{V}$: the output bias vector
• $V$: the output size (e.g., vocabulary size for language modeling)

The two equations capture the complete computation of an RNN. The first equation is the memory update: it blends the current input with the running history. The second equation is the prediction: it projects the memory into output space. Separating memory from prediction makes the architecture conceptually tidy and practically modular.

The $\tanh$ activation is the traditional choice for the hidden state. It maps large positive values toward $+1$ and large negative values toward $-1$, which helps keep the hidden state within a bounded range. As we will see in the chapter on Vanishing Gradients, $\tanh$ also has desirable gradient properties compared to unbounded activations.

You might wonder why RNNs use $\tanh$ rather than ReLU, which works well in feedforward networks. The answer lies in the recurrence. In a feedforward network, each layer is applied once. In an RNN, the same $\mathbf{W}_{hh}$ matrix is multiplied at every time step. If the hidden state can grow without bound (as it would with ReLU and spectral radius greater than 1), gradients explode. The $\tanh$ squashing function forces the hidden state to remain in $(-1, 1)^H$, which dampens but does not eliminate the explosion problem.

There is a deeper issue with ReLU in recurrent settings. The derivative of ReLU is either 0 or 1, which means that when flowing gradients backward through many steps, you either pass them unchanged (if all units are active) or kill them entirely (if units die). The $\tanh$ derivative is always positive, between 0 and 1, meaning gradients shrink but never vanish abruptly. This gives the training process a somewhat more predictable gradient signal, although the shrinking still accumulates to the vanishing gradient problem over long sequences.

Some research has explored using identity initialization (initializing $\mathbf{W}_{hh}$ to the identity matrix) combined with ReLU in what are called IndRNNs (Independently Recurrent Neural Networks). By constraining each hidden unit to only connect to itself in the recurrent step, the instability from matrix multiplication is avoided. But the vanilla Elman RNN with $\tanh$ remains the standard baseline, and the Vanishing Gradients chapter explores this tension in detail and explains why LSTMs and GRUs were designed to address it.

At the very first time step, there is no previous hidden state. The convention is to initialize $\mathbf{h}_0 = \mathbf{0}$: a zero vector of dimension $H$. This is the most common choice, though it is not the only one.

In some settings, the initial hidden state is learned as a parameter. This is particularly useful when the model is conditioned on a context that should influence the entire sequence, such as a class label or a conditioning image embedding. In encoder-decoder architectures (which Part XII covers), the encoder's final hidden state becomes the decoder's initial hidden state, and this transfer of context is the mechanism by which the encoder communicates its understanding of the input to the decoder.

Zero initialization is equivalent to saying "the network starts with no information about the sequence." Whether this is the right inductive bias depends on the task. For language modeling over independent documents, it is perfectly fine: each document starts fresh. For modeling a stream of data with continuity across chunks, you may want to carry the final hidden state from one batch forward as the initial state of the next batch. This technique, called stateful RNN processing or truncated BPTT, is covered in more detail in the Backpropagation Through Time chapter.

In practice, many deep learning libraries (including PyTorch) compute the hidden state update using a single matrix multiplication rather than two:

where $\mathbf{W} \in \mathbb{R}^{H \times (H + D)}$ is the horizontal concatenation of $\mathbf{W}_{hh}$ and $\mathbf{W}_{xh}$, and $[\mathbf{h}_{t-1};\,\mathbf{x}_t]$ is the vertical concatenation of the two vectors. This single matrix-vector product is computationally equivalent to the two-matrix formulation but more efficient to execute as a single GPU kernel call.

The size of the hidden state $H$ is a core hyperparameter. It controls the capacity of the RNN's memory. A larger $H$ gives the network more room to store information about the sequence, but it also increases the number of parameters and computational cost.

The dominant parameter cost comes from the recurrent matrix $\mathbf{W}_{hh} \in \mathbb{R}^{H \times H}$, which scales quadratically with $H$. Doubling the hidden size quadruples the number of recurrent parameters. For this reason, practical RNNs typically use hidden sizes between 128 and 2048, depending on the task and available compute.

The total parameter count for an Elman RNN is:

where $H^2$ comes from $\mathbf{W}_{hh}$, $H \cdot D$ from $\mathbf{W}_{xh}$, $H$ from $\mathbf{b}_h$, $V \cdot H$ from $\mathbf{W}_{hy}$, and $V$ from $\mathbf{b}_y$.

The stacked bars make the growth dynamic clear. Across the hidden sizes shown, the output weight matrix $\mathbf{W}_{hy}$ dominates because the vocabulary size ($V = 10{,}000$) is large. The recurrent matrix $\mathbf{W}_{hh}$ does not overtake it in this range, but its share grows steadily because it scales quadratically with $H$. This asymmetry explains why practitioners often use techniques like weight tying (sharing the embedding matrix with $\mathbf{W}_{hy}$) to reduce the output matrix cost.

The choice of hidden size involves tradeoffs between representational capacity, parameter count, and computational speed. A rule of thumb is to start with hidden sizes of 128 or 256 for character-level models on small datasets, 512 to 1024 for word-level models on medium datasets, and 1024 to 2048 for larger-scale applications. However, the optimal size always depends on the complexity of the task, the amount of training data, and the available compute budget.

One important observation is that the hidden state does not need to be as large as the vocabulary to achieve good language modeling. The hidden state is a continuous vector, and even a 128-dimensional vector can encode rich probabilistic information about possible next tokens through the linear projection into vocabulary space. The compression that happens between $H$-dimensional hidden state and $V$-dimensional vocabulary logits is one of the key ways RNNs generalize: they must learn a compact representation of language that explains the data, rather than memorizing explicit transition probabilities.

An RNN's recurrent connection forms a loop in the graph, which complicates analysis and differentiation. The standard approach is to unroll (also called unfold) the RNN by creating one copy of the network for each time step, eliminating the cycle. The result is a deep feedforward network where depth equals the number of time steps.

For a sequence of length $T$, the unrolled RNN has $T$ copies of the hidden layer, connected in a chain:

Each hidden state $\mathbf{h}_t$ receives two inputs: $\mathbf{x}_t$ from the current position in the sequence, and $\mathbf{h}_{t-1}$ from the preceding hidden state. The same weight matrices $\mathbf{W}_{xh}$, $\mathbf{W}_{hh}$, and $\mathbf{b}_h$ are used at every copy. This is a key point: the unrolling does not create new parameters. The same transformation is applied at every time step, just to different inputs.

Unrolling has an important implication: an RNN processing a sequence of $T$ steps behaves like a feedforward network of depth $T$. This means the gradient must flow backward through $T$ layers of $\tanh$ activations during training. Consider what happens as the gradient flows from $\mathbf{h}_T$ back to $\mathbf{h}_1$. At each step, the chain rule multiplies by the Jacobian of the hidden state update, which includes $\mathbf{W}_{hh}^T$ and the derivative of $\tanh$. If the spectral radius of $\mathbf{W}_{hh}$ is less than 1, these repeated multiplications shrink the gradient toward zero. If it is greater than 1, they amplify it exponentially. This instability is the root of the vanishing and exploding gradient problems, which the next chapter analyzes in mathematical detail.

In practice, PyTorch and other frameworks perform this unrolling automatically through their dynamic computation graph (autograd) system. You do not manually unroll the loop; you simply call the RNN forward pass for each time step and then call loss.backward(). The framework traces the computation graph and computes gradients through the recurrent chain correctly.

For very long sequences, unrolling the full sequence and backpropagating through all $T$ steps is computationally expensive and memory-intensive. The full unrolled graph requires storing all intermediate hidden states for the backward pass, and for sequences of thousands of tokens this can exhaust GPU memory.

Truncated Backpropagation Through Time (truncated BPTT) addresses this by dividing the sequence into segments and backpropagating through each segment independently. The forward pass still carries the hidden state across segment boundaries (so the model sees information from earlier in the sequence during its forward computation), but the gradient does not flow backward across segment boundaries.

The tradeoff is between computational feasibility and gradient signal quality. With truncation length $k$, the model can only learn dependencies up to $k$ steps in the past from gradient information, even if its forward hidden state in principle carries information from further back. In practice, truncation lengths of 50 to 200 tokens are common, and the model learns to encode important information in the hidden state in ways that remain useful even when gradients are truncated.

One of the most powerful aspects of the RNN is that the same parameters are reused at every time step. Unlike a feedforward network with separate weights for each position, the RNN applies the same transformations $\mathbf{W}_{hh}$, $\mathbf{W}_{xh}$, and $\mathbf{W}_{hy}$ regardless of where in the sequence we are.

This parameter sharing has several important consequences:

• Efficiency: The model stays compact regardless of sequence length. A sequence of 100 tokens and a sequence of 10,000 tokens use the same parameters.
• Generalization across positions: A pattern learned at position 5 generalizes to position 500 automatically. The network does not need to re-learn the same patterns for every possible position.
• Length generalization: At test time, the RNN can process sequences longer than any it saw during training, because the weights apply the same computation at every step.
• Translation invariance in time: If you shift the entire sequence by one time step, the model's behavior for each element is unchanged. This mirrors the translation invariance that convolutional networks achieve in the spatial domain, except it applies to the temporal domain.

This is in contrast to a "wide" feedforward approach where you concatenate all token embeddings into a single input vector. That approach requires a separate set of weights for every position, does not generalize across positions, and cannot handle variable-length inputs without padding to the maximum possible length.

The parameter sharing is also what makes RNNs well-suited to streaming applications. Because the model applies the same computation at every step, it can process an online stream of tokens one by one without needing to buffer the entire sequence first. Each new token updates the hidden state, and a prediction can be made immediately after the update.

The parameter sharing in RNNs has a deep theoretical connection to computation. A Turing machine executes the same finite set of rules (a program) at every step, acting on the current tape symbol and machine state to produce a new state and move the tape head. An RNN is structurally similar: it applies the same transformation at every step, taking the current input and hidden state to produce a new hidden state. This analogy motivates the theoretical result (proved by Siegelmann and Sontag in 1992) that RNNs with enough hidden units and rational weights are Turing-complete: they can in principle simulate any computation. Whether practical finite-precision RNNs trained by gradient descent can learn complex programs is a separate empirical question, but the theoretical equivalence is conceptually illuminating.

The RNN architecture is remarkably flexible. By choosing where to read inputs and where to produce outputs, you can adapt the same core equations to many different tasks. This flexibility is one of the reasons RNNs became the dominant approach to sequence modeling in the early deep learning era.

In many-to-one mode, the RNN reads an entire sequence and produces a single output from the final hidden state. The final hidden state $\mathbf{h}_T$ is a compressed representation of the whole sequence, which is then passed through a linear layer to produce the prediction:

This configuration is appropriate for tasks where you need a single label for an entire sequence:

• Sentiment analysis: classify a review as positive or negative
• Document classification: assign a topic label to a news article
• Spam detection: label an email as spam or legitimate
• Acceptability judgment: determine whether a sentence is grammatically well-formed
• Intent classification: identify the intent behind a user's utterance in a dialogue system

The intuition is that by the time the RNN has read the last word, $\mathbf{h}_T$ should contain enough information about the overall message to make a prediction. In practice, the final hidden state can have difficulty "remembering" information from the very beginning of a long sequence. This is the core limitation addressed in the Vanishing Gradients chapter, and it is part of why bidirectional RNNs (discussed below) are often preferred for classification tasks.

An important subtlety: using only the final hidden state discards the intermediate representations. An alternative is to use the average of all hidden states $\frac{1}{T}\sum_t \mathbf{h}_t$, or to take the max across positions (max pooling over time). These pooling strategies can sometimes outperform using only the final state, particularly when the relevant information is distributed throughout the sequence rather than concentrated near the end.

In one-to-many mode, the RNN receives a single input (or a conditioning vector) at the first step and then produces a sequence of outputs, with each output fed back as the input to the next step. This is the autoregressive generation pattern.

The typical setup uses an initial input $\mathbf{x}_0$ (perhaps a start-of-sequence token or a conditioning vector like an image embedding), then generates:

Applications of this configuration include:

• Image captioning: the image embedding is the initial input, and the RNN generates a caption word by word
• Music generation: a seed note or chord launches the generation of a melody
• Unconditional text generation: a start token launches free-form text
• Code generation from a specification: a short descriptor is encoded and the RNN expands it into a longer output

The generation process can use different decoding strategies. Greedy decoding always picks the token with the highest probability, which is fast but can produce suboptimal outputs. Beam search maintains a set of candidate sequences (beams) and selects the one with the highest cumulative log-probability. Temperature sampling adjusts how peaked or flat the probability distribution is. These strategies are covered in the Decoding Strategies chapter.

Many-to-many produces an output at every input step. There are two variants.

Synchronized (same-length input and output): Output $\mathbf{y}_t$ is produced from $\mathbf{h}_t$ at each step. This suits tasks where every input token needs a label, such as part-of-speech tagging or named entity recognition. The sequence labeling chapters in Part VI used CRF and HMM models for these tasks; RNNs provide an alternative that learns features directly from the text without hand-crafted feature templates.

Encoder-decoder (different-length input and output): The input sequence is first encoded into a context vector using a many-to-one RNN (the encoder), then a one-to-many RNN (the decoder) generates the output sequence from that context. This is the foundation of sequence-to-sequence models and is explored in detail in Part XII. The classic application is machine translation: the encoder reads a sentence in English and compresses it into a fixed-size context vector, and the decoder generates the corresponding French sentence one word at a time. The attention mechanism, introduced later, addresses the bottleneck of compressing an entire sentence into a single fixed-size vector.

A standard (unidirectional) RNN processes a sequence from left to right. At each position $t$, it has access only to the tokens up to and including $t$. This is appropriate for language modeling, where you should not peek at future tokens, but it is suboptimal for many other tasks.

Consider named entity recognition: to correctly label "Washington" in a sentence, it helps enormously to see what comes after it. "Washington led the troops" vs. "Washington state has excellent wines" vs. "George Washington founded the nation" each suggest very different entity types. A left-to-right RNN can only use the left context; a bidirectional RNN captures both.

The bidirectional hidden state at position $t$ is:

The combined representation $\mathbf{h}_t^{\text{bi}} \in \mathbb{R}^{2H}$ encodes context from both directions. The forward and backward RNNs have independent parameters, so the total parameter count roughly doubles compared to a unidirectional RNN with the same hidden size.

Bidirectional RNNs are widely used for:

• Named entity recognition and POS tagging: token-level classification tasks that benefit from full sequence context
• Question answering: encoding a question or context passage where both left and right context matter for each token
• Reading comprehension: attending over a passage where understanding each span requires knowledge of the whole document
• Pre-training language representations: BERT and ELMo use bidirectional architectures (though BERT uses transformers rather than RNNs)

The key limitation of bidirectional RNNs is that they require the entire sequence to be available before processing begins. They cannot be used for streaming or online scenarios where you must process and respond to tokens as they arrive. For generation tasks, unidirectional RNNs are necessary.

In PyTorch, nn.RNN supports bidirectionality through the bidirectional=True parameter. The output at each time step has shape (batch, seq_len, 2 * hidden_size) when bidirectional, where the two halves correspond to the forward and backward hidden states.

Just as feedforward networks gain representational power by stacking multiple layers, RNNs can be stacked by feeding the output hidden states of one RNN layer as the input to the next.

In a stacked (deep) RNN with $L$ layers, the hidden state at layer $\ell$ is computed as:

where $\mathbf{h}_t^{(0)} = \mathbf{x}_t$ (the input) and $\mathbf{h}_t^{(\ell-1)}$ is the hidden state from the layer below at the same time step. Each layer has its own independent set of parameters.

The motivation for stacking is hierarchy. The lower layers tend to learn local, surface-level patterns (bigrams, punctuation boundaries, character-level morphology). The higher layers can then learn more abstract, long-range patterns (topic, discourse structure, semantic relationships) on top of these local summaries. This hierarchy of representations mirrors how deep feedforward networks learn a hierarchy of features in computer vision.

In practice, 2-layer stacked RNNs are common. 3 or 4 layers are used in more demanding tasks like machine translation. Beyond 4 layers, training becomes difficult due to compounding gradient instability across both depth and time. Techniques like layer normalization (applied within each RNN step) can help stabilize deeper stacked RNNs.

PyTorch's nn.RNN(num_layers=2) implements a 2-layer stacked RNN. The output tensor has shape (batch, seq_len, hidden_size) (from the final layer), and the h_n output has shape (num_layers, batch, hidden_size) containing the final hidden state of each layer.

Dropout in recurrent networks requires care. Applying standard dropout to the hidden state $\mathbf{h}_t$ at every time step disrupts the information flow through time, which can hurt performance. The standard approach (Gal & Ghahramani, 2016) is variational dropout: use the same dropout mask for all time steps within a sequence, applied only to non-recurrent connections. This drops out connections between layers (from $\mathbf{h}_t^{(\ell-1)}$ to $\mathbf{h}_t^{(\ell)}$) but not the recurrent connections ($\mathbf{h}_{t-1}^{(\ell)}$ to $\mathbf{h}_t^{(\ell)}$). PyTorch implements this through nn.RNN(dropout=0.5, num_layers=2), which applies dropout between layers but not within the recurrent step.

Let us trace through one complete forward pass of an RNN to see the mechanics concretely. We will use a tiny vocabulary and tiny hidden size to keep the numbers tractable.

Consider a vocabulary $\{"a", "b", "c"\}$ with one-hot encodings, a hidden size $H = 2$, and the sequence "a", "b". We initialize $\mathbf{h}_0 = \mathbf{0}$.

Let us trace through the key computations step by step to understand exactly what is happening.

At step 1, the input $\mathbf{x}_a = [1, 0, 0]$ and the initial hidden state $\mathbf{h}_0 = [0, 0]$. The hidden state update computes $\mathbf{W}_{hh}\,\mathbf{h}_0 + \mathbf{W}_{xh}\,\mathbf{x}_a + \mathbf{b}_h$. Because $\mathbf{h}_0 = \mathbf{0}$, the recurrent term vanishes and $\mathbf{h}_1 = \tanh(\mathbf{W}_{xh}[:,0] + \mathbf{0})$, which is just the $\tanh$ of the first column of $\mathbf{W}_{xh}$ (corresponding to the "a" index). The hidden state after seeing "a" encodes the identity of "a" directly.

At step 2, the input is $\mathbf{x}_b = [0, 1, 0]$ and the previous hidden state is $\mathbf{h}_1$ (which depends on "a"). Now both terms contribute. The $\mathbf{W}_{hh}\,\mathbf{h}_1$ term brings forward information about "a", and the $\mathbf{W}_{xh}\,\mathbf{x}_b$ term injects information about "b". The combined pre-activation is a mixture of both, and $\tanh$ squashes the result. The hidden state $\mathbf{h}_2$ therefore encodes the fact that "b" followed "a", not just that the current token is "b".

This example uses only two tokens and two hidden units, but the same computation scales to sequences of thousands of tokens and hidden sizes of thousands of dimensions. The core insight is that the recurrent term in the hidden state update carries information from the past, blended with the current input by the weight matrices.

It is illuminating to think about what gradient information flows back through this tiny example during training. The loss is computed at the output step (using $\mathbf{h}_2$). Backpropagating through $\mathbf{W}_{hy}$ gives gradients that adjust the output projection. Backpropagating through $\mathbf{h}_2$ requires propagating through the $\tanh$ and then through $\mathbf{W}_{hh}$ and $\mathbf{W}_{xh}$ to reach $\mathbf{h}_1$. If the sequence were 10 steps long, the gradient from the loss at step 10 would need to propagate back through 10 applications of $\mathbf{W}_{hh}^T$ and 10 $\tanh$ derivative multiplications. This is where the vanishing gradient problem emerges.

A language model assigns a probability to a sequence of tokens. It can also generate new text by sampling one token at a time, each conditioned on the tokens already generated.

The RNN language model works as follows. At each time step $t$, the network receives the embedding of the current token $\mathbf{x}_t$ and updates its hidden state. The output logits $\mathbf{y}_t$ represent the unnormalized log-probabilities for all tokens in the vocabulary, and a softmax converts these to a probability distribution over the next token:

The training objective is to maximize the log-likelihood of the correct next token at every position, which is equivalent to minimizing the cross-entropy loss:

This objective is defined at every time step rather than only at the end of the sequence. The gradients flow backward through all $T$ positions, giving the model dense supervision signal. This density of supervision is one of the reasons language modeling is such a powerful pre-training objective: the model receives feedback about every token, rather than a single label for the entire sequence.

In the worked example above, we used one-hot encodings: a $V$-dimensional vector with a single 1 at the token's index. This works for tiny vocabularies but is impractical at scale. For a vocabulary of 50,000 tokens, each one-hot vector has 50,000 dimensions, most of which are zero. Multiplying $\mathbf{W}_{xh} \in \mathbb{R}^{H \times V}$ by a one-hot vector is equivalent to just selecting the corresponding column of $\mathbf{W}_{xh}$, which is computationally wasteful.

An embedding layer solves this by representing each token as a dense vector $\mathbf{e}_t \in \mathbb{R}^{E}$ where $E \ll V$ (typically $E = 64$ to $512$). The embedding lookup is implemented as a matrix $\mathbf{E} \in \mathbb{R}^{V \times E}$, and retrieving the embedding for token $i$ means fetching row $i$: $\mathbf{e}_i = \mathbf{E}[i]$. This is computationally equivalent to multiplying by a one-hot vector but implemented as an efficient index operation.

The embedding matrix is jointly trained with the rest of the model by gradient descent. Tokens that appear in similar contexts learn to have similar embeddings. The embedding layer effectively performs a learned dimensionality reduction, converting the sparse high-dimensional token index into a dense, semantically meaningful vector before feeding it into the RNN.

A natural question during training: at step $t$, should the model use its own predicted token as the next input, or should it use the ground-truth token?

Teacher forcing uses the ground-truth token at each step. During training, regardless of what the model predicts, the next input is the actual next token from the training data. This has two benefits:

• Stability: if the model makes an error early in training, it does not get compounded through the rest of the sequence, keeping gradients informative
• Speed: the model receives correct context at every step, which typically makes training converge faster

The downside is exposure bias: at inference time, the model uses its own predictions as inputs (because there is no ground truth to fall back on). If the model's predictions are imperfect, the distribution of inputs at inference differs from training. This mismatch can cause generation quality to degrade, especially for long sequences.

Exposure bias is most noticeable in text generation tasks. A model that trained with teacher forcing has never, during training, seen what happens when it generates a wrong token and must then recover. At inference time, a single early mistake shifts the distribution of inputs to a regime the model did not encounter during training, potentially causing a cascade of errors.

Several approaches attempt to bridge this gap. Scheduled sampling gradually replaces ground-truth tokens with model-generated tokens as training progresses. Minimum-risk training optimizes a sequence-level loss rather than token-level cross-entropy, but requires efficient algorithms for computing expectations over sequences. Professor forcing and curriculum learning offer other perspectives on the training/inference mismatch. Despite these alternatives, teacher forcing remains the standard approach for initial training, with refinements applied as needed.

PyTorch provides nn.RNN as a ready-to-use module. Let us implement a character-level language model from scratch, which will demonstrate all the architectural pieces in a running system.

We will work with a small corpus of text and build a vocabulary of individual characters.

A vocabulary of around 30 characters is typical for character-level models: all lowercase letters, spaces, and punctuation. Each character maps to an integer index, and integer indices will be converted to embeddings inside the model.

Character-level modeling is a useful pedagogical setting because the vocabulary is small and tractable, the task (predicting the next character) is well-defined, and you can directly inspect what the model generates. The challenge for the model is that it must learn word boundaries, morphology, and even syntax purely from the character stream, without any explicit tokenization.

For language modeling, the input at each step is a token, and the target at each step is the next token. We create overlapping windows of a fixed sequence length.

Each training window has seq_len input characters and seq_len target characters, shifted by one position. At every position, the model must predict the next character from the current hidden state. For example, if the input window is "the quick brow", the target window is "he quick brown". The model is simultaneously trained to predict each character given the preceding ones.

The dense supervision from this formulation is important. At seq_len = 30 and batch size 16, each forward pass generates 480 predictions (30 per sequence times 16 sequences), each supervised by a cross-entropy loss. This dense signal is what makes language modeling such an effective training objective.

The model has three components. The embedding layer converts integer token indices into dense vectors, equivalent to looking up a row in an embedding matrix. The nn.RNN layer processes the embedded sequence and returns both the sequence of hidden states (one per time step) and the final hidden state. The linear layer projects each hidden state to vocabulary logits. With batch_first=True, all tensors have the batch dimension first, which is the conventional layout for training loops.

Notice that the forward method accepts an optional initial hidden state h. During generation, we will pass the hidden state from one call to the next, allowing the model to maintain context across generation steps. During training with independent windows, we pass None and let PyTorch initialize to zeros automatically.

The teacher forcing pattern is evident in the training loop: x_batch contains the ground-truth tokens at every position. The model never receives its own output as input during training. The hidden state is re-initialized at the start of each batch; for this training setup we treat each window independently rather than carrying state across windows.

The gradient clipping call clip_grad_norm_(model.parameters(), max_norm=5.0) is a practical necessity for RNN training. Before the gradient update, the gradient vector is scaled down if its norm exceeds the max_norm threshold. This prevents a single large gradient from causing an unstable weight update. The value 5.0 is a common default; for LSTM and GRU models, which have more stable gradients than vanilla RNNs, this threshold is often raised to 1.0 or even omitted.

After training, we can generate text by running the RNN in autoregressive mode: each predicted token becomes the input for the next step.

The temperature parameter controls how peaked the output distribution is. At temperature 1.0, the raw softmax probabilities are used. At temperature below 1.0, the distribution is sharpened (more deterministic), favoring the most probable tokens. At temperature above 1.0, the distribution is flattened (more random), which can produce more creative but less coherent text.

Notice how the generation loop passes h from one step to the next. This is the key difference between generation and training: at inference time, the hidden state must carry context across steps because there is no ground truth to reset to. The model builds up a context vector in h as it generates each character, and that accumulated context influences every subsequent character. If the hidden state forgets early parts of the generation (the vanishing gradient problem), the generated text may lose coherence over long spans.

The key parameters for nn.RNN are:

• input_size: Dimensionality of each input token's embedding vector. Larger values can encode richer token representations.
• hidden_size: Number of hidden units $H$. This is the primary capacity control, but note the quadratic parameter cost from $\mathbf{W}_{hh}$.
• num_layers: Number of stacked RNN layers. With num_layers=2, the hidden state of the first layer is fed as input to the second layer at each time step.
• nonlinearity: Either 'tanh' (default) or 'relu'. The tanh option is standard for the reasons discussed above.
• batch_first: Whether input tensors are (batch, seq, feature) or (seq, batch, feature). Setting batch_first=True is more intuitive for most training loops.
• dropout: Dropout probability applied between RNN layers (not within a layer's recurrent connection). Only effective when num_layers > 1.
• bidirectional: Whether to run a second RNN over the sequence in reverse. The output dimension doubles when set to True.

The hidden state is the computational heart of the RNN. Let us visualize how it evolves as the network processes a sequence, to build intuition about what information it captures.

The heatmap reveals that the hidden state is not static. It shifts at every character, integrating the new input with the running context. Word boundaries (spaces) tend to produce noticeable shifts. This reflects the transition from word-internal character patterns to inter-word context. Some hidden dimensions appear to function as detectors for specific character categories, showing consistent activation patterns across all vowels, or shifting predictably at word boundaries. This continuous updating is the mechanism by which the RNN accumulates context.

The color pattern also reveals something important about the range of activations. Most values stay well within $(-1, 1)$. This reflects the effect of $\tanh$ saturation. Units near $+1$ or $-1$ are in the saturated regime where the gradient of $\tanh$ is close to zero. This saturation is part of the vanishing gradient story: units that are consistently saturated contribute very little gradient signal back through time.

Visualizing the training loss curve shows how quickly the model is learning to predict the next character.

A character-level RNN trained on a short corpus learns letter-transition probabilities, common bigrams and trigrams, and some word-level structure, all from the sequential character stream. The hidden state is doing the heavy lifting of remembering which characters have appeared and in what pattern. The steepest descent in loss typically occurs in the first few epochs as the model moves from near-uniform predictions to learning the most frequent character transitions. Later epochs fine-tune the model's handling of less frequent patterns.

It is worth pausing on what the cross-entropy loss value means in practical terms. A random predictor over a vocabulary of size $V$ has expected cross-entropy $\log_2 V$ bits per token. For our character vocabulary of about 30 characters, that is roughly $\log_2 30 \approx 4.9$ bits. If our trained model achieves a loss of around 2.0 nats ($\approx 2.9$ bits), it has learned meaningful structure that compresses the character stream significantly below the random baseline.

The concept of perplexity converts cross-entropy to a more interpretable scale. Perplexity is $\text{PPL} = e^{\mathcal{L}}$ (in nats) or $2^{\mathcal{L}}$ (in bits). A model with perplexity 10 is, on average, as uncertain between 10 equally likely options at each step. Lower perplexity means the model is more confident and more accurate. For reference, well-tuned word-level language models on large corpora achieve perplexities in the range of 20 to 100, while simple baselines start near the vocabulary size.

Building an RNN for a real task involves several engineering decisions beyond the core architecture.

Real text comes in variable lengths. A naive approach is to pad all sequences in a batch to the maximum length, but this wastes computation on padding tokens and can introduce noise into the gradients (if the loss is accidentally computed over padding positions).

PyTorch handles this through packed sequences. The torch.nn.utils.rnn.pack_padded_sequence function rearranges the batch into a compact representation that skips padding tokens during the RNN forward pass. After the RNN, pad_packed_sequence reverses this packing. The key inputs are the sequence tensor (padded to maximum length) and the actual lengths of each sequence.

When using packed sequences, sort the sequences by length in descending order before packing. PyTorch's pack_padded_sequence requires this (unless you set enforce_sorted=False). The length vector tells the packing routine when to transition between sequences.

For the loss computation, use a loss mask to ignore positions corresponding to padding tokens. The simplest way is to set ignore_index in nn.CrossEntropyLoss to the padding token's index, which instructs the loss function to skip those positions.

Gradient clipping is almost universally used when training RNNs. The exploding gradient problem, where the gradient norm grows exponentially across time steps, can cause catastrophic weight updates that destabilize training.

The clip_grad_norm_ function in PyTorch clips the global gradient norm: if the $L^2$ norm of all gradient vectors exceeds max_norm, each gradient is scaled down proportionally so that the total norm equals max_norm. This is different from clipping each gradient individually (which would change the gradient direction). Global norm clipping preserves the relative directions of the gradient components.

Choosing the right max_norm requires some tuning. A common heuristic is to log the gradient norm during training (before clipping) and set max_norm to roughly the 90th to 95th percentile of the unclipped norms. If gradients are clipped on most steps, the learning rate may be too high; if they are never clipped, the threshold may be unnecessary. The Part X chapter on Gradient Clipping covers this in more detail.

The DataLoader in our implementation shuffles the training windows randomly at each epoch. This works well for independent windows (where the sequence context starts fresh at each window boundary), but it is the wrong approach when you want to carry hidden state across windows (as in stateful BPTT).

For stateful processing, you should arrange the data into non-overlapping chunks and ensure that consecutive batches correspond to consecutive positions in the original sequence. This requires a custom Dataset and sampler that does not shuffle across sequence boundaries. The hidden state from the end of each batch is then detached from the computation graph (using .detach()) and passed as the initial hidden state for the next batch. The .detach() is important: it prevents gradients from flowing backward across the chunk boundary, implementing truncated BPTT.

The softmax computation in the output layer can suffer from numerical instability when logits are large. PyTorch's nn.CrossEntropyLoss combines log_softmax and the negative log-likelihood in a numerically stable way (using the log-sum-exp trick internally), so you should always use CrossEntropyLoss rather than manually computing softmax and then taking the log.

Similarly, avoid using float16 (half precision) for vanilla RNNs unless you have tested carefully. The relatively small values produced by $\tanh$ can underflow in float16 for large hidden sizes, causing training instability. LSTM and GRU with careful numerical design handle mixed precision better.

The Elman RNN introduced an important recurrent architecture in 1990. It demonstrated that neural networks could model temporal dependencies, process variable-length sequences, and share parameters across time. These properties made it suitable for language modeling, speech recognition, and time-series prediction in ways that feedforward networks simply could not match. Elman's work, and the complementary work by Jordan on a related architecture, established the conceptual foundation for all subsequent recurrent architectures.

Despite its elegance, the vanilla RNN has well-documented failure modes. The most significant is the vanishing gradient problem. Because the gradient flows backward through $T$ time steps of $\tanh$ activations and recurrent matrix multiplications, it tends to shrink exponentially with sequence length. Long-range dependencies, where the correct output at step $T$ depends on information from step 1, are extremely difficult to learn. The network effectively has a short memory window even though the recurrence theoretically allows unlimited context. In practice, vanilla RNNs struggle to learn dependencies beyond 10 to 20 time steps in most settings.

The exploding gradient problem appears on the opposite end: when the spectral radius of $\mathbf{W}_{hh}$ exceeds 1, gradients amplify rather than shrink. Gradient clipping, covered in Part X, provides a practical mitigation but does not solve the underlying instability. The exploding gradient problem tends to manifest suddenly during training as a catastrophic loss spike, which can be detected by monitoring gradient norms.

Another fundamental limitation is the bottleneck in information capacity. The hidden state must encode the entire history of the sequence in a fixed-size vector. For a sequence of 1000 tokens compressed into a 256-dimensional hidden state, much of the earlier context is inevitably lost or overwritten. This bottleneck is particularly acute in encoder-decoder architectures where the entire input sentence must be compressed into a single vector before the decoder begins generating. The attention mechanism, introduced in the context of neural machine translation (Bahdanau et al., 2015), was designed specifically to address this bottleneck by allowing the decoder to selectively retrieve relevant parts of the encoder's hidden state at each generation step.

Another limitation is the sequential nature of RNN computation. Each hidden state $\mathbf{h}_t$ depends on $\mathbf{h}_{t-1}$, which means the forward pass cannot be parallelized across time steps. For a sequence of length 1000, you must perform 1000 sequential matrix multiplications. This makes RNNs slow to train on modern hardware that excels at parallel computation. Modern GPUs are optimized for large matrix operations performed in parallel; a sequential chain of 1000 small matrix multiplications is a poor fit for this hardware profile. During the era of RNN dominance (roughly 2013-2017), this was an accepted cost of the architecture. When transformer models demonstrated that parallelizable self-attention could match or exceed RNN performance, the sequential bottleneck became a decisive disadvantage.

These limitations motivated the development of LSTMs and GRUs, which add gating mechanisms to control information flow and mitigate vanishing gradients. We cover LSTM architecture in Chapter 4 of this part and GRU architecture in Chapter 7. The transformer architecture, introduced in Part XIII, goes further by eliminating recurrence entirely in favor of self-attention, enabling full parallelization and dramatically better long-range dependency modeling.

The history of RNNs is not a simple linear progression. Elman's 1990 paper built on earlier work by Jordan (1986) and on Hopfield networks (1982), which were recurrent networks used for associative memory. The 1990s saw significant theoretical development, including the proof that RNNs are Turing-complete (Siegelmann and Sontag, 1992) and the introduction of the LSTM by Hochreiter and Schmidhuber in 1997.

But the first RNN renaissance in practical applications did not come until the early 2010s. Graves et al. (2013) demonstrated that deep bidirectional LSTMs could achieve state-of-the-art performance in speech recognition. Sutskever, Vinyals, and Le (2014) introduced the sequence-to-sequence framework with LSTMs, which enabled neural machine translation at scale. Mikolov et al. (2010) showed that recurrent language models significantly outperformed n-gram baselines on standard benchmarks.

The vanilla Elman RNN played a smaller role in this renaissance than LSTMs and GRUs, but it remains the conceptual foundation. Understanding the Elman RNN gives you the architecture that LSTM and GRU were explicitly designed to improve upon. Every gating mechanism in an LSTM can be understood as an answer to a specific failure mode of the Elman update equation.

Despite being superseded by transformers for most NLP tasks, RNNs remain relevant in settings where:

• Computational efficiency at inference time is critical. RNNs process each new token in $O(H^2)$ time, while transformers require $O(n^2)$ attention over the full context. For streaming applications where $n$ grows without bound, the transformer's quadratic cost becomes prohibitive.
• Streaming inputs require processing tokens one at a time as they arrive, without access to future context. RNNs are naturally causal and online.
• Simple, interpretable sequence models are preferred over large-scale transformer deployments.
• Edge deployment with strict memory constraints. An RNN's memory footprint is $O(H^2)$ (the weights) plus $O(H)$ (the hidden state). A transformer's KV-cache memory grows linearly with sequence length, making long-context inference memory-intensive on constrained hardware.
• Hybrid architectures. Several modern architectures combine attention with recurrent elements (such as Mamba's SSM layers, or Retentive Networks) precisely because they want the best of both worlds: efficient sequential processing with selective attention-like memory access.

This chapter covered the core ideas behind recurrent neural networks:

• Recurrent connections introduce memory into the network by passing the hidden state from one time step to the next, enabling context-sensitive processing of sequences.
• The Elman RNN computes $\mathbf{h}_t = \tanh(\mathbf{W}_{hh}\,\mathbf{h}_{t-1} + \mathbf{W}_{xh}\,\mathbf{x}_t + \mathbf{b}_h)$, with $\tanh$ as the squashing nonlinearity and separate weight matrices for the recurrent and input connections.
• Unrolling through time converts the cyclic RNN graph into a directed acyclic graph of depth $T$, making gradient computation possible via Backpropagation Through Time (BPTT). Truncated BPTT segments this unrolling to make training on long sequences tractable.
• Parameter sharing means the same weights apply at every time step. This gives RNNs positional generalization, efficiency with variable-length sequences, and the ability to handle sequences longer than those seen during training.
• Three core configurations: many-to-one for sequence classification (read all, output once), one-to-many for generation (receive a seed, output a sequence), and many-to-many for sequence labeling (output at every step) or encoder-decoder translation.
• Bidirectional RNNs run a forward and a backward RNN over the sequence and concatenate their hidden states, enabling each position to incorporate both left and right context.
• Stacked RNNs build a hierarchy of representations by feeding the hidden states of one RNN layer as input to the next, enabling more abstract sequential patterns to emerge in higher layers.
• Language modeling with an RNN predicts the probability of each next token conditioned on all prior tokens, trained with cross-entropy loss. Perplexity measures how confident the model is on held-out data.
• Teacher forcing provides ground-truth tokens as inputs during training, improving stability and convergence speed at the cost of a train/inference mismatch called exposure bias.
• Hidden state dimensionality is the primary capacity hyperparameter. The quadratic cost of $\mathbf{W}_{hh}$ makes large hidden sizes expensive.
• PyTorch's nn.RNN provides an efficient implementation with batch_first=True making it easy to integrate into standard training loops, with support for stacking (num_layers) and bidirectionality (bidirectional).
• Practical engineering involves handling variable-length sequences with packed tensors, gradient clipping for training stability, and careful design of the stateful hidden state transfer for truncated BPTT.
• Limitations include vanishing gradients (making long-range dependencies hard to learn), the fixed-size memory bottleneck, and the sequential nature of computation (preventing parallelization across time steps). These limitations motivated LSTMs, GRUs, and ultimately the transformer.

The next chapter explores Backpropagation Through Time in detail, analyzing exactly how gradients flow through the unrolled graph and why long sequences pose fundamental challenges for vanilla RNNs.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about RNN architecture and recurrent neural networks.