LSTM Architecture: Cell State, Gates, and Memory

In the previous chapters, we saw how vanilla RNNs struggle with long sequences. As we established in the Vanishing Gradients chapter, gradients either vanish into insignificance or explode beyond control, making it nearly impossible to learn dependencies that span more than a few dozen timesteps. This limitation motivated researchers to rethink the fundamental architecture of recurrent networks.

Long Short-Term Memory (LSTM) networks, introduced by Hochreiter and Schmidhuber in 1997, solve the vanishing gradient problem through a deceptively simple architectural insight: instead of forcing all information to pass through repeated nonlinear transformations at every timestep, create a dedicated pathway where information can flow relatively unchanged across many timesteps. This "memory highway" allows LSTMs to maintain and access information over hundreds or even thousands of steps, enabling applications that were previously impossible with vanilla RNNs. The key idea is so elegant that it can be stated in a single sentence: separate the cell's memory from its output, and let the memory update additively rather than through repeated multiplication.

The defining feature of an LSTM is its cell state, a separate memory channel that runs parallel to the hidden state. Think of it as a conveyor belt that carries information through time. Unlike the hidden state in vanilla RNNs, which gets completely rewritten at each timestep through a matrix multiplication and nonlinearity, the cell state can preserve information indefinitely by flowing forward with only element-wise operations applied to it.

To appreciate why this matters, consider what happens inside a vanilla RNN at each step. The hidden state is recomputed as $h_t = \tanh(W_h h_{t-1} + W_x x_t + b)$. Both $W_h$ and tanh are involved in that transformation. Tanh squashes any value into the range $[-1, 1]$, so the magnitude of the hidden state is always bounded. When you backpropagate through this squashing operation over many steps, the gradient of the loss with respect to an early hidden state gets multiplied by the derivative of tanh at each step. The derivative of tanh is bounded by 1, often much smaller for large inputs, so the gradient shrinks exponentially. After 50 or 100 steps, the gradient is effectively zero.

The LSTM cell state bypasses this problem in a fundamental way. Information stored in the cell state passes from one timestep to the next primarily through an additive operation: new information is added to old information, with each term selectively scaled by a gate value. Addition does not shrink gradients the way multiplication through nonlinearities does. The gradient of a sum is just the sum of the gradients, which means gradients can travel backward through the cell state pathway without multiplicative attenuation. This additive gradient path is the mathematical core of why LSTMs work.

It is worth pausing to appreciate how unusual this design was when it was proposed in 1997. The mainstream assumption in neural network research at the time was that computation should proceed through stacked layers of learned nonlinear transformations. Every layer or every timestep would apply a weight matrix and a squashing function, and the network would learn by adjusting those weights. Hochreiter and Schmidhuber challenged this by arguing that a memory system needed a way to copy information forward unchanged, not transform it at every step. Their cell state is essentially a write-once, read-selectively buffer: information can be injected into it, it can persist untouched for many steps, and it can later be read out when needed. This design philosophy would later resurface in the residual connections of deep CNNs and in the skip connections that are essential to modern Transformer architectures.

The two key vectors in an LSTM at timestep $t$ are:

• $C_t$: the cell state, which stores long-term memory and flows through time with minimal modification
• $h_t$: the hidden state, which is the output at each timestep and is derived from the cell state through the output gate

In a vanilla RNN, information at timestep $t$ must pass through every intermediate hidden state to reach timestep $t+k$. Each passage involves a matrix multiplication and a tanh activation, which compresses values and causes gradients to shrink exponentially. After enough steps, the original signal is lost in noise.

The cell state sidesteps this problem. Information stored in the cell state can travel from the beginning to the end of a sequence with only element-wise additions and scalings applied to it. Neither of those operations compounds the gradient shrinkage the way repeated matrix multiplications through tanh do. That additive path through time is the core mathematical reason LSTMs can learn long-range dependencies.

If the cell state flowed forward completely unchanged at every step, the network would have no way to update its memory. LSTMs need a principled way to selectively add new information, remove outdated information, and control what gets exposed as output. They accomplish this through gates, learned sigmoid layers that output values between 0 and 1.

Think of gates as dimmer switches rather than on/off toggles. A gate value of 0.0 means "block everything," 0.5 means "let half through," and 1.0 means "let everything through." Because gates use sigmoid activations, their outputs always fall in the $[0, 1]$ range, making them perfect for scaling information flow. The sigmoid function is differentiable everywhere, which is critical for backpropagation. A hard threshold, where values below 0.5 are set to 0 and values above are set to 1, would create a non-differentiable decision boundary that prevents gradient-based learning.

The gate values are not fixed: they are computed fresh at every timestep by passing the current input and previous hidden state through a small learned network. This means the same LSTM cell can behave differently at different points in a sequence. When processing the beginning of a long document, the forget gate might learn to mostly preserve stored information. When a new named entity appears, the input gate might spike to write it into memory. When a question mark arrives, the output gate might activate to expose accumulated context for an answer prediction. All of this behavior is learned implicitly from training data, not handcrafted by the designer.

LSTMs have three gates, each serving a distinct purpose:

• Forget gate: Decides what information to discard from the cell state. When processing a new sentence, the network may want to forget who the previous sentence was about.
• Input gate: Decides what new information to store in the cell state. When the model encounters a new subject, it wants to record that subject.
• Output gate: Decides what portion of the cell state to expose as the hidden state. Not everything stored in memory is relevant to the current prediction.

Every gate takes the same two inputs: the previous hidden state $h_{t-1}$ and the current input $x_t$. The general gate computation is:

where:

• $\sigma$: the sigmoid activation function, which outputs values in $[0, 1]$
• $W$: the weight matrix for this gate (each gate has its own learned weights)
• $[h_{t-1}, x_t]$: the concatenation of the previous hidden state and current input
• $b$: the bias vector for this gate

Because each gate has its own weight matrix, they specialize during training. The forget gate might learn to fire when it sees punctuation marks that signal a sentence boundary; the input gate might learn to fire when it encounters a named entity that needs to be tracked. This specialization is not programmed in: it emerges from gradient descent. What makes the architecture powerful is that the parameterization is rich enough to learn such specialization, and the gating structure provides the mechanism through which that specialization can influence the cell state.

It is worth asking why the LSTM specifically uses sigmoid for gates and tanh for memory computations. The answer is functional: each activation is chosen for what it computes in context.

Sigmoid maps any real-valued input to the range $(0, 1)$. When you multiply the cell state element-wise by a vector of sigmoid outputs, each dimension of the cell state gets scaled by a value between 0 and 1. A value of 0 completely erases that dimension; a value of 1 keeps it unchanged; anything in between produces a partial retention. This is exactly the "dimmer switch" behavior that gates require. No other common activation function produces values strictly in $[0, 1]$ in a natural way that is also differentiable and trainable.

Tanh, by contrast, maps inputs to $(-1, 1)$. This is the appropriate range for candidate values that will be added to the cell state, because a positive candidate increases a dimension's value while a negative candidate decreases it. If you used sigmoid for candidates, you could only ever add positive information to the cell state, never subtract it. The result would be a one-sided accumulation that could not represent the full range of patterns needed in practice. Using tanh allows the cell state to hold both positive and negative magnitudes, which is necessary for representing rich, varied information across different dimensions.

The combination of the two is what makes the full architecture work. The sigmoid gate decides how much to write; the tanh candidate provides what to write. Together they give the network full expressiveness over each cell state dimension.

Now that the gates make sense individually, let us see how they work together to create a complete memory system. The LSTM processes information in four distinct stages per timestep: decide what to forget, decide what to add, update the cell state, and decide what to output. These four stages happen simultaneously in the equations, but understanding them sequentially builds the right intuition.

Before adding new information, the LSTM first filters out what is no longer relevant. The forget gate examines $h_{t-1}$ and $x_t$, and produces a value between 0 and 1 for each element of the cell state. Elements scaled to near 0 get mostly erased; elements scaled to near 1 get preserved.

where:

• $f_t$: the forget gate output, a vector with one value per cell state dimension
• $W_f$: the forget gate's weight matrix
• $b_f$: the forget gate's bias vector

Consider parsing the sentence "The cat sat on the mat. The dog barked." When the model encounters the period and then "The", the forget gate should fire strongly to clear information about "cat" from the cell state, making room for the upcoming "dog" to be stored as the new subject. The forget gate does this by producing values close to 0 for dimensions that are currently holding subject information, effectively zeroing out that memory slot.

In practice, the forget gate tends to learn values close to 1 for most dimensions most of the time. This is the correct behavior for tasks where context persists across many steps. If the forget gate habitually set values to 0, the LSTM would lose its long-term memory too quickly. The forget gate essentially asks: "Given what I just saw and what I remember, which parts of my memory are no longer needed?" and most of the time the answer is: almost nothing needs to be forgotten yet.

The bias $b_f$ in the forget gate is often initialized to a positive value, such as 1.0 or 2.0. The intuition is that a freshly initialized LSTM should err on the side of remembering rather than forgetting. A large positive bias pushes the sigmoid output close to 1, meaning the network starts with a strong tendency to preserve cell state values. During training, it then learns which contexts warrant forgetting. This initialization trick, popularized by Jozefowicz et al. (2015), improves training stability and long-sequence performance.

With old information selectively cleared, the LSTM decides what to write into memory. This stage involves two parallel computations that work together. The input gate decides which positions in the cell state to update, while a tanh layer computes candidate values that could be written.

where:

• $i_t$: the input gate, controlling which cell state positions receive updates
• $\tilde{C}_t$: candidate values that could be added to the cell state
• $W_i, W_C$: separate weight matrices for the input gate and candidate computation
• $b_i, b_C$: corresponding bias vectors

The tanh function is used for the candidates rather than sigmoid because tanh outputs values in $[-1, 1]$, allowing the network to both increase and decrease cell state values. A positive candidate pushes that cell state element upward; a negative candidate pushes it downward. The input gate then scales these proposals, determining how much of each proposed change to apply.

The separation of the input gate from the candidate layer is subtle but important. Why not just compute new values and write them directly? The two-part design allows the network to decouple what kind of information to write from how much of it to write. The candidate layer might compute strong signals for certain dimensions, but the input gate might suppress them because the current context is not appropriate for that kind of update. Conversely, the input gate might activate for dimensions where the candidate values are small but consistent with a persistent pattern the network needs to build up incrementally. This decoupling gives the LSTM a richer vocabulary of memory operations than a single-component update would allow.

Consider what happens when the model encounters the word "not" in the phrase "not happy." The candidate layer might produce a negative value for dimensions associated with positivity, pushing the sentiment representation downward. The input gate activates strongly for those same dimensions to ensure the negation gets applied. Without the separate input gate, the network could not independently control where to update and how much to update.

The actual memory update combines the forgetting and remembering decisions:

where:

• $C_t$: the new cell state at timestep $t$
• $C_{t-1}$: the previous cell state from timestep $t-1$
• $f_t$: the forget gate output, scaling old information
• $i_t$: the input gate output, scaling new candidate information
• $\tilde{C}_t$: the candidate values proposed by the tanh layer
• $\odot$: element-wise (Hadamard) multiplication

This formula looks simple but does a lot. The first term, $f_t \odot C_{t-1}$, selectively preserves old information. If $f_t[j] = 0.9$ for some element $j$, we keep 90% of that element's previous value. The second term, $i_t \odot \tilde{C}_t$, selectively writes new information.

The critical observation is that this update is additive, not purely multiplicative. In a vanilla RNN, information at timestep $t$ reaches timestep $t+k$ only through $k$ sequential matrix multiplications. In an LSTM, information stored in the cell state at timestep $t$ can reach timestep $t+k$ through a series of additions and element-wise scalings. The gradient that flows backward through addition does not get multiplied by a small number at each step the way it would through a tanh-saturated weight matrix. This additive pathway is the core mechanism behind LSTMs solving the vanishing gradient problem.

Consider the gradient of the loss $L$ with respect to the cell state at time $t$:

This gradient flows backward by multiplication with $f_{t+1}$, the forget gate value at the next timestep. If $f_{t+1}$ is close to 1, the gradient passes through almost unchanged. Compare this to a vanilla RNN, where the corresponding gradient involves the derivative of tanh applied to $W_h$, a term that can be very small whenever weights or activations are in the saturating regions of tanh. The LSTM cell state gradient path is far gentler, and a well-trained forget gate that learns to keep values near 1 for important memories creates a near-constant gradient pathway. Hochreiter and Schmidhuber called this the "Constant Error Carousel" (CEC): the cell state can carry an error signal (a gradient) forward in time without it vanishing, as long as the forget gate cooperates.

Let us make this concrete with a numerical example using a 4-dimensional cell state:

After updating the cell state, the LSTM decides what portion of that memory to expose as the current output. The cell state may contain information from many previous steps, but only some of it is relevant to the current prediction. The output gate makes that selection.

where:

• $o_t$: the output gate, deciding which cell state dimensions to expose
• $h_t$: the hidden state, which is the output at this timestep
• $\tanh(C_t)$: the cell state squashed to $[-1, 1]$

The tanh applied to the cell state serves two purposes. First, it bounds the values to $[-1, 1]$, preventing any single dimension from dominating the output. Without this squashing, cell state values could accumulate over many additive updates and grow large in magnitude, causing numerical instability and making the output hard for downstream layers to handle. Second, the tanh re-centers values around zero, which helps downstream computations that typically assume zero-mean inputs.

The output gate then selects which of these bounded values to include in the hidden state. This is where the LSTM distinguishes what it knows and what it reports. The cell state is the network's internal notebook, which may contain many things. The hidden state is what the network decides to share with the world at this particular moment. Suppose the LSTM is generating text and has been tracking the grammatical number (singular or plural) of the current subject. Most of the time that number-tracking information sits quietly in the cell state without being actively used. But when the model reaches a point where a verb needs to agree with the subject, the output gate activates for the dimensions storing number information, pulling it out into the hidden state where it can influence the verb prediction.

This hidden state $h_t$ serves two roles simultaneously: it is the output that downstream layers or classifiers use for predictions, and it is also fed back into the LSTM at the next timestep as $h_{t-1}$. The cell state, by contrast, is internal. It acts as a memory buffer that influences the hidden state through the output gate but is not directly exposed to the rest of the network. This encapsulation is a design virtue: it means the long-term memory (cell state) and the immediate context signal (hidden state) are separate, and the network can develop them differently.

Every timestep runs these six computations in order:

1. Forget gate: $f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$
2. Input gate: $i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$
3. Candidate: $\tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C)$
4. Cell update: $C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$
5. Output gate: $o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$
6. Hidden state: $h_t = o_t \odot \tanh(C_t)$

Each gate has its own weight matrix and bias, giving the network four parameter groups to learn: $(W_f, b_f)$, $(W_i, b_i)$, $(W_C, b_C)$, and $(W_o, b_o)$. During training, backpropagation adjusts these parameters so the gates learn to open and close at the right times for the task. The precise parameter count and a from-scratch implementation are covered in the LSTM Gate Equations chapter.

Notice that steps 1, 2, 3, and 5 all involve the same inputs ($h_{t-1}$ and $x_t$) processed through different weight matrices. In practice, implementations often batch all four of these computations into a single large matrix multiplication, applying one weight matrix of shape $(4 \times \text{hidden\_size}, \text{input\_size} + \text{hidden\_size})$ at once, then splitting the result into four slices. This is both more efficient on modern hardware and mathematically equivalent to computing each gate separately. PyTorch's nn.LSTM uses this batched formulation internally.

Understanding how information flows through an LSTM helps build intuition about what these networks can learn. Consider processing the sentence: "The cat, which had been sleeping on the warm windowsill since early morning, finally stretched." The subject "cat" appears at the beginning, but the verb "stretched" does not arrive until the end. A vanilla RNN loses the subject information before reaching the verb. An LSTM stores the subject in the cell state and retrieves it when it becomes relevant.

The mechanics of this process illustrate each gate's role in a natural context. When "cat" is processed, the input gate fires strongly for dimensions in the cell state designated for tracking the grammatical subject. The candidate values for those dimensions are computed from the word embedding of "cat", encoding its syntactic and semantic properties. Meanwhile, the forget gate remains high: the network does not want to erase any previously accumulated context from the beginning of the passage.

As the long subordinate clause unfolds ("which had been sleeping on the warm windowsill since early morning"), the forget gate continues to protect the subject information. The input gate makes small writes for the content of the clause, but the cell state dimension holding the subject stays relatively stable. When "finally" appears as a signal that the main clause is about to continue, the output gate begins preparing to expose the subject information that was stored.

When "stretched" is reached, the output gate activates for the subject-tracking dimensions. The model can now see both the stored subject (the cat) and the current token (stretched), and it can use this to make predictions about what comes next. This is long-range dependency resolution in action, spanning roughly a dozen tokens.

When "cat" appears, the input gate activates strongly, storing subject information in the cell state. The forget gate remains high through the long subordinate clause, preserving what was stored. When "stretched" arrives, the output gate fires, allowing the stored subject information to inform the verb-agreement prediction. This selective storage, preservation, and retrieval is the LSTM in action.

While the cell state provides long-term memory, the hidden state functions more like a working memory: the information currently active and available for computation. A useful mental model distinguishes the two as follows. The cell state is like a filing cabinet: it can hold a large amount of information over long periods, but accessing any particular piece requires an explicit retrieval action (the output gate). The hidden state is like what is currently on your desk: a small, immediately accessible summary of what is relevant right now.

This distinction explains why many sequence tasks use the final hidden state rather than the final cell state for downstream processing. The cell state at the end of a sequence may contain accumulated memories from early in the sequence that are no longer contextually active. The final hidden state, having been filtered through the output gate, contains a more focused representation of what was relevant at that last timestep. For classification tasks, this focused representation is often more useful as a fixed-size input to a classifier than the raw cell state would be.

It also explains a common practical observation: when fine-tuning or inspecting trained LSTMs, analysis of the hidden states is much more informative than analysis of the cell states. The hidden state is designed to be informative for the current prediction; the cell state is designed to be a stable reservoir. Probing tools that visualize LSTM representations typically examine $h_t$ rather than $C_t$.

The advantage of LSTMs becomes measurable when we compare gradient propagation to vanilla RNNs. Recall from the Vanishing Gradients chapter that vanilla RNN gradients decay exponentially: each timestep multiplies the gradient by a factor typically less than 1. For typical random initialization, this factor is around 0.7 to 0.8, making gradients negligible after 50 or so steps.

LSTMs change this substantially. The additive cell state update allows gradients to flow backward through the addition operation without multiplication by a small value. The forget gate introduces some scaling, but a well-trained forget gate learns values close to 1 for important information, creating a near-constant gradient highway.

The mathematical story is worth tracing explicitly. In a vanilla RNN, the gradient of the loss with respect to the hidden state at step $t$, given the loss depends on step $t+k$, involves a product of $k$ Jacobian matrices, each associated with the tanh nonlinearity and the weight matrix:

Each factor in this product contains the derivative of tanh, which saturates near 0 for large inputs. In the LSTM, the analogous quantity for the cell state path involves only the forget gates:

A product of forget gate values is much gentler than a product of tanh Jacobians. If the network learns forget gates close to 1, this product stays near 1 for many steps, preserving the gradient signal over arbitrarily long distances.

The difference is substantial. Vanilla RNN gradients become negligible after around 50 timesteps. LSTM gradients remain orders of magnitude larger and stay above the practical training threshold for sequences well over 100 steps. This gradient stability is why LSTMs enabled practical sequence modeling at scale: tasks like machine translation, speech recognition, and language modeling all involve dependencies that span many dozens of tokens.

For complex tasks, it is common to stack multiple LSTM layers on top of each other. The output hidden states of the first LSTM layer become the input to the second layer, and so on. Each layer operates on the sequence of hidden states produced by the layer below, allowing deeper layers to model longer-range or more abstract patterns.

A two-layer LSTM processes a sequence as follows. The first layer sees the raw input features $x_1, x_2, \ldots, x_T$ and produces hidden states $h_1^{(1)}, h_2^{(1)}, \ldots, h_T^{(1)}$. These serve as the input to the second layer, which produces $h_1^{(2)}, h_2^{(2)}, \ldots, h_T^{(2)}$. The second layer's outputs can be used for prediction.

Stacking layers tends to improve performance up to a point. Two or three layers are common in many NLP applications. Beyond three layers, training stability can become an issue: the gradients that must flow backward through both time and depth face a compounding attenuation. Dropout between layers helps regularize stacked LSTMs and is almost always used in practice when stacking more than one layer. In PyTorch, the num_layers argument handles stacking automatically, and the dropout argument applies dropout between all layers except the last.

One subtlety with stacked LSTMs: each layer has its own cell state and hidden state. When you initialize the LSTM with hidden = (h_0, c_0), both tensors have shape (num_layers, batch, hidden_size) to accommodate one state per layer. This is easy to overlook when moving from single-layer to multi-layer configurations.

A standard LSTM processes sequences left to right. At each timestep, it has seen all previous tokens but none of the future ones. For tasks like named entity recognition, sentiment analysis, or document classification, the model has access to the entire input at inference time. In those cases, each token's representation can benefit from both past and future context.

A bidirectional LSTM (Bi-LSTM) addresses this by running two LSTMs over the same sequence: one in the forward direction (left to right) and one in the backward direction (right to left). At each timestep, the forward LSTM has seen all tokens up to position $t$, and the backward LSTM has seen all tokens from position $t$ onward. The hidden states from both directions are concatenated to produce a representation that encodes bidirectional context.

For position $t$ in a bidirectional LSTM with hidden size $d$, the combined representation has dimension $2d$:

where $h_t^{\rightarrow}$ is the forward hidden state and $h_t^{\leftarrow}$ is the backward hidden state.

The gains from bidirectionality can be substantial for understanding tasks. Consider labeling the word "bank" in a sentence. In "He walked to the river bank to fish," the word "bank" is a geographical feature. In "He visited the bank to deposit a check," it is a financial institution. A unidirectional LSTM would need to predict the label for "bank" before seeing the disambiguating words later in the sentence. A bidirectional LSTM incorporates both the context before and after "bank" simultaneously, making the correct label much easier to determine.

The key limitation of bidirectional LSTMs is that they cannot be used for autoregressive generation. When generating text one token at a time, there is no future context to pass to the backward layer. Bi-LSTMs are therefore primarily used in encoder roles: understanding and representing existing text, not producing new text. In PyTorch, setting bidirectional=True in nn.LSTM enables bidirectional processing, and the output has shape (batch, seq_len, 2 * hidden_size).

PyTorch provides nn.LSTM as a built-in, highly optimized implementation. Understanding its interface is essential before building anything with LSTMs.

The parameter count formula reveals how LSTM scaling works. With input size $d_x$ and hidden size $d_h$, the total parameters are $4 \times (d_x + d_h) \times d_h + 4 \times d_h$. For a common configuration like $d_x = 256$ and $d_h = 512$, that is roughly 1.6 million parameters per LSTM layer. This is a modest count compared to transformer layers with similar hidden sizes, though transformers achieve their expressiveness through attention rather than gated recurrence.

The output tensor contains $h_t$ at every timestep. This is what you use for per-token prediction tasks: named entity recognition, part-of-speech tagging, sequence-to-sequence encoding, or any task where you need a representation for each position in the input. The hidden and cell tuple contains only the final states, which is sufficient for sequence-level tasks like classification. If you only need the final summary representation, using hidden[-1] (the last layer's final hidden state) is both efficient and semantically appropriate.

By default, nn.LSTM initializes the hidden and cell states to zero when no initial state is provided. For many tasks, this default is fine. But there are important cases where explicit initialization matters.

When processing sequences in minibatches, sequences within a batch are unrelated to each other. You want the hidden state to be zero at the start of every new sequence, and PyTorch handles this automatically when you call lstm(x) without passing a hidden state.

When processing long sequences in chunks for memory efficiency, you need to carry the hidden state from one chunk to the next. Here you would save the (hidden, cell) output from one call and pass it as input to the next, carrying state across chunk boundaries.

When using the LSTM as a language model decoder initialized from an encoder, you would set the initial hidden state to the encoder's final hidden state, allowing the decoder to condition its generation on the encoded input. This encoder-decoder pattern was the dominant architecture for machine translation before transformers replaced it.

One of the most common LSTM applications is classifying entire sequences: deciding the sentiment of a review, the intent of a query, or the topic of a document. The approach is to run the LSTM over the full sequence and use the final hidden state as a fixed-size summary vector for a linear classifier. The LSTM does the hard work of compressing a variable-length sequence into a fixed-size representation; the classifier on top is just a linear layer applied to that representation.

The simplicity of this architecture is one of LSTMs' strengths for classification. You stack an LSTM on top of an embedding layer, apply the LSTM to get a sequence of hidden states, take the last hidden state, and pass it through a linear layer. The LSTM does all the work of summarizing variable-length sequences into fixed-size vectors.

A variation that often works better in practice uses the mean of all hidden states rather than just the last one. The final hidden state represents the network's processing state after the last token, which gives disproportionate weight to the end of the sequence. Averaging over all positions gives equal weight to information throughout the sequence. Max pooling across the time dimension is another common alternative: take the element-wise maximum across time, which captures the strongest activation for each hidden state dimension regardless of when it occurred. In practice, mean pooling and max pooling both tend to outperform using only the final hidden state on classification tasks, especially for long sequences.

Language modeling is among the most natural applications for LSTMs. The model reads characters one at a time and learns to predict the next character. Because individual characters carry minimal meaning on their own, the model must internalize word-level, phrase-level, and sentence-level patterns to make accurate predictions.

Character-level modeling was historically significant because it required no tokenization decisions and could handle any input text without preprocessing. The famous 2015 blog post by Andrej Karpathy, "The Unreasonable Effectiveness of Recurrent Neural Networks," demonstrated that LSTM character models trained on large corpora could generate surprisingly coherent text, structured code, and even plausible mathematical notation. This work helped establish LSTMs as the dominant sequence modeling architecture before the transformer era and inspired many researchers to explore what kinds of structure an LSTM could learn to represent internally.

The character model uses a small embedding layer before the LSTM. Although character indices could be passed directly as one-hot vectors, an embedding layer allows the model to learn a dense representation for each character. In a well-trained model, characters with similar roles (all lowercase vowels, all punctuation marks) tend to cluster together in the embedding space, making the downstream LSTM's job easier.

Note that the LSTM returns the hidden state, which is passed back into the model during generation. This stateful generation pattern, where the hidden state is carried from one call to the next, is what enables the model to generate coherent text: each new character prediction takes into account the most recent character and the full history encoded in the recurrent state.

The input-target offset by one position is the standard autoregressive training setup. The model sees a character and must predict the next one. By training on all consecutive pairs in the corpus, the model learns the statistical regularities of the text: which characters commonly follow which others, which word patterns are frequent, and which punctuation conventions appear.

Even on a small corpus with brief training, the LSTM generates plausible character sequences. The temperature parameter plays an large effect in generation quality. Low temperatures (0.3 to 0.5) make the model more conservative: it selects characters with high probability more consistently, producing repetitive but well-formed text. High temperatures (1.2 to 1.5) make the model more exploratory: it sometimes picks lower-probability characters, producing more varied but potentially less coherent output. Temperature 1.0 corresponds to unmodified sampling from the model's distribution. In practice, values around 0.7 to 0.9 often produce a good balance of coherence and diversity.

Given more data and training, the model would learn word boundaries, punctuation conventions, and longer structural patterns. This is the foundation underlying language models that operate at the subword level, though modern systems have replaced LSTMs with transformers as the underlying architecture.

When configuring nn.LSTM:

• input_size: Dimensionality of the input at each timestep. For embedded text, this is the embedding dimension.
• hidden_size: Dimensionality of both the hidden state and cell state. Larger values (128 to 1024) provide more capacity but increase computation.
• num_layers: Number of stacked LSTM layers. More layers can model more complex patterns but require more gradient clipping and careful tuning.
• batch_first: When True, expects input shape (batch, seq_len, features). Using True aligns with standard data loaders.
• dropout: Applied between LSTM layers when num_layers > 1. Values in the range 0.1 to 0.5 reduce overfitting.
• bidirectional: When True, processes the sequence in both directions simultaneously, doubling the effective hidden size. Useful for classification but inapplicable for generation.

The original LSTM architecture spawned a family of related designs that modify the gating structure in different ways. Understanding these variants illuminates which parts of the LSTM architecture are necessary and which admit alternatives.

Standard LSTM gates receive only $h_{t-1}$ and $x_t$ as inputs. A variant proposed by Gers and Schmidhuber (2000) adds "peephole connections" that also feed the cell state directly into the gates:

The intuition is that the gates should be able to look at the memory contents in addition to the previously exposed hidden state. Consider a clock-like memory: the gate deciding when to reset a counter would benefit from seeing the current count directly, rather than inferring it through the hidden state. Peephole connections add a small number of parameters per gate and can improve performance on tasks requiring precise timing, such as speech recognition with long silence gaps. In practice, the improvement is often modest, and peephole connections are not included in PyTorch's standard nn.LSTM.

The Gated Recurrent Unit (GRU), introduced by Cho et al. (2014), simplifies the LSTM by merging the cell state and hidden state into a single vector and replacing the three-gate system with two gates: a reset gate and an update gate. The GRU has fewer parameters than an LSTM of the same hidden size and tends to train faster. On many tasks, GRUs and LSTMs perform comparably. The choice between them is often empirical. The next chapter covers GRUs in depth.

The LSTM's forget gate and input gate are independent, which means it is possible for both to have high values simultaneously. A coupled variant forces them to be complementary:

This ensures that the total information in each cell state dimension remains roughly constant: if the forget gate retains 70% of the old value, the input gate contributes exactly 30% of the new candidate. This hard coupling reduces the parameter count and enforces a conservation-like property on the cell state. It works well on some tasks but can be too restrictive for others.

Training LSTMs effectively requires attention to several details beyond the architecture itself. These practical considerations can mean the difference between a model that trains smoothly and one that fails to converge.

Despite the LSTM's protection against vanishing gradients, exploding gradients remain a risk, particularly in deep or wide LSTMs. A single unusually large gradient update can destabilize the weights irreversibly. Gradient clipping addresses this by rescaling the gradient vector if its norm exceeds a threshold. This is standard practice for any RNN training:

The threshold max_norm=1.0 is a common default, though values up to 5.0 are used in some applications. If training loss is erratic or shows sudden spikes, increasing the clipping threshold (or checking that clipping is enabled) is usually the first debugging step.

The LSTM's weight matrices are typically initialized with small random values, often from a uniform distribution over $[-\sqrt{1/\text{hidden\_size}}, \sqrt{1/\text{hidden\_size}}]$. PyTorch's nn.LSTM uses this initialization by default. More specialized initializations, such as orthogonal initialization for the recurrent weight matrix $W_h$, can improve training on tasks with very long sequences by preserving the norm of the gradient through time.

The forget gate bias initialization is one of the most reliable practical tricks. Setting $b_f = 1.0$ or $b_f = 2.0$ at initialization costs nothing and often measurably improves convergence on tasks with long-range dependencies. The intuition is that a freshly initialized LSTM should err on the side of remembering rather than forgetting, and a large positive forget gate bias achieves exactly that. PyTorch does not do this by default, but it is easy to apply after construction by directly modifying the bias parameter.

Training on very long sequences requires storing all intermediate hidden and cell states for the backward pass. For sequences of thousands of tokens, this can exhaust GPU memory. Truncated BPTT addresses this by processing the sequence in fixed-length chunks, backpropagating only within each chunk while carrying the hidden state forward across chunk boundaries.

The trade-off is explicit: truncated BPTT prevents the model from learning dependencies that span chunk boundaries. A chunk size of 35 tokens means the model cannot directly learn from dependencies longer than 35 tokens via gradients. In practice, many of the dependencies that matter in language are shorter than common chunk sizes, and the cell state's ability to carry information forward across chunk boundaries provides some implicit long-range memory even when gradients do not flow backward across them. Truncated BPTT is the standard training regime for language model training on modern hardware.

The LSTM architecture has clear limitations that are worth understanding before choosing it for a task.

The most significant practical limitation is sequential computation. Computing $h_t$ requires $h_{t-1}$, which requires $h_{t-2}$, and so on. This causal dependency means that no matter how many processors are available, an LSTM must take $T$ sequential steps for a sequence of length $T$. Modern GPUs designed for massively parallel computation cannot be fully utilized on long sequences under this constraint. Transformers, by contrast, process all positions simultaneously in a single attention pass, making them dramatically faster to train when sequences are long and batches are large. This sequential bottleneck is the primary reason LSTMs have been largely displaced by transformers in large-scale language modeling, despite their excellent gradient properties.

Memory requirements during training also scale linearly with sequence length. Backpropagation through time must retain every intermediate hidden and cell state so gradients can be computed on the backward pass. For sequences of thousands of tokens, this memory cost becomes prohibitive on GPU. Truncated BPTT mitigates the cost by limiting how many steps backward gradients are propagated, at the price of sacrificing the ability to learn the very longest dependencies.

LSTMs also do not provide a natural mechanism for direct attention between arbitrary timesteps. If the model needs to compare the token at position 10 with the token at position 100, it must carry the position 10 information forward through 90 hidden states before making the comparison. Attention mechanisms, which allow any two positions to interact directly in a single computation step, provide a fundamentally richer interface for long-range relationship modeling. This is one of the key architectural advantages of the transformer that drove its widespread adoption.

Despite these limitations, LSTMs fundamentally changed sequence modeling. Before LSTMs, no neural architecture could reliably learn dependencies spanning more than a few dozen tokens. After LSTMs, machine translation, speech recognition, handwriting recognition, and language modeling at scale all became tractable. The Hochreiter and Schmidhuber (1997) paper remains among the most cited in all of machine learning, with over 70,000 citations as of 2024.

LSTMs also introduced a new design primitive: the gated memory cell. Before LSTMs, most network layers transformed information through learned nonlinear functions. LSTMs showed that networks could also learn to dynamically route and preserve information, with the routing decisions driven by the data itself. This gating idea directly inspired the GRU (Gated Recurrent Unit), covered in the next chapter, and more distantly influenced the attention mechanisms that underpin Transformer models. The residual connections essential to training deep transformers also echo the LSTM cell state's strategy of enabling identity paths for information to flow without being transformed at every step.

Understanding the historical context of LSTMs helps appreciate the significance of the contribution. In the mid-1990s, several groups were working on the vanishing gradient problem, but most approaches involved either gradient clipping, careful initialization, or modifications to the training algorithm rather than changes to the architecture itself. Hochreiter's 1991 diploma thesis, written before the LSTM paper, contained a detailed analysis of why gradients vanish in standard RNNs, laying the theoretical foundation for the solution.

The 1997 LSTM paper was initially received with limited enthusiasm, partly because the architecture seemed complicated relative to simpler RNNs and partly because the computing hardware of the era was not capable of training LSTMs on the scale needed to demonstrate their advantages clearly. It was not until the 2010s, when GPU-based deep learning made large-scale sequence modeling practical, that LSTMs became the dominant approach. The breakthrough applications, including Google's translation system, Apple's Siri, Amazon's Alexa, and numerous speech recognition systems, all relied heavily on LSTMs trained at scales that would have been unimaginable in 1997.

The story of LSTMs is a story about an idea that was correct and important, but required the right computational context to be fully appreciated. It also illustrates a pattern common in deep learning: architectural innovations often precede by years or decades the practical context in which they prove decisive. Understanding the architecture deeply positions you to recognize when a similar conceptual advance becomes practically relevant in the future.

This chapter introduced the LSTM architecture and the core innovations that allow it to learn long-range dependencies.

The central idea is the cell state: an information highway that accumulates information through additive updates rather than repeated matrix multiplications. This additive structure keeps gradients from vanishing over long sequences, allowing the network to propagate learning signals across hundreds of timesteps. The mathematical reason is direct: gradients flowing back through the additive cell state update are multiplied by the forget gate value at each step, rather than by the derivative of a saturating nonlinearity, keeping the gradient signal alive over long distances.

Three gates coordinate access to the cell state:

• The forget gate selectively discards information from the previous cell state, controlled by a learned sigmoid layer that examines the current input and previous hidden state. Initializing its bias to a positive value at the start of training is a simple trick that improves long-range learning.
• The input gate and candidate layer together control what new information to write into memory: the candidate layer proposes values using tanh (enabling both positive and negative updates), while the input gate scales how much of each proposal is applied. This two-part design decouples what to write from how much to write.
• The output gate filters the cell state to produce the current hidden state output, creating a fundamental distinction between what the network knows (cell state) and what it currently reports (hidden state).

These gates are learned, not fixed. They specialize during training based on the task, learning when to remember, when to forget, and what to output. The PyTorch nn.LSTM module packages all six equations into a single optimized call, with output providing per-timestep hidden states and hidden, cell providing the final states.

LSTMs introduced the gating abstraction into neural network design, a contribution whose influence extends far beyond recurrent networks. The GRU simplifies the LSTM by merging the cell and hidden states and using two gates instead of three. Transformer attention generalizes the idea of selective access to memory through a fundamentally different mechanism that operates without sequential computation. Both of these advances owe a conceptual debt to the LSTM's demonstration that learned routing of information can be more powerful than learned transformation alone.

In the LSTM Gate Equations chapter, we will work through the mathematics precisely: derive the parameter count, implement all six equations from scratch in NumPy, and verify that the result matches PyTorch. That mathematical grounding will prepare you for LSTM Gradient Flow, where we analyze exactly why the cell state path prevents vanishing gradients.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about LSTM architecture.