LSTM Architecture › Information Flow in LSTMs

Michael BrenndoerferMay 11, 20255 min read
Language AI HandbookAll chapters
Part 11 of 67Recurrent Neural Networks
Chapter 77LSTM Architecture › Information Flow in LSTMs
Reading section 5 of 14 · Information Flow in LSTMs · 5 minSection 5 of 14 · 5 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 77 of 513

Section 5 of 14 · 5 min read

LSTM Architecture › Information Flow in LSTMsOne section · Sections in this chapter5 / 14 ↓
Sections in this chapter14
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding how information flows through an LSTM helps build intuition about what these networks can learn. Consider processing the sentence: "The cat, which had been sleeping on the warm windowsill since early morning, finally stretched." The subject "cat" appears at the beginning, but the verb "stretched" does not arrive until the end. A vanilla RNN loses the subject information before reaching the verb. An LSTM stores the subject in the cell state and retrieves it when it becomes relevant.

The mechanics of this process illustrate each gate's role in a natural context. When "cat" is processed, the input gate fires strongly for dimensions in the cell state designated for tracking the grammatical subject. The candidate values for those dimensions are computed from the word embedding of "cat", encoding its syntactic and semantic properties. Meanwhile, the forget gate remains high: the network does not want to erase any previously accumulated context from the beginning of the passage.

As the long subordinate clause unfolds ("which had been sleeping on the warm windowsill since early morning"), the forget gate continues to protect the subject information. The input gate makes small writes for the content of the clause, but the cell state dimension holding the subject stays relatively stable. When "finally" appears as a signal that the main clause is about to continue, the output gate begins preparing to expose the subject information that was stored.

When "stretched" is reached, the output gate activates for the subject-tracking dimensions. The model can now see both the stored subject (the cat) and the current token (stretched), and it can use this to make predictions about what comes next. This is long-range dependency resolution in action, spanning roughly a dozen tokens.

In[9]:
Code
# Simulated gate activation trace for the example sentence above.
# These are illustrative values, not from a real trained model,
# but representative of what a trained LSTM produces on such input.
words = [
    "The",
    "cat",
    ",",
    "which",
    "had",
    "been",
    "sleeping",
    "...",
    "finally",
    "stretched",
    ".",
]
positions = np.arange(len(words))

forget_gates = [0.9, 0.3, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.4, 0.2]
input_gates = [0.2, 0.9, 0.1, 0.3, 0.2, 0.2, 0.3, 0.1, 0.3, 0.1, 0.1]
output_gates = [0.3, 0.5, 0.2, 0.4, 0.3, 0.3, 0.4, 0.2, 0.5, 0.9, 0.3]
cell_state_trace = [
    0.1,
    0.85,
    0.82,
    0.80,
    0.78,
    0.76,
    0.74,
    0.72,
    0.70,
    0.35,
    0.1,
]
Out[10]:
Visualization
Line plot of cell state value over time, spiking at cat and remaining high through intervening words before dropping at the period.
Cell state trace across the example sentence, showing one dimension corresponding to subject-tracking memory. The cell state spikes when 'cat' is stored and remains elevated across the entire intervening clause, then drops when the sentence ends. This plateau demonstrates the LSTM's ability to preserve information unchanged across many timesteps.
Out[11]:
Visualization
Grouped bar chart showing forget gate, input gate, and output gate activations for each word in the sentence.
Gate activation patterns across the example sentence. The input gate spikes at 'cat' to write the subject into memory. The forget gate remains high through the subordinate clause, protecting stored information. The output gate peaks at 'stretched', retrieving the stored subject to inform the verb prediction.

When "cat" appears, the input gate activates strongly, storing subject information in the cell state. The forget gate remains high through the long subordinate clause, preserving what was stored. When "stretched" arrives, the output gate fires, allowing the stored subject information to inform the verb-agreement prediction. This selective storage, preservation, and retrieval is the LSTM in action.

The Hidden State as a Working Memory

While the cell state provides long-term memory, the hidden state functions more like a working memory: the information currently active and available for computation. A useful mental model distinguishes the two as follows. The cell state is like a filing cabinet: it can hold a large amount of information over long periods, but accessing any particular piece requires an explicit retrieval action (the output gate). The hidden state is like what is currently on your desk: a small, immediately accessible summary of what is relevant right now.

This distinction explains why many sequence tasks use the final hidden state rather than the final cell state for downstream processing. The cell state at the end of a sequence may contain accumulated memories from early in the sequence that are no longer contextually active. The final hidden state, having been filtered through the output gate, contains a more focused representation of what was relevant at that last timestep. For classification tasks, this focused representation is often more useful as a fixed-size input to a classifier than the raw cell state would be.

It also explains a common practical observation: when fine-tuning or inspecting trained LSTMs, analysis of the hidden states is much more informative than analysis of the cell states. The hidden state is designed to be informative for the current prediction; the cell state is designed to be a stable reservoir. Probing tools that visualize LSTM representations typically examine hth_t rather than CtC_t.