LSTM Architecture › Summary

This chapter introduced the LSTM architecture and the core innovations that allow it to learn long-range dependencies.

The central idea is the cell state: an information highway that accumulates information through additive updates rather than repeated matrix multiplications. This additive structure keeps gradients from vanishing over long sequences, allowing the network to propagate learning signals across hundreds of timesteps. The mathematical reason is direct: gradients flowing back through the additive cell state update are multiplied by the forget gate value at each step, rather than by the derivative of a saturating nonlinearity, keeping the gradient signal alive over long distances.

Three gates coordinate access to the cell state:

• The forget gate selectively discards information from the previous cell state, controlled by a learned sigmoid layer that examines the current input and previous hidden state. Initializing its bias to a positive value at the start of training is a simple trick that improves long-range learning.
• The input gate and candidate layer together control what new information to write into memory: the candidate layer proposes values using tanh (enabling both positive and negative updates), while the input gate scales how much of each proposal is applied. This two-part design decouples what to write from how much to write.
• The output gate filters the cell state to produce the current hidden state output, creating a fundamental distinction between what the network knows (cell state) and what it currently reports (hidden state).

These gates are learned, not fixed. They specialize during training based on the task, learning when to remember, when to forget, and what to output. The PyTorch nn.LSTM module packages all six equations into a single optimized call, with output providing per-timestep hidden states and hidden, cell providing the final states.

LSTMs introduced the gating abstraction into neural network design, a contribution whose influence extends far beyond recurrent networks. The GRU simplifies the LSTM by merging the cell and hidden states and using two gates instead of three. Transformer attention generalizes the idea of selective access to memory through a fundamentally different mechanism that operates without sequential computation. Both of these advances owe a conceptual debt to the LSTM's demonstration that learned routing of information can be more powerful than learned transformation alone.

In the LSTM Gate Equations chapter, we will work through the mathematics precisely: derive the parameter count, implement all six equations from scratch in NumPy, and verify that the result matches PyTorch. That mathematical grounding will prepare you for LSTM Gradient Flow, where we analyze exactly why the cell state path prevents vanishing gradients.