Long Short-Term Memory (1997) › The LSTM Solution: Gated Memory

Hochreiter and Schmidhuber's breakthrough came from recognizing that memory in neural networks required explicit architectural support, not just clever training techniques. They introduced a fundamentally new kind of neural unit, one equipped with specialized mechanisms for maintaining and manipulating information over extended periods. The key insight was that a neural network needs both a long-term memory pathway resistant to gradient vanishing and a sophisticated control system for deciding what information to preserve, what to discard, and what to make available at any given moment.

At the heart of the LSTM lies the cell state, a dedicated memory channel that runs through the entire sequence. Unlike the hidden state in a traditional RNN, which gets completely rewritten at each time step through a nonlinear transformation, the cell state can carry information forward with minimal modification. This pathway provides a route for gradients to flow backward through time without repeatedly passing through squashing nonlinearities that cause vanishing. The cell state can be thought of as a conveyor belt carrying information through time, with the ability to selectively add or remove information at each step.

Controlling access to this memory pathway are three gating mechanisms, neural structures that learn to regulate information flow. Each gate consists of a sigmoid layer that outputs values between 0 and 1, acting as a learned filter. An output of 0 means "block this information completely," while an output of 1 means "let everything through." Values between 0 and 1 allow partial information flow. By learning the parameters of these gates, the network learns what information is important to remember, what can be forgotten, and what should be exposed at the current time step.

The forget gate examines the previous hidden state and current input to decide what information from the previous cell state should be discarded. This allows the network to deliberately forget irrelevant information, preventing the memory from becoming cluttered with outdated context. If we are processing a document with multiple paragraphs on different topics, the forget gate can learn to discard information about the previous paragraph when starting a new one.

The input gate determines what new information should be added to the cell state. It works in two parts: a gate that decides which values to update, and a tanh layer that creates candidate values that could be added. Together, these components let the network selectively incorporate new information based on its relevance. When processing a sentence, the input gate might learn to strongly incorporate the subject noun, moderately incorporate descriptive adjectives, and largely ignore filler words like "the" or "a."

The output gate controls what information from the cell state becomes visible in the hidden state that gets passed to subsequent layers or used for predictions. This separation between what is remembered and what is output allows the network to maintain information in the cell state that might be needed later without necessarily using it right now. The cell state might remember that we are in the middle of a quoted statement, even though this information is only relevant for certain predictions.

The complete LSTM update process, while more complex than a simple RNN, creates a powerful memory system. At each time step, the network first decides what to forget, then what new information to incorporate, then updates the cell state by applying these decisions, and finally determines what to output. This orchestrated sequence of operations, all learned from data, enables LSTMs to automatically discover what information needs to be remembered across long time spans and what can be safely discarded.