Long Short-Term Memory (1997) › Visualizing the Information Flow

The diagram above provides a window into how these mathematical operations combine to create a functioning memory system. Following the flow of information through the LSTM cell reveals the elegant choreography of memory management.

The cell state, represented by the horizontal green pathway running through the top of the diagram, forms the spine of the LSTM's memory system. Unlike the hidden state in a traditional RNN, which gets completely transformed at each time step, the cell state maintains a more direct connection between past and present. Information can flow along this pathway from the distant past to the current moment with relatively little modification, creating the "memory highway" that enables long-range dependency learning.

The three gates appear as the orange sigmoid layers, each examining the previous hidden state and current input to make decisions about memory management. The leftmost gate is the forget gate, determining what information from $C_{t-1}$ should be discarded. Moving right, we encounter the input gate mechanism, consisting of both a sigmoid gate and a tanh layer that together create and filter new candidate information. The rightmost gate is the output gate, controlling what information from the updated cell state becomes visible in the current hidden state.

The yellow circles representing pointwise operations show where the actual memory manipulation occurs. The multiplication operations serve as information filters, implementing the decisions made by the gates. When the forget gate outputs values near 0 for certain cell state dimensions, the corresponding multiplication nearly eliminates that information from the cell state. When it outputs values near 1, information passes through essentially unchanged. Because this filtering operates independently on each dimension, the network can retain one piece of information while discarding another according to its relevance.

The tanh operations perform a different but equally important role. The tanh that generates candidate values $\tilde{C}_t$ creates proposed updates to the cell state, bounded between -1 and 1 to prevent the cell state from growing without bound. The tanh applied before the output gate ensures that the hidden state remains in a controlled numerical range, improving training stability. Together, the sigmoid gates and tanh activations create a balanced system where information can be preserved over long periods without numerical instability.

Perhaps most remarkably, this architecture achieves its sophistication through relatively simple mathematical operations applied in a carefully designed sequence. There are no exotic functions or complex algorithms, just sigmoids, tanh, and basic arithmetic operations. The power emerges from how these simple pieces are arranged and how the gates learn to coordinate their actions. During training, the network discovers that certain gates should open when specific patterns appear in the input, that some information should be held in the cell state for many time steps, and that other information should be quickly discarded. This learned coordination transforms the LSTM from a static architecture into an adaptive memory system.