LSTM Gate Equations › The Output Gate

The cell state now contains the LSTM's updated memory, but not all of this memory is relevant for the current moment. Consider a language model that has stored information about the subject of a sentence ("The cat"), the ongoing action ("is chasing"), and various contextual details. When predicting the next word, only some of this stored information is immediately relevant. The output gate acts as a filter, selecting which parts of the internal memory to expose for the current output.

This separation between internal memory (cell state) and external output (hidden state) is a key architectural insight. The cell state can accumulate and preserve information over long time spans, while the hidden state provides a task-relevant summary at each step. The output gate bridges these two representations.

Following the same pattern as the other gates, the output gate examines the current context:

where:

• $\mathbf{o}_t \in \mathbb{R}^h$: the output gate activation vector, controlling what to expose
• $\mathbf{W}_o \in \mathbb{R}^{h \times d}$: weight matrix that learns which input features signal "this memory is now relevant"
• $\mathbf{U}_o \in \mathbb{R}^{h \times h}$: weight matrix that learns which processing contexts warrant exposing certain memories
• $\mathbf{b}_o \in \mathbb{R}^h$: bias vector

The structure matches the other gates: linear combination followed by sigmoid. But the purpose is different. While the forget and input gates control what goes into memory, the output gate controls what comes out. This asymmetry allows the LSTM to store information that isn't immediately useful but may become relevant later.