LSTM Gate Equations › The Hidden State Computation

The final step converts the internal cell state into the external hidden state. This hidden state serves dual purposes: it becomes the LSTM's output for later processing (feeding into the next layer or making predictions), and it provides the recurrent context for the next time step.

The hidden state is computed by filtering the normalized cell state through the output gate:

$$\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)$$

where:

• $\mathbf{h}_t \in \mathbb{R}^h$: the new hidden state, the LSTM's output at this time step
• $\mathbf{o}_t \in \mathbb{R}^h$: the output gate activation (values between 0 and 1)
• $\tanh(\mathbf{c}_t)$: the cell state squashed to the range $(-1, 1)$

Notice that we apply $\tanh$ to the cell state before gating. This serves two important purposes:

1. Normalization: The cell state can accumulate values well outside $[-1, 1]$ through repeated additions. If the forget gate stays near 1 and the input gate keeps adding positive values, the cell state could grow to 10, 100, or larger. Applying $\tanh$ compresses these potentially large values back into a bounded range, preventing the hidden state from exploding.

2. Consistency: The hidden state feeds into subsequent layers and provides recurrent context. Having it bounded in $(-1, 1)$ makes the network's behavior more predictable and stable. Downstream layers can rely on receiving inputs within a consistent range.

With the hidden state computed, we've completed one full LSTM step. The hidden state $\mathbf{h}_t$ now flows in two directions:

1. Forward to the next layer (or to the output): providing the LSTM's representation of the sequence up to time $t$
2. Recurrently to the next time step: serving as $\mathbf{h}_{t-1}$ when processing $\mathbf{x}_{t+1}$

Meanwhile, the cell state $\mathbf{c}_t$ flows only recurrently, carrying the LSTM's long-term memory forward without being directly exposed to downstream processing. This dual-track architecture, with the cell state as a protected memory channel and the hidden state as a filtered output, is what gives LSTMs their ability to capture long-range dependencies.