LSTM Gate Equations › Summary

This chapter translated the intuition of LSTM gates into precise mathematics and working code. The key equations form a system where three gates control information flow through a memory cell.

The Gate Equations:

• Forget gate: $\mathbf{f}_t = \sigma(\mathbf{W}_f \mathbf{x}_t + \mathbf{U}_f \mathbf{h}_{t-1} + \mathbf{b}_f)$ decides what to discard from memory
• Input gate: $\mathbf{i}_t = \sigma(\mathbf{W}_i \mathbf{x}_t + \mathbf{U}_i \mathbf{h}_{t-1} + \mathbf{b}_i)$ decides how much new information to write
• Candidate: $\tilde{\mathbf{c}}_t = \tanh(\mathbf{W}_c \mathbf{x}_t + \mathbf{U}_c \mathbf{h}_{t-1} + \mathbf{b}_c)$ proposes new values to store
• Output gate: $\mathbf{o}_t = \sigma(\mathbf{W}_o \mathbf{x}_t + \mathbf{U}_o \mathbf{h}_{t-1} + \mathbf{b}_o)$ decides what to expose as output

The State Updates:

• Cell state: $\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t$ combines filtered memory with gated new information
• Hidden state: $\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)$ exposes filtered cell state as output

Parameter Count:

An LSTM cell has $4h(d + h + 1)$ parameters, where $d$ is the input dimension (e.g., word embedding size), $h$ is the hidden dimension, and the factor of 4 accounts for the four components (forget gate, input gate, candidate, output gate). This is four times the parameters of a vanilla RNN. This reflects the four separate learnable transformations.

Implementation Insights:

• Initializing the forget gate bias to 1 improves training by starting with full memory retention
• Combining weight matrices into single operations improves computational efficiency
• The cell state can grow unbounded, so applying tanh before the output gate maintains numerical stability

With these equations internalized, you can now trace exactly how information flows through an LSTM, debug unexpected behaviors, and make informed decisions about when to use this architecture. The next chapter examines how these equations enable better gradient flow compared to vanilla RNNs.