LSTM Gate Equations › The LSTM Cell at a Glance

Before diving into individual equations, let's establish the notation and see all the components together. An LSTM cell at time step $t$ receives three inputs and produces two outputs.

Inputs:

• $\mathbf{x}_t \in \mathbb{R}^d$: the input vector at time $t$ (e.g., a word embedding)
• $\mathbf{h}_{t-1} \in \mathbb{R}^h$: the hidden state from the previous time step
• $\mathbf{c}_{t-1} \in \mathbb{R}^h$: the cell state from the previous time step

Outputs:

• $\mathbf{h}_t \in \mathbb{R}^h$: the new hidden state
• $\mathbf{c}_t \in \mathbb{R}^h$: the new cell state

The hidden dimension $h$ is a hyperparameter you choose. A larger $h$ gives the model more capacity but increases computation and memory requirements.

The LSTM computes these outputs through four interacting components: the forget gate, input gate, candidate cell state, and output gate. Each gate is a small neural network that produces values between 0 and 1, acting as a soft switch that controls information flow.