Long Short-Term Memory (1997) › The Mathematics of Gating

To make these concepts precise, we need to examine the mathematical formulation of LSTM operations. While the architecture may appear complex at first, each component serves a clear purpose in the memory management system.

The forget gate computes:

$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$$

Here, $\sigma$ denotes the sigmoid function, which maps its input to values between 0 and 1. The weight matrix $W_f$ and bias $b_f$ are learned parameters, while $[h_{t-1}, x_t]$ represents the concatenation of the previous hidden state and current input. The resulting vector $f_t$ contains values between 0 and 1 for each element of the cell state, indicating how much of that element should be retained.

The input gate operates in two stages. First, it determines which values to update:

$$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$$

Then, a separate layer creates candidate values for addition to the cell state:

$$\tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C)$$

The tanh function produces values between -1 and 1, allowing both positive and negative updates to the cell state. The combination of $i_t$ and $\tilde{C}_t$ determines what new information gets incorporated.

With both forget and input gates computed, the cell state update becomes:

$$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$$

where $\odot$ denotes element-wise multiplication. This equation elegantly captures the memory update process: the previous cell state is scaled by the forget gate (discarding irrelevant information), and the new candidate values are scaled by the input gate (incorporating relevant new information), then these are combined through addition.

Finally, the output gate determines what gets exposed:

$$o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$$ $$h_t = o_t \odot \tanh(C_t)$$

The cell state passes through a tanh to push values into the range between -1 and 1, then the output gate selectively filters this signal to produce the hidden state.

This mathematical structure explains why LSTMs solve the vanishing gradient problem. Notice that the cell state update includes an additive term: $C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$. During backpropagation, gradients can flow backward through this addition operation without passing through repeated multiplicative transformations that cause exponential growth or decay. The cell state provides a highway for gradient flow, while the gates learn when to let gradients through and when to block them. When the forget gate learns to output values near 1 for certain cell state elements, it creates a path for gradients to flow unimpeded across many time steps, enabling the learning of long-range dependencies that were impossible for simple RNNs.