LSTM Gate Equations › The Cell State Update

We've now assembled all the ingredients: a forget gate that knows what to discard, an input gate that knows how much to write, and a candidate that proposes what to write. The cell state update combines these three components into a single equation that forms the heart of the LSTM.

This is where the "information highway" metaphor becomes concrete. The cell state flows through time, modified at each step by two operations: selective forgetting and selective addition. The key insight is that information can flow unchanged for many time steps if the forget gate stays near 1 and the input gate stays near 0. This enables LSTMs to capture long-range dependencies.

The new cell state is computed as:

$$\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t$$

where:

• $\mathbf{c}_t \in \mathbb{R}^h$: the new cell state
• $\mathbf{c}_{t-1} \in \mathbb{R}^h$: the previous cell state (the memory we're updating)
• $\mathbf{f}_t \in \mathbb{R}^h$: the forget gate activation (values between 0 and 1)
• $\mathbf{i}_t \in \mathbb{R}^h$: the input gate activation (values between 0 and 1)
• $\tilde{\mathbf{c}}_t \in \mathbb{R}^h$: the candidate cell state (values between -1 and 1)
• $\odot$: the Hadamard (element-wise) product, where $[\mathbf{a} \odot \mathbf{b}]_j = a_j \cdot b_j$

The equation has two terms, each serving a distinct purpose:

Term 1: Filtered Memory ($\mathbf{f}_t \odot \mathbf{c}_{t-1}$)

This term takes the previous cell state and scales each dimension by the corresponding forget gate value. If $f_t^{(j)} = 0.9$ for dimension $j$, then 90% of that dimension's previous value is retained. If $f_t^{(j)} = 0.1$, only 10% survives. This element-wise multiplication allows the network to selectively preserve some memories while erasing others, all within the same time step.

Term 2: New Information ($\mathbf{i}_t \odot \tilde{\mathbf{c}}_t$)

This term takes the candidate's proposal and scales it by the input gate. If $i_t^{(j)} = 0.8$ for dimension $j$, then 80% of the candidate's proposed value for that dimension gets written. If $i_t^{(j)} = 0.1$, the candidate's proposal is largely ignored. This gating prevents irrelevant information from contaminating the cell state.

The Sum: Combining Old and New

Adding these terms creates the new cell state. Each dimension independently combines its filtered old value with its gated new value. This additive structure gives gradients a direct path: during backpropagation, gradients can flow directly through the addition operation without being squashed by nonlinearities.

Let's walk through a concrete example to see exactly how the gates interact. Suppose the cell state has dimension $h = 4$ and at time $t$ we have:

• Previous cell state: $\mathbf{c}_{t-1} = [0.5, -0.3, 0.8, 0.1]$
• Forget gate: $\mathbf{f}_t = [0.9, 0.1, 0.7, 0.5]$
• Input gate: $\mathbf{i}_t = [0.2, 0.8, 0.3, 0.6]$
• Candidate: $\tilde{\mathbf{c}}_t = [0.4, 0.9, -0.2, 0.7]$

Step 1: Apply the forget gate to the previous cell state

$$\mathbf{f}_t \odot \mathbf{c}_{t-1} = [0.9, 0.1, 0.7, 0.5] \odot [0.5, -0.3, 0.8, 0.1] = [0.45, -0.03, 0.56, 0.05]$$

Step 2: Apply the input gate to the candidate

$$\mathbf{i}_t \odot \tilde{\mathbf{c}}_t = [0.2, 0.8, 0.3, 0.6] \odot [0.4, 0.9, -0.2, 0.7] = [0.08, 0.72, -0.06, 0.42]$$

Step 3: Sum to get the new cell state

$$\mathbf{c}_t = [0.45, -0.03, 0.56, 0.05] + [0.08, 0.72, -0.06, 0.42] = [0.53, 0.69, 0.50, 0.47]$$

Interpreting Each Dimension:

• Dimension 1 ($0.5 \to 0.53$): The forget gate was high (0.9), preserving most of the old value. The input gate was low (0.2), adding little new information. Result: minimal change, memory preserved.

• Dimension 2 ($-0.3 \to 0.69$): The forget gate was low (0.1), nearly erasing the old value. The input gate was high (0.8), strongly writing the candidate's positive value. Result: dramatic reversal, old memory replaced with new information.

• Dimension 3 ($0.8 \to 0.50$): Moderate forget gate (0.7) kept most of the old value. Low input gate (0.3) added a small negative candidate contribution. Result: gradual decrease.

• Dimension 4 ($0.1 \to 0.47$): Balanced gates (0.5 and 0.6) created a blend of old and new. Result: moderate change incorporating both sources.

This example illustrates the LSTM's power: each dimension of the cell state can be updated independently, with different balances of remembering and writing, all controlled by the learned gate values.