LSTM Gradient Flow › How the Forget Gate Modulates Gradient
Language AI HandbookAll chapters LSTM Gradient Flow › How the Forget Gate Modulates GradientOne section · Sections in this chapter5 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The forget gate plays a dual role in the LSTM: it controls what information is retained in the cell state during the forward pass, and it modulates the gradient magnitude during the backward pass. These two roles are inseparable, and understanding their relationship clarifies why the LSTM is such a well-designed architecture.
The Forget Gate During Backpropagation
During the forward pass, the forget gate scales when computing . During backpropagation, the same forget gate values scale the gradient when it flows backward through the cell state. The forget gate values used in the forward pass become the gradient multipliers in the backward pass.
This has a remarkable implication: when the network learns to retain information by setting the forget gate near 1 (meaning "keep this memory"), it simultaneously creates a low-resistance gradient highway through that dimension of the cell state. The architectural choices that enable long-term memory also enable long-range gradient flow. The two desiderata reinforce each other: wanting to learn long-range dependencies means wanting to preserve cell state values, and preserving cell state values means allowing gradients to flow. The LSTM architecture makes these two goals isomorphic.
Conversely, when the network decides that some information is no longer needed and sets the forget gate near 0, it simultaneously blocks gradient flow in that dimension. This means the optimizer cannot assign credit to inputs from before the "forgetting" event for whatever happens after. In some sense, this is correct behavior: if the network has decided to discard old information, there's no useful signal to be had by looking even further back. The gradient barrier and the information barrier coincide.
Gradient Behavior Under Different Forget Gate Regimes
The forget gate can take values anywhere in , and the resulting gradient behavior spans a wide range. Understanding these regimes helps you reason about LSTM behavior during training.
When : The network has decided to forget the previous cell state in this dimension. Gradients also cannot flow backward through this dimension, effectively severing the credit assignment link to earlier inputs. This is appropriate when earlier inputs are irrelevant. For example, in a sentence classification task, the forget gate might close after a sentence boundary, preventing earlier sentence context from influencing predictions about the current sentence.
When : The network has decided to preserve the cell state in this dimension. Gradients flow nearly unchanged, enabling credit to be assigned to inputs many timesteps in the past. This is the constant error carousel in action. A language model that has encountered an opening quotation mark might keep the forget gate open in a "waiting for close quote" dimension, allowing it to learn that the distribution of words inside quotes differs from words outside quotes.
When : The gradient is halved at each timestep. Over steps, the gradient decays as , similar to the vanilla RNN but potentially slower depending on the weight matrix spectral radius. This regime provides neither strong memory nor strong forgetting: it's a neutral state, typical in untrained or poorly initialized networks.
This analysis reveals a subtle tension: for the LSTM to learn long-range dependencies, it must learn to set forget gate values near 1 for the relevant dimensions. But if the forget gate starts near 1 for all dimensions, gradients flow freely at the cost of the network potentially failing to clear stale information. Good LSTM training involves learning forget gate values that are near 1 when information should be retained and near 0 when it should be discarded.
Gradient Highway Through the Cell State
A useful analogy: think of the cell state as a highway and the forget gate as a throttle on each lane. When the throttle is fully open (forget gate near 1), traffic (gradient signal) flows freely. When it's fully closed (forget gate near 0), the lane is blocked. Unlike the vanilla RNN, which has a traffic jam at every intersection (every timestep involves a full matrix multiplication), the cell state highway can have long stretches of open road where information and gradients travel without obstruction.
This analogy extends further. The input gate and candidate cell state contribute an on-ramp at each timestep: new information can merge onto the highway. The output gate controls an off-ramp: the highway can contribute to the hidden state. But the core gradient flow along the highway is governed by the forget gate alone.
To make the traffic analogy precise: in a vanilla RNN, every intersection has a speed trap that slows traffic. The slow-down factor at each intersection is determined by the weight matrix and the local input, and there's no way to ask for exceptions. In the LSTM, the highway has a control mechanism that can be set differently for each lane. Some lanes might flow at full speed (forget gate near 1), others might be moderately throttled (forget gate near 0.7), and others might be completely blocked (forget gate near 0). The control mechanism is learned from data, so the network can discover which lanes need to be open for the task at hand.
import numpy as np
def sigmoid(x):
return 1 / (1 + np.exp(-np.clip(x, -20, 20)))
# Demonstrate gradient decay: RNN vs LSTM for different forget gate values
T = 100 # sequence length
steps = np.arange(1, T + 1)
# Vanilla RNN: gradient multiplied by effective spectral norm at each step
# Typical effective multiplier ~0.85 per step for a stable vanilla RNN
rnn_mult = 0.85
rnn_gradient = rnn_mult**steps
# LSTM cell state path with different forget gate values
f_09 = 0.9 # moderate retention
f_095 = 0.95 # high retention
f_099 = 0.99 # very high retention
lstm_gradient_09 = f_09**steps
lstm_gradient_095 = f_095**steps
lstm_gradient_099 = f_099**stepsGradient magnitudes after 100 timesteps: Vanilla RNN (mult=0.85): 8.75e-08 LSTM cell (f=0.90): 2.66e-05 LSTM cell (f=0.95): 5.92e-03 LSTM cell (f=0.99): 3.66e-01
The gradient magnitudes after 100 timesteps show a dramatic difference. The vanilla RNN gradient decays to essentially zero, while the LSTM with a forget gate close to 1 retains a meaningful fraction of the original gradient. This is the quantitative advantage of the constant error carousel.
