Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

LSTM Gradient Flow › Comparison with Vanilla RNN Gradient Norms

Michael BrenndoerferMay 12, 20255 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 79LSTM Gradient Flow › Comparison with Vanilla RNN Gradient Norms

Reading section 8 of 17 · Comparison with Vanilla RNN Gradient Norms · 5 minSection 8 of 17 · 5 min

Section 8 of 17 · 5 min read

LSTM Gradient Flow › Comparison with Vanilla RNN Gradient NormsOne section · Sections in this chapter8 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The comparison above uses a simple simulation, but we can also reason analytically about the difference in gradient norm decay rates.

Exponential Decay Rate AnalysisLink Copied

For a vanilla RNN with hidden size $h$ , the gradient norm decays as:

\mathbb{E}[\|\nabla_{\mathbf{h}_0} L\|] \approx C \cdot \rho^T

where $\rho$ is the effective per-step gradient multiplier, related to the spectral radius of $\mathbf{W}_{hh}$ and the average $\tanh$ derivative. For stable vanilla RNNs (those that don't explode), $\rho < 1$ , and typically $\rho$ is in the range 0.7 to 0.95.

For the LSTM cell state path with forget gate values initialized with bias $b_f$ , the gradient decay rate is:

\rho_{\text{LSTM}} \approx \sigma(b_f)

where $\sigma$ is the sigmoid function. With $b_f = 1$ , $\rho_{\text{LSTM}} \approx \sigma(1) \approx 0.73$ . With $b_f = 2$ , $\rho_{\text{LSTM}} \approx 0.88$ . With $b_f = 3$ , $\rho_{\text{LSTM}} \approx 0.95$ .

This is why many practitioners initialize LSTM forget gate biases with positive values (commonly between 1 and 3). A positive forget gate bias starts the network in a "remember more" regime, which creates favorable gradient flow conditions at the beginning of training.

Forget Gate Bias InitializationLink Copied

The forget gate bias deserves special attention. PyTorch's default initialization sets all LSTM biases to zero, giving initial forget gate values near 0.5. But Jozefowicz et al. (2015) showed empirically that initializing the forget gate bias to 1 improves performance on many tasks. Gers and Schmidhuber (2000) used even larger biases.

The theory aligns with the practice: a forget gate bias of 1 gives an initial forget gate value of $\sigma(1) \approx 0.73$ per step, so the initial gradient decay over 100 steps is $0.73^{100} \approx 1.5 \times 10^{-13}$ . That still vanishes! But over shorter ranges (say, 10 to 20 steps), the gradient is still meaningful, giving the optimizer a foothold to begin adjusting weights. From that starting point, the network can learn to push forget gate values toward 1 in the relevant dimensions, improving long-range gradient flow over time.

With a bias of 3, the initial forget gate is $\sigma(3) \approx 0.95$ , so the decay over 100 steps is $0.95^{100} \approx 0.0059$ : still 0.6% of the original gradient survives, giving a meaningful learning signal even for very long sequences.

The choice of forget gate bias is thus a form of implicit regularization. A low bias says "start by forgetting most things; learn to remember selectively." A high bias says "start by remembering everything; learn to forget selectively." For most NLP tasks, learning to remember selectively from a forgetting starting point is harder than learning to forget selectively from a remembering starting point, which is why positive biases generally help.

This initialization strategy is easy to implement in PyTorch. After creating an LSTM layer, you can directly set the forget gate bias. PyTorch stores the biases as a concatenation of four sub-vectors in order: input gate, forget gate, cell gate, and output gate, each of length hidden_size. So the forget gate bias occupies the range [hidden_size : 2*hidden_size] in the bias vectors:

In[9]:

Code

import torch
import torch.nn as nn

lstm_demo = nn.LSTM(input_size=16, hidden_size=32, batch_first=True)
_hidden_size = 32

with torch.no_grad():
    lstm_demo.bias_ih_l0[_hidden_size : 2 * _hidden_size].fill_(1.0)
    lstm_demo.bias_hh_l0[_hidden_size : 2 * _hidden_size].fill_(1.0)

Out[10]:

Console

Forget gate initial value (after setting bias=1.0 in each): 0.8808
This matches sigma(2.0) = 0.8808

This simple change can measurably improve convergence on tasks with long-range dependencies, with no other changes to the architecture or training setup. The reason PyTorch uses two separate bias vectors (bias_ih and bias_hh) is historical: it mirrors the structure of the weight matrices. In practice, only the sum of the two bias vectors affects the gate computation, so setting both to 1.0 gives a combined forget gate pre-activation bias of 2.0, corresponding to $\sigma(2.0) \approx 0.88$ .

In[11]:

Code

# Analyze the effect of forget gate bias on gradient decay
biases = [0, 1, 2, 3, 4]
T_values = [10, 25, 50, 100]

results = {}
for b in biases:
    f_init = sigmoid(b)
    fractions = [f_init**T for T in T_values]
    results[b] = (f_init, fractions)

Out[12]:

Console

Gradient fraction surviving T timesteps by forget gate bias:
  Bias    f_init  T= 10  T= 25  T= 50  T=100
--------------------------------------------------------------
     0    0.5000  9.77e-04  2.98e-08  8.88e-16  7.89e-31
     1    0.7311  4.36e-02  3.97e-04  1.58e-07  2.48e-14
     2    0.8808  2.81e-01  4.19e-02  1.75e-03  3.07e-06
     3    0.9526  6.15e-01  2.97e-01  8.81e-02  7.76e-03
     4    0.9820  8.34e-01  6.35e-01  4.04e-01  1.63e-01

The table quantifies the impact of forget gate bias initialization. A bias of 3 preserves 0.6% of the gradient over 100 steps, while a bias of 0 leaves essentially nothing. These fractions may seem small, but they represent the difference between a completely zero gradient signal and a meaningful one that can drive learning.