LSTM Gradient Flow › The Constant Error Carousel

The vanishing gradient problem in vanilla RNNs arises because every step of backpropagation through time multiplies by both a weight matrix and an activation function derivative. Over many timesteps, these repeated multiplications compound, and gradients shrink exponentially toward zero. The LSTM sidesteps this by creating a path where gradients can flow with far fewer multiplicative barriers.

To understand why the LSTM solution is elegant, we first need to recall why vanilla RNNs fail. As we derived in the Backpropagation Through Time and Vanishing Gradients chapters, the gradient of the loss $L$ with respect to the hidden state $\mathbf{h}_k$ at an earlier timestep $k$, when the loss is computed at timestep $T$, involves a product of Jacobians:

$$\frac{\partial L}{\partial \mathbf{h}_k} = \frac{\partial L}{\partial \mathbf{h}_T} \prod_{t=k+1}^{T} \frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_{t-1}}$$

where each local Jacobian is:

$$\frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_{t-1}} = \text{diag}(\tanh'(\mathbf{z}_t)) \cdot \mathbf{W}_{hh}$$

where:

• $\mathbf{z}_t = \mathbf{W}_{hh} \mathbf{h}_{t-1} + \mathbf{W}_{xh} \mathbf{x}_t + \mathbf{b}$: the pre-activation at timestep $t$
• $\text{diag}(\tanh'(\mathbf{z}_t))$: a diagonal matrix of elementwise $\tanh$ derivatives, all values between 0 and 1
• $\mathbf{W}_{hh}$: the recurrent weight matrix

The product of $T - k$ such matrices, each with spectral norm typically less than 1 (because $|\tanh'(z)| \leq 1$ and we need $\lambda_{\max}(\mathbf{W}_{hh}) < 1$ for stable training), drives the gradient toward zero exponentially in $T - k$. For sequences of length 100 or more, gradients can shrink to $10^{-30}$ or smaller, making them numerically meaningless.

This practical inconvenience creates a fundamental barrier to learning. The optimizer receives near-zero gradient signals for parameters that affect early timesteps, so those parameters barely update. The network is effectively blind to its own behavior at the beginning of a sequence, no matter how many training examples you throw at it. Increasing the learning rate doesn't help; it only amplifies the near-zero signals into slightly larger near-zero signals, or risks blowing up the hidden state path entirely. The problem is architectural rather than a tuning issue.

The LSTM cell state update equation is:

$$\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t$$

where:

• $\mathbf{c}_t \in \mathbb{R}^h$: the cell state at timestep $t$
• $\mathbf{f}_t \in \mathbb{R}^h$: the forget gate activations (values in $(0, 1)$)
• $\mathbf{c}_{t-1} \in \mathbb{R}^h$: the cell state from the previous timestep
• $\mathbf{i}_t \in \mathbb{R}^h$: the input gate activations
• $\tilde{\mathbf{c}}_t \in \mathbb{R}^h$: the candidate cell state
• $\odot$: element-wise (Hadamard) multiplication

This update has two critical structural properties. First, it is additive: the new cell state is the old cell state scaled by $\mathbf{f}_t$ plus a new term. Second, the previous cell state passes through only an element-wise multiplication, not a full matrix multiplication. There is no $\mathbf{W}_{hh}$ multiplying the cell state as it travels through time.

Now consider the gradient of the loss with respect to the cell state at an earlier timestep. Using the chain rule through the cell state pathway only:

$$\frac{\partial L}{\partial \mathbf{c}_k} = \frac{\partial L}{\partial \mathbf{c}_T} \odot \prod_{t=k+1}^{T} \mathbf{f}_t$$

where:

• $\frac{\partial L}{\partial \mathbf{c}_T}$: the loss gradient at the final timestep
• $\prod_{t=k+1}^{T} \mathbf{f}_t$: the element-wise product of forget gate values from $k+1$ to $T$

Each element of this product is a product of $T - k$ values from the forget gate, all in $(0, 1)$. Unlike the vanilla RNN, there is no weight matrix and no nonlinearity derivative in this path. The only multiplicative factor is the forget gate activations.

The name "constant error carousel" was chosen deliberately. In the original paper, Hochreiter and Schmidhuber showed that if the forget gate produces a value of exactly 1 for some dimension, the gradient of that dimension is preserved exactly from one timestep to the next: a constant error signal circling in the carousel of time without decay. In practice, gates are never exactly 1, but they can be close enough that useful gradient signal survives hundreds of timesteps, something impossible for vanilla RNNs.

The word "carousel" is also apt. Imagine a merry-go-round that keeps spinning without friction. Each time the error (gradient) passes a given point in time, it arrives with the same magnitude it had before. There's no damping mechanism, no weight matrix to shrink it, no nonlinearity to saturate it. When forget gates are near 1, the LSTM approximates this frictionless carousel. When forget gates are near 0, the carousel stops, and gradients from earlier times can no longer reach the current position. The LSTM thus gives the network explicit control over whether the carousel continues spinning in any given dimension.

The 1997 paper was remarkable for its time because it took the vanishing gradient problem seriously as a mathematical obstacle rather than an empirical nuisance. Hochreiter had already identified the problem in his 1991 diploma thesis, and the LSTM was his principled architectural response. Understanding this history helps you appreciate that the constant error carousel is not an accident or an empirical observation; it's a deliberate design goal, baked into the architecture from the start.