LSTM Gradient Flow: Constant Error Carousel Explained

In the previous chapters, we saw how vanilla RNNs suffer from vanishing gradients and how LSTMs introduce gates to control information flow. But we haven't yet examined the mathematical reason why LSTMs solve the vanishing gradient problem. The answer lies in how gradients flow through the cell state, a mechanism Hochreiter and Schmidhuber called the constant error carousel in their original 1997 paper.

This chapter analyzes LSTM gradient flow from first principles. We'll derive the gradient equations, show why the cell state acts as a "gradient highway," compare gradient behavior between vanilla RNNs and LSTMs quantitatively, examine how the forget gate modulates gradient magnitude, and explore the deep analogy between LSTM cell state connections and residual networks. Along the way, we'll work through a concrete numerical example, analyze initialization strategies in detail, examine peephole connections, discuss stacked and bidirectional configurations, and trace the historical influence of the constant error carousel on modern architecture design. By the end, you'll understand that LSTMs avoid vanishing gradients, why they do so, and under what conditions they still require careful treatment.

The vanishing gradient problem in vanilla RNNs arises because every step of backpropagation through time multiplies by both a weight matrix and an activation function derivative. Over many timesteps, these repeated multiplications compound, and gradients shrink exponentially toward zero. The LSTM sidesteps this by creating a path where gradients can flow with far fewer multiplicative barriers.

To understand why the LSTM solution is elegant, we first need to recall why vanilla RNNs fail. As we derived in the Backpropagation Through Time and Vanishing Gradients chapters, the gradient of the loss $L$ with respect to the hidden state $\mathbf{h}_k$ at an earlier timestep $k$, when the loss is computed at timestep $T$, involves a product of Jacobians:

where each local Jacobian is:

where:

• $\mathbf{z}_t = \mathbf{W}_{hh} \mathbf{h}_{t-1} + \mathbf{W}_{xh} \mathbf{x}_t + \mathbf{b}$: the pre-activation at timestep $t$
• $\text{diag}(\tanh'(\mathbf{z}_t))$: a diagonal matrix of elementwise $\tanh$ derivatives, all values between 0 and 1
• $\mathbf{W}_{hh}$: the recurrent weight matrix

The product of $T - k$ such matrices, each with spectral norm typically less than 1 (because $|\tanh'(z)| \leq 1$ and we need $\lambda_{\max}(\mathbf{W}_{hh}) < 1$ for stable training), drives the gradient toward zero exponentially in $T - k$. For sequences of length 100 or more, gradients can shrink to $10^{-30}$ or smaller, making them numerically meaningless.

This practical inconvenience creates a fundamental barrier to learning. The optimizer receives near-zero gradient signals for parameters that affect early timesteps, so those parameters barely update. The network is effectively blind to its own behavior at the beginning of a sequence, no matter how many training examples you throw at it. Increasing the learning rate doesn't help; it only amplifies the near-zero signals into slightly larger near-zero signals, or risks blowing up the hidden state path entirely. The problem is architectural rather than a tuning issue.

The LSTM cell state update equation is:

where:

• $\mathbf{c}_t \in \mathbb{R}^h$: the cell state at timestep $t$
• $\mathbf{f}_t \in \mathbb{R}^h$: the forget gate activations (values in $(0, 1)$)
• $\mathbf{c}_{t-1} \in \mathbb{R}^h$: the cell state from the previous timestep
• $\mathbf{i}_t \in \mathbb{R}^h$: the input gate activations
• $\tilde{\mathbf{c}}_t \in \mathbb{R}^h$: the candidate cell state
• $\odot$: element-wise (Hadamard) multiplication

This update has two critical structural properties. First, it is additive: the new cell state is the old cell state scaled by $\mathbf{f}_t$ plus a new term. Second, the previous cell state passes through only an element-wise multiplication, not a full matrix multiplication. There is no $\mathbf{W}_{hh}$ multiplying the cell state as it travels through time.

Now consider the gradient of the loss with respect to the cell state at an earlier timestep. Using the chain rule through the cell state pathway only:

where:

• $\frac{\partial L}{\partial \mathbf{c}_T}$: the loss gradient at the final timestep
• $\prod_{t=k+1}^{T} \mathbf{f}_t$: the element-wise product of forget gate values from $k+1$ to $T$

Each element of this product is a product of $T - k$ values from the forget gate, all in $(0, 1)$. Unlike the vanilla RNN, there is no weight matrix and no nonlinearity derivative in this path. The only multiplicative factor is the forget gate activations.

The name "constant error carousel" was chosen deliberately. In the original paper, Hochreiter and Schmidhuber showed that if the forget gate produces a value of exactly 1 for some dimension, the gradient of that dimension is preserved exactly from one timestep to the next: a constant error signal circling in the carousel of time without decay. In practice, gates are never exactly 1, but they can be close enough that useful gradient signal survives hundreds of timesteps, something impossible for vanilla RNNs.

The word "carousel" is also apt. Imagine a merry-go-round that keeps spinning without friction. Each time the error (gradient) passes a given point in time, it arrives with the same magnitude it had before. There's no damping mechanism, no weight matrix to shrink it, no nonlinearity to saturate it. When forget gates are near 1, the LSTM approximates this frictionless carousel. When forget gates are near 0, the carousel stops, and gradients from earlier times can no longer reach the current position. The LSTM thus gives the network explicit control over whether the carousel continues spinning in any given dimension.

The 1997 paper was remarkable for its time because it took the vanishing gradient problem seriously as a mathematical obstacle rather than an empirical nuisance. Hochreiter had already identified the problem in his 1991 diploma thesis, and the LSTM was his principled architectural response. Understanding this history helps you appreciate that the constant error carousel is not an accident or an empirical observation; it's a deliberate design goal, baked into the architecture from the start.

To deeply understand why the cell state path is so much better for gradient flow, we need to compare how addition and multiplication behave during backpropagation. This comparison is the core insight behind LSTMs, ResNets, Highway Networks, and many other modern architectures.

Consider a simple computation: $z = x + y$. The partial derivatives are $\partial z / \partial x = 1$ and $\partial z / \partial y = 1$. During backpropagation, if we receive an upstream gradient $\partial L / \partial z$, then:

Addition is a gradient router: it copies the incoming gradient unchanged to both inputs. This means that in a long chain of additions, the gradient at the start of the chain equals the gradient at the end. No decay, no amplification.

This property has a direct implication. If you can route a gradient signal through an all-addition path from a loss function back to an early parameter, that parameter receives the same magnitude of gradient signal regardless of how many "steps" it took to get there. The distance in time (or depth) becomes irrelevant. The gradient doesn't know how far it has traveled through an additive path.

Now consider $z = f \cdot c$ where $f$ is a learnable gate value. The partial derivative with respect to $c$ is $\partial z / \partial c = f$. During backpropagation:

A single element-wise multiplication scales the gradient by $f$. Chaining $n$ such multiplications:

If each $f_t$ is 0.9, then after 50 timesteps: $0.9^{50} \approx 0.005$. After 100 steps: $0.9^{100} \approx 0.00003$. The gradient has decayed to essentially nothing. But if each $f_t$ is close to 1, say 0.99, then $0.99^{100} \approx 0.37$: still 37% of the original signal survives.

This is the key contrast with vanilla RNNs. In a vanilla RNN, the multiplicative factor at each step is $\text{diag}(\tanh'(\mathbf{z}_t)) \cdot \mathbf{W}_{hh}$, whose spectral norm is typically well below 1. In an LSTM, the multiplicative factor along the cell state path is just $\mathbf{f}_t$, a vector of sigmoid outputs. The LSTM can be trained to keep these values near 1 for dimensions where long-term memory is needed.

The main difference lies in who controls the multiplier, rather than in its magnitude. In the vanilla RNN, the gradient multiplier is determined by the weight matrix and the input at each step. The network has no way to "decide" to preserve gradients in certain dimensions; the gradient multiplier is the byproduct of the same weights that compute the hidden state transformation. In the LSTM, the forget gate is a learned, dedicated mechanism for controlling gradient flow. The network can separately learn "how to transform information" (through the candidate cell state) and "how to protect gradient flow" (through the forget gate). This separation of concerns is architecturally elegant and practically powerful.

Let's derive the full gradient through the cell state more carefully. The partial derivative of $\mathbf{c}_t$ with respect to $\mathbf{c}_{t-1}$ is:

This looks complex, but the dominant term in practice is $\text{diag}(\mathbf{f}_t)$. The other terms involve derivatives of the gates with respect to the cell state, which in standard LSTMs (without peephole connections) are zero: the gates depend on $\mathbf{h}_{t-1}$ and $\mathbf{x}_t$, not directly on $\mathbf{c}_{t-1}$.

For a standard LSTM:

Since $\tilde{\mathbf{c}}_t = \tanh(\mathbf{W}_c \mathbf{x}_t + \mathbf{U}_c \mathbf{h}_{t-1} + \mathbf{b}_c)$ and $\mathbf{h}_{t-1} = \mathbf{o}_{t-1} \odot \tanh(\mathbf{c}_{t-1})$, there is an indirect path through $\mathbf{h}_{t-1}$. But this path still passes through a tanh nonlinearity and an additional gate multiplication, making it substantially smaller than the direct $\mathbf{f}_t$ path.

The primary gradient flow through the cell state is thus:

This approximation captures the essence of the constant error carousel.

Abstract derivations are easier to internalize when paired with a concrete example. Let's trace a gradient backward through a small LSTM for exactly three timesteps, keeping all numbers visible.

Consider an LSTM with a single hidden dimension ($h = 1$) processing a sequence of three timesteps. This is a toy model, but it captures the essential structure. At each step, the gate values are scalars rather than vectors.

Suppose the learned (or assumed for this example) gate values are:

• Timestep 1: $f_1 = 0.95$, $i_1 = 0.3$, $o_1 = 0.8$
• Timestep 2: $f_2 = 0.90$, $i_2 = 0.5$, $o_2 = 0.7$
• Timestep 3: $f_3 = 0.85$, $i_3 = 0.4$, $o_3 = 0.9$

And the candidate cell state values (the $\tanh$ of the gate pre-activation) are:

• Timestep 1: $\tilde{c}_1 = 0.6$
• Timestep 2: $\tilde{c}_2 = -0.3$
• Timestep 3: $\tilde{c}_3 = 0.7$

We start with $c_0 = 0$.

The cell state evolves as $c_t = f_t \cdot c_{t-1} + i_t \cdot \tilde{c}_t$:

Step 1: $c_1 = 0.95 \cdot 0 + 0.3 \cdot 0.6 = 0.18$

Step 2: $c_2 = 0.90 \cdot 0.18 + 0.5 \cdot (-0.3) = 0.162 - 0.15 = 0.012$

Step 3: $c_3 = 0.85 \cdot 0.012 + 0.4 \cdot 0.7 = 0.0102 + 0.28 = 0.2902$

The cell state carries information across time, here starting at 0, rising to 0.18, dipping to 0.012 (the negative candidate overwriting most of the stored value), then jumping to 0.29.

Suppose the loss at timestep 3 has gradient $\partial L / \partial c_3 = 1.0$ with respect to the cell state.

We ask: what is the gradient with respect to $c_0$? In the cell state path, this is:

Nearly 73% of the original gradient reaches $c_0$ through the cell state path. Now compare what would happen in a vanilla RNN. The multiplicative factor at each step would be $\tanh'(z_t) \cdot w_{hh}$. For typical values, suppose these factors are $0.7$, $0.6$, $0.65$. Then:

Only 27% survives after three steps, and the decay rate is steeper per step. Extrapolating to 30 steps: if the RNN decay factor per step averages $0.64$ and the LSTM forget gate product per step averages $0.90$, then after 30 steps the RNN retains $0.64^{30} \approx 10^{-5}$ of the gradient while the LSTM cell retains $0.90^{30} \approx 0.042$. That is roughly a 4,000-fold difference in gradient magnitude.

The gradient $\partial L / \partial c_0$ tells the optimizer how to adjust anything that influenced $c_0$ in order to reduce the loss. Since $c_0 = 0$ by convention (the initial cell state), this gradient acts on the initial state itself. In practice, many implementations make $c_0$ trainable: the optimizer can learn a better starting cell state for the task.

More importantly, the cell state gradient also flows back through the input gate and candidate cell state at step 1 to reach the weights that processed the very first input. This is how the LSTM achieves long-range credit assignment: the cell state acts as a near-lossless wire carrying the loss signal backward to wherever it needs to go.

The forget gate plays a dual role in the LSTM: it controls what information is retained in the cell state during the forward pass, and it modulates the gradient magnitude during the backward pass. These two roles are inseparable, and understanding their relationship clarifies why the LSTM is such a well-designed architecture.

During the forward pass, the forget gate scales $\mathbf{c}_{t-1}$ when computing $\mathbf{c}_t$. During backpropagation, the same forget gate values scale the gradient when it flows backward through the cell state. The forget gate values used in the forward pass become the gradient multipliers in the backward pass.

This has a remarkable implication: when the network learns to retain information by setting the forget gate near 1 (meaning "keep this memory"), it simultaneously creates a low-resistance gradient highway through that dimension of the cell state. The architectural choices that enable long-term memory also enable long-range gradient flow. The two desiderata reinforce each other: wanting to learn long-range dependencies means wanting to preserve cell state values, and preserving cell state values means allowing gradients to flow. The LSTM architecture makes these two goals isomorphic.

Conversely, when the network decides that some information is no longer needed and sets the forget gate near 0, it simultaneously blocks gradient flow in that dimension. This means the optimizer cannot assign credit to inputs from before the "forgetting" event for whatever happens after. In some sense, this is correct behavior: if the network has decided to discard old information, there's no useful signal to be had by looking even further back. The gradient barrier and the information barrier coincide.

The forget gate can take values anywhere in $(0, 1)$, and the resulting gradient behavior spans a wide range. Understanding these regimes helps you reason about LSTM behavior during training.

When $f_t \approx 0$: The network has decided to forget the previous cell state in this dimension. Gradients also cannot flow backward through this dimension, effectively severing the credit assignment link to earlier inputs. This is appropriate when earlier inputs are irrelevant. For example, in a sentence classification task, the forget gate might close after a sentence boundary, preventing earlier sentence context from influencing predictions about the current sentence.

When $f_t \approx 1$: The network has decided to preserve the cell state in this dimension. Gradients flow nearly unchanged, enabling credit to be assigned to inputs many timesteps in the past. This is the constant error carousel in action. A language model that has encountered an opening quotation mark might keep the forget gate open in a "waiting for close quote" dimension, allowing it to learn that the distribution of words inside quotes differs from words outside quotes.

When $f_t \approx 0.5$: The gradient is halved at each timestep. Over $T$ steps, the gradient decays as $0.5^T$, similar to the vanilla RNN but potentially slower depending on the weight matrix spectral radius. This regime provides neither strong memory nor strong forgetting: it's a neutral state, typical in untrained or poorly initialized networks.

This analysis reveals a subtle tension: for the LSTM to learn long-range dependencies, it must learn to set forget gate values near 1 for the relevant dimensions. But if the forget gate starts near 1 for all dimensions, gradients flow freely at the cost of the network potentially failing to clear stale information. Good LSTM training involves learning forget gate values that are near 1 when information should be retained and near 0 when it should be discarded.

A useful analogy: think of the cell state as a highway and the forget gate as a throttle on each lane. When the throttle is fully open (forget gate near 1), traffic (gradient signal) flows freely. When it's fully closed (forget gate near 0), the lane is blocked. Unlike the vanilla RNN, which has a traffic jam at every intersection (every timestep involves a full matrix multiplication), the cell state highway can have long stretches of open road where information and gradients travel without obstruction.

This analogy extends further. The input gate and candidate cell state contribute an on-ramp at each timestep: new information can merge onto the highway. The output gate controls an off-ramp: the highway can contribute to the hidden state. But the core gradient flow along the highway is governed by the forget gate alone.

To make the traffic analogy precise: in a vanilla RNN, every intersection has a speed trap that slows traffic. The slow-down factor at each intersection is determined by the weight matrix and the local input, and there's no way to ask for exceptions. In the LSTM, the highway has a control mechanism that can be set differently for each lane. Some lanes might flow at full speed (forget gate near 1), others might be moderately throttled (forget gate near 0.7), and others might be completely blocked (forget gate near 0). The control mechanism is learned from data, so the network can discover which lanes need to be open for the task at hand.

The LSTM does not completely eliminate the vanishing gradient problem; it creates a path where vanishing is far less likely. To understand this precisely, we need to distinguish between the different gradient paths through an LSTM.

When we backpropagate through an LSTM, gradients flow through multiple paths simultaneously. Consider the gradient flowing from timestep $T$ back to timestep $k$. There are several routes:

The cell state path flows directly through the cell state with only forget gate multiplications. As derived above, this path has the gradient:

The hidden state path flows through the hidden states, passing through the output gate and the tanh nonlinearity at each step, in addition to the cell state dynamics. This path is much more complex and does experience more gradient attenuation.

The gate paths flow through the individual gate computations (forget, input, output, candidate), each involving sigmoid or tanh nonlinearities and weight matrices.

The key insight is that during optimization, the gradient flowing through the cell state path provides the essential long-range credit assignment signal. The other paths contribute additional gradient information for nearby timesteps. Together, they enable the LSTM to learn both long-range and short-range dependencies. You can think of the cell state path as the "slow lane" for long-distance gradient traffic, while the hidden state path is the "fast lane" for short-range gradient adjustment.

For the LSTM cell state path, the gradient vanishes when:

This happens when forget gate values are consistently small across many timesteps. Unlike vanilla RNNs, the forget gate values are learned, and the network can be trained to keep them near 1 when long-range memory is needed. The vanilla RNN has no such control mechanism; its gradient multipliers are determined by fixed weight matrices applied to potentially varying inputs, with no learned switch to protect gradient flow.

Additionally, because the cell state path involves only element-wise operations (no matrix multiplication), the spectral radius concern that drives vanilla RNN gradient explosions and vanishing is absent. There is no matrix whose eigenvalues need to be tuned; there are only per-dimension forget gate values between 0 and 1.

The LSTM does not solve all gradient problems. The gradient flowing through the hidden state path still experiences attenuation. The output gate tanh nonlinearity has a maximum derivative of 1 but typically much smaller values. The gradient path from the hidden state through the output gate to the cell state introduces multiplicative factors less than 1.

If the forget gate learns to produce very small values, the cell state path itself can vanish. In sequence modeling tasks where inputs frequently reset the relevant context, this is appropriate behavior. But if the network incorrectly learns small forget gate values for dimensions that should carry long-range information, it will fail to learn those dependencies.

The LSTM provides the architectural possibility of vanishing-free gradient flow. Whether that possibility is realized depends on the training dynamics, initialization, and the structure of the target task.

Rather than reasoning about gradient flow in the abstract, we can measure it directly. Comparing the empirical gradient norms for vanilla RNNs and LSTMs trained on a simple sequence task reveals the theoretical differences in striking quantitative terms.

A common approach is to initialize both networks with the same architecture size, train on identical data for a small number of steps, and then measure how the gradient norm with respect to the initial hidden state (or cell state for LSTMs) varies as a function of sequence length.

For a sequence of length $T$, the relevant quantity is:

for the RNN, and:

for the LSTM cell state path. As $T$ increases, how do these norms change?

The comparison above uses a simple simulation, but we can also reason analytically about the difference in gradient norm decay rates.

For a vanilla RNN with hidden size $h$, the gradient norm decays as:

where $\rho$ is the effective per-step gradient multiplier, related to the spectral radius of $\mathbf{W}_{hh}$ and the average $\tanh$ derivative. For stable vanilla RNNs (those that don't explode), $\rho < 1$, and typically $\rho$ is in the range 0.7 to 0.95.

For the LSTM cell state path with forget gate values initialized with bias $b_f$, the gradient decay rate is:

where $\sigma$ is the sigmoid function. With $b_f = 1$, $\rho_{\text{LSTM}} \approx \sigma(1) \approx 0.73$. With $b_f = 2$, $\rho_{\text{LSTM}} \approx 0.88$. With $b_f = 3$, $\rho_{\text{LSTM}} \approx 0.95$.

This is why many practitioners initialize LSTM forget gate biases with positive values (commonly between 1 and 3). A positive forget gate bias starts the network in a "remember more" regime, which creates favorable gradient flow conditions at the beginning of training.

The forget gate bias deserves special attention. PyTorch's default initialization sets all LSTM biases to zero, giving initial forget gate values near 0.5. But Jozefowicz et al. (2015) showed empirically that initializing the forget gate bias to 1 improves performance on many tasks. Gers and Schmidhuber (2000) used even larger biases.

The theory aligns with the practice: a forget gate bias of 1 gives an initial forget gate value of $\sigma(1) \approx 0.73$ per step, so the initial gradient decay over 100 steps is $0.73^{100} \approx 1.5 \times 10^{-13}$. That still vanishes! But over shorter ranges (say, 10 to 20 steps), the gradient is still meaningful, giving the optimizer a foothold to begin adjusting weights. From that starting point, the network can learn to push forget gate values toward 1 in the relevant dimensions, improving long-range gradient flow over time.

With a bias of 3, the initial forget gate is $\sigma(3) \approx 0.95$, so the decay over 100 steps is $0.95^{100} \approx 0.0059$: still 0.6% of the original gradient survives, giving a meaningful learning signal even for very long sequences.

The choice of forget gate bias is thus a form of implicit regularization. A low bias says "start by forgetting most things; learn to remember selectively." A high bias says "start by remembering everything; learn to forget selectively." For most NLP tasks, learning to remember selectively from a forgetting starting point is harder than learning to forget selectively from a remembering starting point, which is why positive biases generally help.

This initialization strategy is easy to implement in PyTorch. After creating an LSTM layer, you can directly set the forget gate bias. PyTorch stores the biases as a concatenation of four sub-vectors in order: input gate, forget gate, cell gate, and output gate, each of length hidden_size. So the forget gate bias occupies the range [hidden_size : 2*hidden_size] in the bias vectors:

The table quantifies the impact of forget gate bias initialization. A bias of 3 preserves 0.6% of the gradient over 100 steps, while a bias of 0 leaves essentially nothing. These fractions may seem small, but they represent the difference between a completely zero gradient signal and a meaningful one that can drive learning.

Understanding the LSTM cell state gradient path is greatly aided by comparing it to residual networks (ResNets), introduced by He et al. (2016). The two architectures solve the same fundamental problem using the same mathematical trick: creating an additive shortcut for gradient flow.

In a standard deep network, each layer transforms its input: $\mathbf{x}_{l+1} = F(\mathbf{x}_l)$ where $F$ includes weight matrices and nonlinearities. During backpropagation, the gradient through $n$ such layers involves a product of $n$ Jacobians, which can vanish.

ResNets add a skip connection: $\mathbf{x}_{l+1} = F(\mathbf{x}_l) + \mathbf{x}_l$. Now the gradient is:

The identity matrix $\mathbf{I}$ ensures that even if the network learns $F(\mathbf{x}_l) \approx 0$ (effectively turning off the learned transformation), the gradient still flows backward with magnitude 1. The addition creates a guaranteed gradient highway.

The LSTM cell state update:

is structurally analogous to the ResNet update:

In both cases, the output is the sum of the input (scaled by a gate in the LSTM case, or passed through unchanged in ResNet) plus a learned transformation. In both cases, the gradient of the output with respect to the input contains a term that doesn't pass through the learned transformation.

The key difference is that the LSTM's skip connection is learnable (the forget gate scales the cell state), while ResNet's skip connection passes the input unchanged. The LSTM therefore has more flexibility: it can open or close the gradient highway depending on what it has learned, while ResNet always keeps the highway fully open.

This difference has tradeoffs. ResNet's unconditional skip connection guarantees gradient flow without any training: even an untrained ResNet has a clear gradient path. The LSTM's gated skip connection needs to be learned into the "open" position before it starts helping. This is one reason why forget gate bias initialization matters so much for LSTMs: it puts the gate in an approximately "open" position before training starts, mimicking the ResNet's unconditional highway.

This analogy clarifies why the cell state is the "memory" component and the hidden state is the "communication" component. The cell state is the gradient highway, designed for long-range information storage. The hidden state is exposed to the outside world (fed to output layers, used by attention mechanisms, passed to the next layer), but it is more "narrow" in gradient flow because it passes through additional nonlinearities.

Later architectures like the GRU (which we'll explore in the next chapter) and Transformer architectures incorporate the residual connection insight even more directly. Transformers use explicit skip connections between every sublayer. This ensures that gradient flow is guaranteed at every depth level. In a sense, the LSTM's cell state foreshadowed this design pattern: create an additive path that bypasses the heavy computations so that gradient can flow freely. The progression from LSTM to ResNet to Transformer represents an evolution in how clearly and explicitly this design principle is applied.

In the standard LSTM, the gates (forget, input, output) are computed using only the previous hidden state $\mathbf{h}_{t-1}$ and the current input $\mathbf{x}_t$. They cannot directly observe the cell state $\mathbf{c}_{t-1}$ or $\mathbf{c}_t$. Gers and Schmidhuber (2000) proposed peephole connections as an extension that gives gates direct access to the cell state.

With peephole connections, the forget, input, and output gate equations become:

where:

• $\mathbf{V}_f, \mathbf{V}_i \in \mathbb{R}^h$: peephole weight vectors for forget and input gates (connected to $\mathbf{c}_{t-1}$)
• $\mathbf{V}_o \in \mathbb{R}^h$: peephole weight vector for output gate (connected to $\mathbf{c}_t$)
• All $\odot$ represent element-wise multiplication, making these diagonal weight connections

Note that the forget and input gates see $\mathbf{c}_{t-1}$ (the previous cell state), while the output gate sees $\mathbf{c}_t$ (the current cell state, available after the cell state update). This asymmetry is intentional.

Peephole connections change the gradient calculation because now the gate values depend on the cell state. The full partial derivative $\partial \mathbf{c}_t / \partial \mathbf{c}_{t-1}$ (which was approximately $\text{diag}(\mathbf{f}_t)$ in the standard LSTM) now includes additional terms from the peephole connections. The peephole terms add complexity but also allow the network to make gating decisions based on cell state values, potentially enabling more precise control over memory retention. Greff et al. (2017) conducted an extensive ablation study and found that peephole connections provide modest improvements on some tasks but are not always worth the additional complexity.

Peephole connections are most useful for tasks that require precise timing. Consider counting the exact number of steps between two events, or generating periodic patterns at exact intervals. In these cases, the cell state encodes timing information (e.g., a counter), and the gates need to make decisions based on the counter value. Peepholes allow the gate to directly read the counter rather than reading it indirectly through the hidden state.

The distinction matters because the hidden state $\mathbf{h}_{t-1} = \mathbf{o}_{t-1} \odot \tanh(\mathbf{c}_{t-1})$ is a lossy transformation of the cell state: it applies a tanh nonlinearity that compresses values into $[-1, 1]$, and then multiplies by the output gate, which may attenuate the signal further. Reading the cell state directly through peepholes gives access to the uncompressed, unattenuated value.

For most NLP tasks, standard LSTMs without peepholes perform comparably, which is why peephole connections are rarely used in practice today.

The constant error carousel dramatically improves gradient flow through the cell state, but it does not eliminate all gradient problems in LSTMs. Exploding gradients remain a concern, and gradient clipping (as covered in the Gradient Clipping chapter of Part X) is still commonly used.

While the cell state path is protected by the forget gate, the hidden state path is not. The hidden state $\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)$ depends on the output gate and the cell state. Gradients flowing through the hidden state path encounter weight matrices ($\mathbf{W}_o, \mathbf{U}_o$) whose spectral radii are not bounded below 1.

In practice, the combined effect of cell state and hidden state gradient paths is that vanishing gradients are largely mitigated by the cell state path (as long as forget gate values are reasonable), while exploding gradients remain possible through the hidden state path when weight matrices have large singular values.

Gradient clipping (typically clipping the global gradient norm to a threshold like 1.0 or 5.0) addresses the exploding gradient problem without interfering with the long-range learning enabled by the cell state path. The clipping operation truncates large gradient vectors to lie within a ball of fixed radius, preventing runaway updates while preserving the direction of the gradient.

To see why clipping is still necessary, consider what happens when the LSTM encounters an unusual input sequence. Even with a well-trained forget gate, certain inputs can trigger large activations in the output gate or the candidate cell state, leading to a spike in the gradient through the hidden state path. Without clipping, this spike could destabilize training by causing a very large parameter update that undoes previously learned behavior. With clipping, the spike is attenuated, and training continues smoothly.

Gradient clipping should generally be applied when training LSTMs on:

• Long sequences (T > 50) where rare events can cause occasional large gradients
• Tasks with variable-length sequences where long sequences dominate the gradient distribution
• Networks with large hidden dimensions (h > 512) where weight matrices are large
• Early in training before weights have stabilized

A practical rule: start with gradient clipping enabled (norm threshold 1.0) and monitor gradient norms during training. If gradient norms are consistently below the threshold (the clip never activates), you can remove it. If clipping activates frequently, the model may have an architectural issue or a learning rate that's too high.

During LSTM training, monitoring gradient health helps catch problems early. Several metrics are useful:

The gradient norm (before and after clipping) tells you how often clipping activates and whether the network's effective gradient magnitude is shrinking over time. A gradient norm that slowly trends downward during training can indicate the onset of vanishing gradients even in an LSTM, especially if the forget gate values have not yet been pushed toward 1.

The forget gate mean across the hidden dimensions tells you whether the network is learning to preserve information. A mean below 0.3 suggests the network is forgetting most things at each step, which is problematic for long-range tasks. If you observe this, consider increasing the forget gate bias or reducing the learning rate for the gate parameters.

The cell state variance across a batch tells you whether cell states are collapsing toward a constant or diverging toward very large values. Healthy cell states should have moderate variance that remains stable through training. A collapsing cell state variance often indicates that the forget gate is stuck near 0, while an exploding cell state variance suggests the input gate is consistently producing large values without adequate forgetting.

Combining these diagnostics into a training monitoring loop takes only a few extra lines of code and can save hours of debugging when something goes wrong.

Single-layer LSTMs are the building block, but real applications often use deeper or bidirectional variants. Understanding how gradient flow changes in these configurations is important for practical deployment.

A stacked LSTM consists of multiple LSTM layers where the hidden state output of one layer becomes the input sequence for the next. If we have $L$ layers, the output of layer $l$ at each timestep feeds into layer $l+1$:

Each layer has its own cell state and its own forget gate, so each layer has its own gradient highway. The key question is how gradients flow vertically (through layers) versus horizontally (through time).

Vertically, gradients flow from layer $L$ down to layer 1, passing through the hidden state outputs at each layer. This path does not have the cell state protection: the hidden state $\mathbf{h}_t^{(l)}$ is a compressed version of the cell state, and gradients flowing through the hidden state encounter the full multiplicative barriers of weight matrices and nonlinearities at each layer transition.

Horizontally, within each layer, gradients flow through that layer's cell state with the same forget gate protection as in a single-layer LSTM. So a stacked LSTM has strong horizontal gradient flow within each layer but potentially weaker vertical gradient flow across layers.

This suggests that for stacked LSTMs on very long sequences, the outer layers may struggle to learn from early parts of the sequence because their gradients must pass through multiple vertical layers before reaching layer 1. In practice, residual connections between LSTM layers can alleviate vertical gradient attenuation in stacked configurations: adding a skip connection $\mathbf{h}_t^{(l)} = \text{LSTM}_l(\cdot) + \mathbf{h}_t^{(l-1)}$ creates an additive path from lower to upper layers, replicating the ResNet benefit in the depth dimension.

A bidirectional LSTM processes the sequence in both directions: one LSTM goes left to right, another goes right to left. Their outputs at each position are concatenated (or summed) to produce a representation that captures context from both past and future.

From a gradient flow perspective, the two LSTMs are independent. Each has its own cell state, its own forget gate, and its own gradient highway. Gradients from the loss flow back through both LSTMs simultaneously (in parallel) during backpropagation. There is no interaction between the two gradient flows except through the shared loss function.

Bidirectional LSTMs are standard for sequence labeling tasks like named entity recognition and part-of-speech tagging, where the label at each position depends on both the preceding and following context. They are also useful for document classification when you want the model to have access to the full sequence before making a prediction.

The key practical consideration for bidirectional LSTMs is that they cannot be used for online (streaming) prediction, where you need to produce an output before seeing the entire sequence. This limits their applicability to generation tasks, where you're restricted to causal (left-to-right) models. For tasks like translation and summarization where the full input is available before generating any output, bidirectional encoders paired with unidirectional decoders are a natural choice, and this is exactly the architecture used in the original sequence-to-sequence models.

In this section we build a minimal LSTM that records gradient norms during a backward pass, then compare them against a vanilla RNN on the same task. This makes the theoretical gradient flow advantages directly observable.

We'll implement both models in PyTorch, train them on a simple sequence classification task, and measure gradient norms with respect to the first-position input.

After training, we can inspect the learned forget gate values to see whether the network has learned to keep them near 1 for the relevant dimensions.

The key parameters for LSTM gradient flow behavior are:

• forget_bias: Initial bias for the forget gate. Values between 1 and 3 are common. Higher values create better initial gradient flow conditions and are recommended for tasks with long sequences.
• hidden_size: The number of dimensions in the cell state. Larger hidden sizes provide more capacity for both memory storage and gradient flow.
• gradient clip norm: The global gradient norm threshold for clipping. Typically 1.0 to 5.0. Applied to prevent exploding gradients through the hidden state path.
• sequence length: The practical limit for gradient flow depends on the forget gate values. At f=0.95 per step, meaningful gradients survive up to 50 to 100 steps. At f=0.99, up to several hundred steps.

The constant error carousel appeared in print in 1997, in the journal Neural Computation. At that time, sequence modeling was dominated by Hidden Markov Models and early connectionist approaches, and the vanishing gradient problem was widely acknowledged as a severe obstacle but not yet solved in a principled way. The LSTM paper made several contributions simultaneously: it diagnosed the problem mathematically, designed an architecture that addressed it, and demonstrated the architecture on tasks that required memory of hundreds of timesteps, tasks that were completely out of reach for vanilla RNNs.

The 1997 paper did not yet include the forget gate. The original LSTM used a constant memory cell with only input and output gates, relying on a linear recurrence to prevent gradient decay. The forget gate was added by Gers et al. in 2000 in their "Learning to Forget: Continual Prediction with LSTM" paper. This addition turned out to be critical: without the ability to actively forget, the LSTM cell state would accumulate information indefinitely, causing problems for tasks with natural reset points like paragraph boundaries or topic changes.

The decade following the original paper saw steady but moderate adoption. LSTMs were applied to handwriting recognition, speech recognition, and language modeling, achieving state-of-the-art results in each domain. But the breakthrough that brought LSTMs into mainstream NLP was the sequence-to-sequence architecture introduced by Sutskever et al. in 2014. By using one LSTM to encode a source sentence into a fixed-size hidden state and another LSTM to decode the target sentence from that hidden state, they achieved dramatic improvements in machine translation.

The Attention Mechanism, introduced by Bahdanau et al. in 2015, extended sequence-to-sequence LSTMs further by allowing the decoder to attend to all encoder hidden states, rather than only the final one. This addressed a bottleneck in the LSTM encoder: the final hidden state could only carry a limited amount of information about the full input sequence. Attention made the encoder's entire hidden state sequence available to the decoder, dramatically improving performance on long sentences. The combination of LSTMs with attention became the dominant architecture for sequence-to-sequence tasks until Transformers arrived in 2017.

Understanding this historical progression is valuable because it illuminates the design decisions embedded in modern architectures. The Transformer's attention mechanism can be seen as taking the skip-connection insight from LSTM cell states and the attention insight from Bahdanau to their logical conclusion: every position can directly attend to every other position without any sequential bottleneck. The residual connections in Transformers mirror the LSTM cell state's additive gradient highway. The layer normalization in Transformers addresses the same training stability concern that motivated forget gate bias initialization in LSTMs. These connections are not coincidental; they reflect a continuous evolutionary thread in how the field has approached gradient flow and training stability.

The constant error carousel was a breakthrough when introduced in 1997. It provided a principled solution to the vanishing gradient problem that had stymied sequence modeling for years, and it directly enabled the LSTM to achieve state-of-the-art results on tasks requiring memory of hundreds to thousands of timesteps.

The gradient highway through the cell state does not completely solve the vanishing gradient problem. It relocates the difficulty. Whether gradients can flow depends on the learned forget gate values. If the network doesn't learn to open the gradient highway when needed (either due to poor initialization, a difficult optimization landscape, or irrelevant long-range dependencies), vanishing gradients remain possible. The LSTM provides the capability; training provides the realization.

The residual connection analogy turned out to be deeply predictive. When He et al. introduced ResNets in 2015, they were solving the same problem in feed-forward networks. Transformer architectures then applied residual connections at every sublayer, creating explicit gradient highways throughout the network depth. The principle that "addition enables gradient flow" connects the LSTM cell state, ResNet skip connections, and Transformer layer norms into a unified design philosophy.

One important limitation is that the LSTM cell state gradient highway only helps for the cell state dimension. The hidden state, which is the actual output of the LSTM at each step and the interface with the rest of the network, still experiences more gradient attenuation. For tasks where fine-grained hidden state representations matter (like sequence labeling, where the hidden state at each position is used for prediction), the gradient advantage may be less pronounced than the theory suggests.

Another limitation is computational: the four-gate architecture of the LSTM is computationally expensive compared to vanilla RNNs. Each timestep requires four matrix multiplications instead of one. This cost motivated the development of the GRU (covered in the next chapter), which achieves similar gradient flow improvements with two gates instead of four, reducing computation by roughly half. For tasks where the cell state bottleneck is not needed (tasks with moderate sequence lengths and relatively simple memory requirements), the GRU is often a better choice on computational grounds.

A third limitation is that LSTMs process sequences sequentially: each timestep must complete before the next can start. This sequential dependency prevents parallelization across the time dimension, which is a major performance bottleneck for modern GPU hardware. Transformers, which process all positions simultaneously through matrix operations, are dramatically faster to train on hardware designed for parallel computation. This parallelism advantage, more than any fundamental capability difference, is the primary reason Transformers have displaced LSTMs for most NLP tasks.

Despite these limitations, LSTMs remain the canonical example of how to engineer gradient flow. Their influence extends beyond RNNs: the idea of creating learnable gating mechanisms that can open or close gradient pathways appears in Highway Networks, ResNets, attention mechanisms, and numerous other architectures. Understanding the LSTM gradient flow analysis illustrates a fundamental design principle for deep networks.

The constant error carousel also highlights a broader lesson about neural network design: the gradient flow problem and the representation learning problem are deeply intertwined. A good architecture is not just one that can represent the required computations; it's one where the optimization process can efficiently discover the right parameters. The LSTM showed that these two goals can be addressed together through careful architectural choices, and this insight has guided the design of architectures from ResNets to Transformers.

The LSTM constant error carousel solves the vanishing gradient problem by creating an additive pathway for gradient flow through the cell state. Key takeaways:

• The cell state update is additive: $\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t$. Addition copies gradients unchanged, while multiplication scales them.
• The gradient through the cell state path decays only as a product of forget gate values, not as a product of weight matrix Jacobians and nonlinearity derivatives.
• The forget gate plays a dual role: it controls memory retention in the forward pass and modulates gradient magnitude in the backward pass. Learning to retain information means learning to preserve gradients.
• Forget gate bias initialization matters. Biases of 1 to 3 create favorable gradient flow conditions at the start of training and are recommended for tasks with long sequences.
• The cell state gradient path is directly analogous to the residual connection in ResNets: both use addition to create a gradient highway that bypasses the learned transformations.
• Peephole connections allow gates to observe the cell state directly, enabling more precise timing-based control, at the cost of additional parameters.
• Exploding gradients remain possible through the hidden state path. Gradient clipping (norm threshold 1.0 to 5.0) is still recommended for LSTM training on long sequences.
• The GRU, covered in the next chapter, achieves similar gradient flow improvements with a simpler two-gate architecture that reduces computational cost.
• Stacked LSTMs have strong horizontal gradient flow within each layer but may exhibit weaker vertical gradient flow across layers; residual connections between layers address this.
• Bidirectional LSTMs process sequences in both directions and provide representations that capture context from past and future, but cannot be used for online (streaming) prediction.
• LSTMs process sequences sequentially, which prevents parallelization across time and is the primary reason Transformers have displaced them for most NLP tasks at scale.
• The principle that "addition creates gradient highways" connects LSTM cell states, ResNet skip connections, and Transformer residual connections into a unified design philosophy for deep networks.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about LSTM gradient flow and the constant error carousel.