LSTM Gradient Flow › LSTM Gradient Flow

In the previous chapters, we saw how vanilla RNNs suffer from vanishing gradients and how LSTMs introduce gates to control information flow. But we haven't yet examined the mathematical reason why LSTMs solve the vanishing gradient problem. The answer lies in how gradients flow through the cell state, a mechanism Hochreiter and Schmidhuber called the constant error carousel in their original 1997 paper.

This chapter analyzes LSTM gradient flow from first principles. We'll derive the gradient equations, show why the cell state acts as a "gradient highway," compare gradient behavior between vanilla RNNs and LSTMs quantitatively, examine how the forget gate modulates gradient magnitude, and explore the deep analogy between LSTM cell state connections and residual networks. Along the way, we'll work through a concrete numerical example, analyze initialization strategies in detail, examine peephole connections, discuss stacked and bidirectional configurations, and trace the historical influence of the constant error carousel on modern architecture design. By the end, you'll understand that LSTMs avoid vanishing gradients, why they do so, and under what conditions they still require careful treatment.