LSTM Gradient Flow › Summary

The LSTM constant error carousel solves the vanishing gradient problem by creating an additive pathway for gradient flow through the cell state. Key takeaways:

• The cell state update is additive: $\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t$. Addition copies gradients unchanged, while multiplication scales them.
• The gradient through the cell state path decays only as a product of forget gate values, not as a product of weight matrix Jacobians and nonlinearity derivatives.
• The forget gate plays a dual role: it controls memory retention in the forward pass and modulates gradient magnitude in the backward pass. Learning to retain information means learning to preserve gradients.
• Forget gate bias initialization matters. Biases of 1 to 3 create favorable gradient flow conditions at the start of training and are recommended for tasks with long sequences.
• The cell state gradient path is directly analogous to the residual connection in ResNets: both use addition to create a gradient highway that bypasses the learned transformations.
• Peephole connections allow gates to observe the cell state directly, enabling more precise timing-based control, at the cost of additional parameters.
• Exploding gradients remain possible through the hidden state path. Gradient clipping (norm threshold 1.0 to 5.0) is still recommended for LSTM training on long sequences.
• The GRU, covered in the next chapter, achieves similar gradient flow improvements with a simpler two-gate architecture that reduces computational cost.
• Stacked LSTMs have strong horizontal gradient flow within each layer but may exhibit weaker vertical gradient flow across layers; residual connections between layers address this.
• Bidirectional LSTMs process sequences in both directions and provide representations that capture context from past and future, but cannot be used for online (streaming) prediction.
• LSTMs process sequences sequentially, which prevents parallelization across time and is the primary reason Transformers have displaced them for most NLP tasks at scale.
• The principle that "addition creates gradient highways" connects LSTM cell states, ResNet skip connections, and Transformer residual connections into a unified design philosophy for deep networks.