Vanishing Gradients › Summary

The vanishing gradient problem in RNNs arises because BPTT requires computing a product of $T - t$ Jacobian matrices, one per timestep between the current position and the position of the loss. Each Jacobian is bounded in spectral radius by the product of the weight matrix and the activation derivative, which is less than 1 for saturating nonlinearities like tanh and sigmoid. The product of many such factors shrinks exponentially with sequence length.

The key takeaways from this chapter are:

• The gradient product bound: $\|\prod_{k} D_k W_h\| \leq (\gamma \cdot \|W_h\|)^{T-t}$ decays exponentially when $\gamma \cdot \|W_h\| < 1$, which holds for all practical initializations with saturating nonlinearities.
• Sigmoid is worse than tanh: sigmoid's maximum derivative of 0.25 accelerates gradient decay compared to tanh's maximum of 1. In a chain of 10 sigmoid steps, the gradient decays by at least a factor of $10^6$.
• Exploding vs vanishing: exploding gradients are detectable and fixable with gradient clipping; vanishing gradients are silent and architecturally catastrophic. The asymmetry makes vanishing the harder problem.
• Effective gradient distance: vanilla RNNs can only reliably learn dependencies spanning at most a few dozen timesteps, far too short for many practical NLP tasks.
• The credit assignment failure: vanishing gradients cause the network to assign zero credit to distant timesteps, systematically biasing it toward local patterns even when global dependencies are what matter.
• Partial solutions fall short: careful initialization, truncated BPTT, and echo state networks all address symptoms without solving the underlying architectural problem.
• LSTM motivation: the vanishing gradient analysis defines exactly what the LSTM must provide, which is an additive update path for a separate memory cell that allows gradients to bypass the multiplicative decay. Addition distributes gradient uniformly, unlike multiplication which amplifies or shrinks it.

The next chapter introduces the LSTM architecture and shows how the cell state and gate mechanisms provide the solution to everything identified here.