Vanishing Gradients › Vanishing Gradients

In the previous chapter, we derived Backpropagation Through Time (BPTT) and saw how gradients flow backward through an unrolled RNN. We noted that the same weight matrix $W_h$ multiplies the gradient at every timestep. That single observation, almost a footnote at the end of the BPTT derivation, turns out to be the central obstacle of training recurrent networks on long sequences. This chapter explores what happens to those gradients as they travel backward across dozens or hundreds of timesteps, why they almost always collapse to zero, and why this collapse makes RNNs unreliable for learning long-range dependencies. Understanding this failure mode thoroughly is the essential motivation for the LSTM architecture we will study next.

Before examining the mathematics, consider the task that any sequence model must accomplish. Language is deeply hierarchical. A sentence can contain a relative clause that interrupts the main clause. A paragraph can open with a claim that its final sentence resolves. A document can introduce a character whose actions hundreds of words later depend on a motivation established at the beginning. Any model that hopes to understand language must be able to connect information across these distances. For an RNN, this means the gradient signal that tells the network "your prediction at time $T$ was wrong because of the information at time $t$" must survive a journey through $T - t$ steps. As we will see, for realistic sequence lengths, that journey is almost impossible.