Vanishing Gradients › Effective Gradient Distance

The vanishing gradient problem means that the signal from distant timesteps is effectively zero. We can formalize this with the concept of effective gradient distance: the maximum number of timesteps over which a gradient signal remains large enough to update the weights.

If each timestep contributes a multiplicative factor of $\rho < 1$ to the gradient magnitude, then after $k$ steps, the gradient has magnitude proportional to $\rho^k$. The effective gradient distance is roughly the $k$ at which $\rho^k$ falls below the numerical noise floor, often taken as $10^{-4}$ relative to the initial magnitude:

$$k_{\text{eff}} = \frac{\log(10^{-4})}{\log(\rho)} = \frac{-4 \log(10)}{\log(\rho)}$$

For $\rho = 0.9$, this gives $k_{\text{eff}} \approx 87$ steps, which seems large. But typical RNN training uses $\rho$ values closer to 0.7 to 0.8, giving $k_{\text{eff}}$ between 25 and 50 steps. Sequences of hundreds of tokens far exceed this budget, and dependencies across paragraphs or documents are completely invisible to the gradient signal.

The effective gradient distance defines what we can call the memory horizon of the network: the temporal window within which the network can learn. Beyond this horizon, weight updates are effectively zero and the network behaves as if those earlier tokens never existed.

The memory horizon is not a sharp cutoff. Gradients decay continuously, and the rate of decay varies across different directions in the hidden state space. Some information can survive longer than average if it happens to align with the dominant eigenvectors of $W_h$. But the average behavior is exponential decay, and for most practical tasks, the memory horizon falls well short of what is needed.

Consider the implications for language modeling. Subject-verb agreement in English typically requires connecting a noun to a verb separated by a short clause. For a sentence like "The man who lives in the old house is friendly," the subject "man" and the verb "is" are separated by six tokens. A network with a memory horizon of 20 steps could in principle learn this dependency. But for a sentence like "The man who lives in the old house near the park with the beautiful fountain and the ancient oak trees that the children always play around in summer is friendly," the subject and verb are separated by more than 30 tokens. Such examples reveal how even moderate clause embeddings can push dependencies beyond the vanilla RNN's effective range.

In practice, the memory horizon for a vanilla RNN trained on natural language is often estimated at roughly 10 to 15 tokens. This is enough for local syntactic patterns, short-distance agreement, and simple phrase-level semantics. It is not enough for clause-level dependencies, paragraph-level coherence, or document-level structure.