Vanishing Gradients › Why Long-Range Dependencies Require Stable Gradient Flow

To understand intuitively why long-range dependencies break down, consider training a network to predict that "the cat, which sat on the mat for a very long time, was hungry." The word "hungry" must agree with "cat," which appeared 10 tokens earlier. The gradient of the prediction error on "hungry" must travel backward through "was," "time," "very," "a," "for," "mat," "the," "on," "sat," "which" before reaching "cat." Each of those 10 backward steps multiplies the gradient by a factor less than 1, shrinking it toward zero.

The weight update that would strengthen the "cat to hungry" connection is proportional to this vanished gradient. With a gradient of $10^{-8}$, the weight update is effectively zero regardless of the learning rate. The network cannot learn this dependency, no matter how many epochs you train for.

This is not a problem of insufficient training time or learning rate tuning. It is a fundamental limitation of the architecture: the gradient signal cannot reliably propagate information over long distances because multiplication by many small factors inevitably destroys the signal.

The vanishing gradient problem is a specific manifestation of the broader credit assignment problem in machine learning. Credit assignment asks: given that the network made an error at step $T$, which earlier decisions were responsible, and by how much? For a vanilla RNN, the credit assignment problem is particularly severe because the gradient is the only mechanism for tracing responsibility backward through time.

In a feedforward network, credit is assigned through the depth of the network (layers correspond to steps of transformation). In a recurrent network, credit must be assigned through the time dimension of the sequence. The longer the sequence, the harder the credit assignment problem. And unlike feedforward networks, where depth can be controlled by architecture design, sequence length is a property of the data and cannot be arbitrarily reduced without discarding information.

The vanishing gradient problem means that for distant timesteps, the credit assigned by gradient descent is essentially zero. The network assigns full credit to recent steps and no credit to distant ones, producing a systematic bias toward local patterns regardless of which patterns matter for the task. This is not a soft or gradual bias: for sequences longer than the memory horizon, the credit assignment for distant tokens rounds to exactly zero in floating-point arithmetic.