Vanishing Gradients › Empirical Evidence of Vanishing in Practice
Language AI HandbookAll chapters Vanishing Gradients › Empirical Evidence of Vanishing in PracticeOne section · Sections in this chapter6 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Beyond the theoretical analysis, empirical experiments confirm that vanilla RNNs consistently fail to learn long-range dependencies. Several classic experiments demonstrate this.
Copying task: The network receives a sequence of symbols, then must reproduce the symbols after a delay. Vanilla RNNs succeed easily for delays of 5 to 10 steps but fail almost completely for delays of 50 to 100 steps. The required information simply cannot survive the gradient decay. This task is entirely artificial, with no linguistic complexity, yet the RNN cannot solve it for long delays. This eliminates confounding factors: it is not that the task is linguistically hard. It is that the gradient cannot travel far enough to train the relevant weights.
Counting task: The network must count occurrences of a pattern across a long sequence. The count must be maintained in the hidden state, but gradients that would reinforce count-preserving behavior vanish before reaching the early timesteps. The network either fails to maintain the count at all or maintains it correctly for short sequences and fails for long ones.
Language modeling: When trained on natural language, vanilla RNNs learn local word patterns (bigrams, trigrams, short syntactic structures) reasonably well. They fail to capture subject-verb agreement across long clauses, pronoun coreference, or topic coherence across paragraphs. The perplexity of vanilla RNNs on held-out text plateaus at values that reflect this shallow modeling: the network has learned a sophisticated local grammar but cannot connect information across the document.
Sentiment analysis across long reviews: A review that opens with extensive praise and ends with a qualification can fool a vanilla RNN. The positive signal at the beginning is washed out by the long gradient chain before it influences the final prediction. LSTMs handle such long-range sentiment signals much more reliably.
These empirical failures match the theoretical prediction: the effective receptive field of a vanilla RNN is bounded by the effective gradient distance, not the full sequence length. No amount of additional training data or hyperparameter tuning can extend the effective gradient distance. The problem is structural, not empirical.