Vanishing Gradients › The Vanishing Gradient Problem in Deep Feedforward Networks
Language AI HandbookAll chapters Vanishing Gradients › The Vanishing Gradient Problem in Deep Feedforward NetworksOne section · Sections in this chapter14 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before leaving this topic, it is worth connecting the RNN vanishing gradient problem to an analogous problem in deep feedforward networks. In a feedforward network with many layers, the gradient must propagate backward through each layer in sequence. If each layer applies a saturating nonlinearity, the gradient is repeatedly multiplied by derivative factors less than 1, producing the same exponential decay we see in RNNs.
This is why very deep feedforward networks were difficult to train before the introduction of batch normalization (2015) and residual connections (2015). A 20-layer network with sigmoid activations would experience the same gradient collapse over 20 backward passes through layers that an RNN experiences over 20 backward steps through time. The problems are mathematically identical.
The solutions are also analogous. Residual connections in feedforward networks create gradient shortcuts that bypass the multiplicative chain, just as the LSTM cell state creates a gradient highway through time. Batch normalization keeps activations away from saturation zones, reducing the damping factor at each step, just as proper initialization can reduce damping in RNNs.
This connection is not a coincidence. Both problems arise from the same root cause: repeated multiplication of Jacobian matrices through a chain of composed functions, where each Jacobian has spectral properties that lead to exponential decay. The solutions in both cases involve creating additive pathways that allow gradients to bypass the multiplicative chain.
The correspondence between depth in feedforward networks and time in recurrent networks is precise. A feedforward network with layers and a recurrent network with sequence length both require gradient signals to traverse or multiplicative factors, respectively. This analogy guided the development of residual connections: researchers who had solved the depth problem in feedforward networks recognized the same structure in the time dimension of RNNs and applied similar architectural ideas.
The connection between RNN vanishing gradients and deep network vanishing gradients also explains why ReLU activations helped feedforward networks train more easily. Unlike sigmoid and tanh, the ReLU function has a derivative of exactly 1 for positive inputs and 0 for negative inputs. For active neurons (positive pre-activation), ReLU contributes no damping to the gradient. This eliminates the saturation problem in feedforward networks. However, ReLU does not directly help with vanishing gradients in the time dimension of RNNs, because the issue there is the recurrent weight matrix, not the nonlinearity alone.