Vanishing Gradients › Eigenvalue Analysis: Why Products of Matrices Shrink or Explode

The behavior of a matrix product depends critically on the spectral properties of each factor. Let's analyze what happens when you repeatedly multiply similar matrices together.

Consider a simplified version where $W_h$ is a scalar weight $w$ (or, more precisely, where we track the behavior along the dominant eigenvalue direction). Each factor in the gradient product contributes a scaling of approximately $|\lambda_1(D_k \cdot W_h)|$, where $\lambda_1$ denotes the largest singular value and $D_k = \text{diag}(\tanh'(z_k))$ is the diagonal scaling matrix from the nonlinearity.

Since $|\tanh'(z)| \leq 1$ for all $z$ (the tanh derivative is at most 1, achieved at $z = 0$), and in practice the hidden states often saturate the tanh function, $D_k$ contributes factors strictly less than 1.

The gradient product norm satisfies:

$$\left\| \prod_{k=t+1}^{T} D_k W_h \right\| \leq \prod_{k=t+1}^{T} \|D_k W_h\| \leq (\gamma \cdot \|W_h\|)^{T-t}$$

where $\gamma = \max_k \|D_k\| < 1$ is the maximum gain from the nonlinearity across all timesteps.

This inequality has two regimes:

• If $\gamma \cdot \|W_h\| < 1$, the product shrinks exponentially in $T - t$. This is the vanishing gradient regime.
• If $\gamma \cdot \|W_h\| > 1$, the product grows exponentially. This is the exploding gradient regime.

The boundary between the two is the unit circle in eigenvalue space. Practical networks almost always fall into the vanishing regime because the tanh nonlinearity enforces $\gamma < 1$, and weight matrices are initialized to have spectral radius near or below 1 for stability.

To understand the behavior of the gradient product more deeply, it helps to think about the eigenvalue decomposition of $W_h$. Suppose $W_h$ is diagonalizable with eigenvalues $\lambda_1, \lambda_2, \ldots, \lambda_n$, sorted by absolute value so that $|\lambda_1| \geq |\lambda_2| \geq \cdots \geq |\lambda_n|$.

When you multiply the matrix by itself many times, the eigenvectors associated with large eigenvalues dominate the product. For a matrix power $W_h^k$, the contribution from the $i$-th eigendirection scales as $\lambda_i^k$. For directions where $|\lambda_i| < 1$, the product collapses exponentially. For directions where $|\lambda_i| > 1$, the product grows exponentially.

In the gradient product, the situation is analogous but more complex because each factor also includes the diagonal scaling from the nonlinearity. Nevertheless, the dominant behavior is determined by the spectrum of $W_h$. The gradient signal propagates reliably only through eigendirections with eigenvalue magnitude near or above 1. All other directions lose information exponentially fast.

This has a subtle but important consequence for what RNNs can remember. The information that survives long backward passes is not arbitrary: it is precisely the information aligned with the dominant eigenvectors of $W_h$. The recurrent weight matrix effectively selects which aspects of the hidden state are memorable and which are forgettable based purely on its spectral structure, not based on what the task requires. This spectral selectivity is one reason vanilla RNNs are difficult to reason about and hard to train for specific long-range tasks.

While the analysis above uses tanh, the original LSTM paper and many early RNNs used sigmoid activations in some layers. Sigmoid is even more problematic for gradient flow.

The sigmoid function $\sigma(z) = \frac{1}{1 + e^{-z}}$ has a maximum derivative of $0.25$, occurring at $z = 0$:

$$\sigma'(z) = \sigma(z)(1 - \sigma(z)) \leq 0.25$$

where:

• $\sigma(z)$ is the sigmoid output, always in $(0, 1)$
• $1 - \sigma(z)$ is the complement, also in $(0, 1)$
• Their product is maximized when $\sigma(z) = 0.5$, giving $0.5 \times 0.5 = 0.25$

This means each sigmoid gate contributes at most a factor of $0.25$ to the gradient. A chain of 10 sigmoid layers multiplies the gradient by at most $0.25^{10} \approx 10^{-6}$. Over 100 timesteps, the gradient magnitude is bounded by $0.25^{100}$, which is effectively zero in floating-point arithmetic.

Tanh has a maximum derivative of 1 (at $z = 0$), so it is somewhat better, but the maximum is only achieved exactly at zero. In practice, hidden states drift away from zero as the network processes sequences, and the effective gain drops well below 1.

To appreciate the interplay between saturation and vanishing gradients, consider what happens during the forward pass of a long sequence. At each step, the network computes a new hidden state and passes it through tanh. If the inputs are informative and the network is responding strongly, the pre-activation values $z_k$ will have large magnitudes, and tanh will saturate them to values near $\pm 1$. This saturation is desirable from a representational standpoint: it allows the network to make confident decisions about whether information is present. But the same saturation that makes the forward representation less brittle makes the backward gradient fragile. A hidden state near $\pm 1$ has a derivative near zero, contributing nearly zero gain to the gradient product at that step. The very steps where the network is most confident about its representation are the steps that contribute most to gradient decay.

This creates a fundamental tension. Informative hidden states tend to saturate, which kills gradient flow. Unsaturated hidden states near zero have maximum gradient flow, but they also represent uncertain, low-magnitude activations. The network cannot simultaneously have rich, confident representations and efficient gradient flow when using saturating nonlinearities.