Vanishing Gradients › The Gradient Product Problem

To train an RNN, we need to compute how the loss at time $T$ depends on the hidden state at some earlier time $t$. This is exactly the gradient $\frac{\partial L_T}{\partial h_t}$, where $L_T$ is the loss at the final timestep and $h_t$ is the hidden state at time $t$.

Recall from the BPTT derivation that the hidden state evolves as:

$$h_t = \tanh(W_h h_{t-1} + W_x x_t + b)$$

where:

• $h_t \in \mathbb{R}^n$ is the hidden state vector at time $t$
• $W_h \in \mathbb{R}^{n \times n}$ is the recurrent weight matrix
• $W_x \in \mathbb{R}^{n \times d}$ maps input $x_t$ to the hidden layer
• $b \in \mathbb{R}^n$ is the bias

Applying the chain rule, the gradient of $h_T$ with respect to $h_t$ involves all intermediate steps:

$$\frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} \frac{\partial h_k}{\partial h_{k-1}}$$

where each factor in the product is the Jacobian matrix:

$$\frac{\partial h_k}{\partial h_{k-1}} = \text{diag}(\tanh'(z_k)) \cdot W_h$$

where:

• $z_k = W_h h_{k-1} + W_x x_k + b$ is the pre-activation at step $k$
• $\tanh'(z_k)$ is the element-wise derivative of tanh evaluated at $z_k$
• $\text{diag}(\tanh'(z_k))$ is the diagonal Jacobian of the tanh nonlinearity

So the full gradient becomes:

$$\frac{\partial h_T}{\partial h_t} = \prod_{k=t+1}^{T} \text{diag}(\tanh'(z_k)) \cdot W_h$$

This is a product of $T - t$ matrices. When $T - t$ is large, this product determines whether learning succeeds or fails.

To understand why this product is dangerous, think about what it represents physically. Each factor $\text{diag}(\tanh'(z_k)) \cdot W_h$ is a local linear approximation to how perturbations to $h_{k-1}$ propagate to $h_k$. Multiplying these local approximations together gives the global sensitivity: how much does a small change to $h_t$ affect $h_T$? The answer, for long sequences, is almost always: very little. The product of many factors smaller than one collapses to zero.

It is worth pausing to understand why the chain rule produces this particular structure and why there is no way around it within the RNN formulation. The chain rule is not a choice or an approximation: it is the exact rule for computing how composed functions depend on their inputs. The RNN is a composed function. $h_T$ depends on $h_{T-1}$, which depends on $h_{T-2}$, all the way back to $h_t$. The gradient of this chain of compositions is exactly the product of the local derivatives.

This means the vanishing gradient problem is not a bug in the BPTT algorithm or a consequence of a bad implementation. It is built into the structure of the RNN itself. The recurrence relation that makes RNNs powerful at processing sequences is also the source of the gradient problem. You cannot have sequential composition without a multiplicative gradient chain.

The only escape is to change the recurrence relation itself, which is precisely what LSTM does. Rather than composing a function at every timestep (multiplying), the LSTM adds to a memory cell. Addition distributes gradient flow uniformly backward, as we will see in the LSTM chapter.