Backpropagation Through Time › Comparing BPTT Across RNN Variants

Once you understand BPTT for a vanilla RNN, it is informative to consider how the same algorithm behaves for LSTMs and GRUs. These architectures do not require a different training algorithm; they use BPTT exactly as described above. What they change is the structure of the forward computation, which in turn changes the structure of the Jacobians that appear in the backward pass.

An LSTM maintains two state vectors at each timestep: the hidden state $\mathbf{h}_t$ (sometimes called the output state) and the cell state $\mathbf{c}_t$ (the memory). The key gradient path in an LSTM runs through the cell state, not the hidden state. The cell state update equation is:

$$\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t$$

where $\mathbf{f}_t$ is the forget gate and $\mathbf{i}_t$ is the input gate. The gradient of $\mathbf{c}_t$ with respect to $\mathbf{c}_{t-1}$ is simply $\text{diag}(\mathbf{f}_t)$: a diagonal matrix containing the forget gate values. Because forget gate values are squashed by a sigmoid into $(0, 1)$, this Jacobian is always diagonal and always has entries between 0 and 1. The forget gate is learned and can be set close to 1.0 for dimensions that need long-term memory. When the forget gate is near 1, gradients flow backward through the cell state with almost no attenuation.

This is a fundamentally different gradient regime from the vanilla RNN, where the recurrent Jacobian $\text{diag}(1 - \mathbf{h}_j^2) \cdot \mathbf{W}_{hh}^T$ involves both the tanh derivative (always in $(0, 1)$) and a full matrix $\mathbf{W}_{hh}^T$ that can have large or small spectral properties depending on training. The LSTM essentially replaces that matrix product with a learned diagonal gate, giving the model explicit control over gradient flow.

A Gated Recurrent Unit (GRU) achieves similar benefits with a simpler structure, maintaining only one state vector. The update rule involves a reset gate $\mathbf{r}_t$ and an update gate $\mathbf{z}_t$:

$$\mathbf{h}_t = (1 - \mathbf{z}_t) \odot \mathbf{h}_{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t$$

The gradient of $\mathbf{h}_t$ with respect to $\mathbf{h}_{t-1}$ has a component proportional to $\text{diag}(1 - \mathbf{z}_t)$. When the update gate $\mathbf{z}_t$ is close to 0, this component is close to 1, passing gradients backward with minimal attenuation. The GRU achieves a similar constant-error-carousel effect to the LSTM through the interpolation structure of the update gate.

Both LSTMs and GRUs still use BPTT. The benefit they provide is architectural: by creating gradient paths that do not inevitably pass through full matrix products and saturated nonlinearities, they allow BPTT to compute meaningful gradients even at large distances from the output. The training algorithm is unchanged; the model is designed to be more amenable to training.