Backpropagation Through Time › Limitations and Practical Considerations

BPTT is the correct training algorithm for RNNs, but it comes with practical constraints that shaped how RNN-based systems were designed for many years.

The most significant limitation is the cost of backpropagating through long sequences. Every additional timestep adds a matrix multiplication to the gradient path, and each multiplication tends to shrink the gradient (if the spectral radius of $\mathbf{W}_{hh}$ is less than 1) or grow it (if greater than 1). This sensitivity to the eigenvalue spectrum of the recurrent weight matrix is mathematically fundamental. BPTT faithfully computes the true gradient; the problem is that the true gradient is often very small or very large for long sequences.

Truncated BPTT is the standard practical solution, but it introduces its own issues. The truncation window acts as a hard limit on the dependencies the gradient can capture. A model trained with chunk size 35 simply cannot learn from gradient signal originating more than 35 steps in the past, no matter how informative that signal would be. This is not a memory issue; it is a deliberate algorithmic choice that trades gradient completeness for computational tractability. For some tasks, such as parsing long documents or understanding multi-paragraph narratives, this limitation is severe.

Memory also presents a real barrier for full BPTT on modern hardware. During the backward pass, PyTorch needs to retain all intermediate activations from the forward pass to compute gradients. For a sequence of length 1000 with a hidden state of dimension 512 and batch size 32, this amounts to hundreds of megabytes for a single sequence, before accounting for optimizer state. Gradient checkpointing offers a middle ground: instead of storing all activations, you recompute them during the backward pass, trading compute for memory. PyTorch supports this via torch.utils.checkpoint.checkpoint_sequential.

The sequential nature of RNN computation is arguably the deeper limitation. Because $\mathbf{h}_t$ depends on $\mathbf{h}_{t-1}$, the forward pass cannot be parallelized across timesteps. For a sequence of length 1000, you must execute 1000 sequential matrix operations, regardless of available hardware parallelism. Modern GPUs can process the matrix operations for a batch of sequences in parallel, but they cannot process different timesteps of the same sequence in parallel. This makes RNNs fundamentally slow to train compared to transformers, which can process all positions in a sequence simultaneously through attention.

Despite these limitations, BPTT remains the conceptual foundation for training all sequence models. The gated architectures (LSTMs, GRUs) covered in the next chapters do not replace BPTT; they improve the gradient flow properties of the network so that BPTT works better. Even modern transformer architectures, which sidestep recurrence entirely through attention, are trained using the same chain-rule-based backpropagation, just through a different computational graph. Understanding BPTT deeply is the foundation for understanding why LSTMs work, why transformers are preferred for long sequences, and what any future sequence model needs to address.

One subtle practical issue deserves mention: the handling of the initial hidden state. Typical practice initializes $\mathbf{h}_0 = \mathbf{0}$. In truncated BPTT, the hidden state at the start of each chunk is the detached final state from the previous chunk. This detachment means the model cannot update weights to make better use of the carried-forward state based on losses from future chunks. In practice this is acceptable, but it does mean the first few steps of each chunk may be slightly miscalibrated as the hidden state re-accumulates context.

An alternative approach sometimes used in practice is to learn the initial hidden state as a parameter. By treating $\mathbf{h}_0$ as a trainable vector, the model can learn a good starting state that minimizes the warmup problem at the beginning of each sequence or chunk. This is a small but non-negligible improvement for tasks where the first few tokens of a sequence are important.