Backpropagation Through Time › Practical Pitfalls and Implementation Advice

Training RNNs with BPTT involves several common pitfalls that can cause silently incorrect or very slow training.

When iterating over a dataset multiple times, a common mistake is to carry the hidden state from the end of one epoch into the beginning of the next without detaching it. If the hidden state from epoch $n$'s last batch is passed directly into epoch $n+1$'s first batch without calling .detach(), the computational graph grows across batches. By the time you call .backward() on the epoch $n+1$ loss, PyTorch will try to backpropagate through the hidden state all the way back to wherever the graph started in epoch $n$. This can lead to memory errors (the graph is unexpectedly large) or incorrect gradients (the graph spans multiple batches of unrelated data).

The safe practice is to always initialize the hidden state fresh at the start of each independent sequence in your dataset, or to call .detach() on any hidden state that is being carried across batch boundaries.

When processing a dataset of multiple independent sequences (documents, sentences, examples), you must reset the hidden state between sequences. Each sequence in the dataset is independent; the hidden state from processing document A should not be used as the initial state for processing document B. Failing to reset leads to a subtle bug where the model's predictions at the start of each sequence are influenced by the content of the previous sequence.

This bug is particularly insidious because the training loss may still decrease: the model might learn to use the leaked context from the previous sequence for certain tasks. But the model's behavior at inference time will be wrong if sequences are processed in a different order or in isolation, because the expected hidden state contamination will not be present.

While vanishing gradients are the chronic problem, exploding gradients are the acute one. A single exploding gradient event can corrupt a model's weights catastrophically, requiring restarting training from a checkpoint. Gradient clipping, covered in detail in the Gradient Clipping chapter, is the standard remedy: before applying the optimizer step, you rescale the global gradient norm to a maximum threshold value if it exceeds that threshold.

In PyTorch, gradient clipping is one line:

This should be called after loss.backward() and before optimizer.step(). The typical threshold for RNN training is 1.0 to 5.0, with 1.0 being a conservative and widely-used default.

The learning rate for RNN training typically needs to be smaller than for feedforward networks, because the gradient scale varies significantly with sequence length. When you increase the sequence length (or chunk size), the gradient norms change, and a learning rate that worked well for short sequences may be too large or too small for longer ones. Using adaptive optimizers like Adam helps, because Adam normalizes gradients per-parameter, but even with Adam, careful learning rate tuning is often necessary when changing the sequence length.

A related consideration is learning rate warmup: starting with a very small learning rate and gradually increasing it to the target value over the first few thousand steps. Warmup helps because early in training, the model's hidden states are poorly initialized and the gradient surface is unstable. Large early gradient steps can send weights to poor regions of parameter space from which recovery is slow.