Backpropagation Through Time › Truncated BPTT

Full BPTT requires storing all hidden states and all activations for the entire sequence before running the backward pass. For a sequence of length $T$, this means $O(T)$ memory for activations and $O(T)$ computation for the backward pass. When $T$ is thousands or tens of thousands of tokens, full BPTT becomes impractical.

Truncated BPTT addresses this by dividing the sequence into fixed-length chunks and running a separate forward/backward pass through each chunk. Within each chunk, gradients flow freely. Across chunks, the hidden state is passed forward but gradients are stopped at the chunk boundary.

The sequence of length $T$ is divided into non-overlapping chunks of length $k$ (the truncation length). For chunk $c$ spanning timesteps $[ck+1, (c+1)k]$:

1. The forward pass starts with the final hidden state from the previous chunk, $\mathbf{h}_{ck}$, carried forward without gradient tracking
2. The forward pass runs through all $k$ steps in the chunk, accumulating hidden states and losses
3. The backward pass propagates gradients back through the $k$ steps in the chunk only
4. The weight gradients are accumulated across all chunks

The stopping of gradients at chunk boundaries is implemented using PyTorch's detach() operation, which creates a new tensor with the same values but no gradient history.

The forward-pass hidden state still carries information from earlier in the sequence. When chunk $c$ begins, the initial hidden state $\mathbf{h}_{ck}$ encodes everything the network saw in chunks 0 through $c-1$. That information is used in the forward pass to make predictions and compute losses. But the backward pass cannot trace gradients through $\mathbf{h}_{ck}$ into earlier chunks, because detach() has severed the computational graph. The result is an asymmetry: the model can use long-range information to make predictions, but cannot learn from gradient signal that would require looking beyond the current chunk boundary.

Truncated BPTT makes a practical trade-off: it gives up the ability to learn dependencies that span more than $k$ timesteps in exchange for bounded memory and predictable computation per update step. In practice, for most RNN training tasks, a truncation length between 20 and 200 captures the dependencies that matter. The exact value is a hyperparameter that depends on the sequence structure, and it interacts with the model's capacity to represent longer-range information in its hidden state.

The following visualization shows how gradient coverage changes with the truncation window relative to the full sequence length. Each bar represents the fraction of the sequence covered by one backward pass:

With $k = 20$, the 200-step sequence requires 10 separate backward passes. With $k = 100$, only 2 passes are needed. The trade-off is that larger chunks require more memory to store activations during the forward pass of each chunk.

The truncation length $k$ controls the trade-off between several competing factors:

• Memory cost: full BPTT requires $O(T)$ memory; truncated BPTT requires $O(k)$ memory per chunk
• Computational cost: each backward pass covers $k$ steps; you run $T/k$ backward passes per sequence
• Gradient quality: longer chunks capture more long-range dependencies; shorter chunks lose information about distant correlations

A common choice is $k = 35$ for language modeling, matching the LSTM training paper by Zaremba et al. (2014). For tasks with explicit long-range dependencies, larger $k$ values or gated architectures such as LSTMs help.

Choosing the truncation length is not purely a memory decision; it should reflect the structure of the data. If the sequence contains strong dependencies that span 100 tokens, a truncation window of 20 will prevent the model from ever learning them through gradient updates. In practice, you can estimate the relevant dependency range by examining what information would be needed to make accurate predictions. For next-word prediction in typical English text, dependencies rarely need to span more than a few dozen words, which explains why modest truncation windows work well for language modeling.

One underappreciated consideration is the interaction between the truncation length and the learning dynamics. Very short truncation windows produce gradients that are effectively local to a small context window. The model learns to make predictions using only nearby context, which is fast but limits what it can ultimately learn. Very long truncation windows give the model more gradient signal but also give it vanishing gradient problems of its own, since the gradient still has to propagate through $k$ matrix multiplications to reach the beginning of the chunk.

The memory comparison between full BPTT and truncated BPTT is significant. For a sequence of length $T$ with hidden size $d$ and batch size $B$:

• Full BPTT activations: $O(T \cdot d \cdot B)$ bytes (all hidden states stored)
• Truncated BPTT activations: $O(k \cdot d \cdot B)$ bytes (only current chunk's states stored)

For long sequences (e.g., $T = 1000$, $k = 35$), truncated BPTT reduces memory by roughly $T / k = 28.6\times$. The computational cost per step is similar, but the memory savings allow much larger batch sizes or hidden state sizes.

Beyond hidden states, the backward pass also requires storing intermediate activations, including the pre-activation values (the inputs to the tanh function) at each step, since the tanh derivative must be computed from these values. The total activation memory scales with the depth of the backward pass, making truncation doubly beneficial: it reduces both the hidden state storage and the intermediate activation storage.