Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Backpropagation Through Time › Memory Tradeoffs: Full vs. Truncated BPTT

Michael BrenndoerferMay 10, 20253 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 75Backpropagation Through Time › Memory Tradeoffs: Full vs. Truncated BPTT

Reading section 8 of 15 · Memory Tradeoffs: Full vs. Truncated BPTT · 3 minSection 8 of 15 · 3 min

Section 8 of 15 · 3 min read

Backpropagation Through Time › Memory Tradeoffs: Full vs. Truncated BPTTOne section · Sections in this chapter8 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's visualize the memory tradeoff concretely:

In[14]:

Code

def compute_memory_requirements(
    seq_lengths, chunk_sizes, hidden_dim, batch_size, bytes_per_float=4
):
    """
    Estimate memory requirements for full and truncated BPTT.
    Memory proportional to number of stored activations.
    """
    results = {}

    for seq_len in seq_lengths:
        # Full BPTT: store all hidden states for entire sequence
        full_mem_mb = (seq_len * hidden_dim * batch_size * bytes_per_float) / (
            1024**2
        )
        results[seq_len] = {"full": full_mem_mb}

        for chunk_size in chunk_sizes:
            # Truncated BPTT: store only current chunk's hidden states
            trunc_mem_mb = (
                chunk_size * hidden_dim * batch_size * bytes_per_float
            ) / (1024**2)
            results[seq_len][f"trunc_{chunk_size}"] = trunc_mem_mb

    return results


seq_lengths_mem = [50, 100, 250, 500, 1000]
chunk_sizes_mem = [25, 50, 100]
hidden_dim_large = 512
batch_size_large = 32

mem_data = compute_memory_requirements(
    seq_lengths_mem, chunk_sizes_mem, hidden_dim_large, batch_size_large
)

Out[15]:

Visualization

Bar chart comparing memory usage for full and truncated BPTT across different sequence lengths. — Estimated memory usage (in MB) for storing hidden state activations during full BPTT versus truncated BPTT with different chunk sizes (hidden size 512, batch size 32). Full BPTT memory grows linearly with sequence length, while truncated BPTT maintains constant memory bounded by the chunk size regardless of how long the full sequence is.

The chart makes the advantage of truncated BPTT clear. For a sequence of length 1000 with hidden dimension 512 and batch size 32, full BPTT requires roughly 64 MB of activations just for the hidden states. Truncated BPTT with chunk size 50 reduces this to about 3.2 MB per chunk, a $1000/50 = 20\times$ reduction. This is before accounting for the even larger memory needed to store the pre-activation values (inputs to the tanh function), which are also needed for computing the tanh derivatives during the backward pass.