Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Recurrent Memory › Effective Context Length

Michael BrenndoerferUpdated July 6, 20257 min read

Language AI HandbookAll chapters

Part 18 of 67Long Context

Chapter 132Recurrent Memory › Effective Context Length

Reading section 4 of 14 · Effective Context Length · 7 minSection 4 of 14 · 7 min

Section 4 of 14 · 7 min read

Recurrent Memory › Effective Context LengthOne section · Sections in this chapter4 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The recurrence mechanism creates a dependency chain across segments. Information from segment 1 flows to segment 2 through the cached hidden states. Segment 2's hidden states then carry forward information to segment 3. This chain means the effective context length grows beyond a single segment, bounded by how far information can propagate through the hidden states.

For an $N$ -layer transformer with segment length $L$ , the maximum effective context length is $O(N \times L)$ . To understand why, consider how information propagates layer by layer:

Layer 1 of segment $\tau$ receives cached states from layer 0 of segment $\tau - 1$ , giving it access to 1 previous segment
Layer 2 of segment $\tau$ receives cached states from layer 1 of segment $\tau - 1$ , which already incorporated information from segment $\tau - 2$ at its own layer 1
Layer $n$ can potentially access information from $n$ segments back through this chain of cached representations

The depth of the network multiplies the effective context window: the same depth that provides representational power also provides temporal reach. A deeper network can represent more complex patterns and carry information farther back.

Think of this layer-by-layer propagation as a game of telephone played upward through the layers. At layer 1, each segment's representation contains information from at most the immediately preceding segment. At layer 2, each representation has been informed by a layer-1 representation that already incorporated one segment's worth of history. By the time information reaches the top layer, it has been aggregated across $N$ segments, even though no single attention computation ever looked more than one segment back.

The key insight here is that depth and context are not independent in Transformer-XL. Every extra layer you add to the network also extends the effective context window by one segment length. This means increasing model depth is simultaneously an investment in representational capacity and in temporal reach. Conversely, it also means that shallow models will have limited effective context even when using segment-level recurrence.

In[10]:

Code

def compute_effective_context(num_layers: int, segment_length: int) -> dict:
    """
    Compute the effective context length for Transformer-XL.

    The effective context grows with depth because information
    propagates one segment further back at each layer.
    """
    # At layer n, information can potentially come from n segments back
    # This is because cached states at layer n contain information
    # that was already aggregated from previous segments at layer n-1

    # Direct attention span (what a single layer can see)
    direct_span = 2 * segment_length  # current + one cached segment

    # Maximum theoretical span considering all layers
    max_theoretical_span = num_layers * segment_length + segment_length

    # Information decay means effective span is somewhat less
    # Upper layers have access to more distant information but it's diluted

    return {
        "num_layers": num_layers,
        "segment_length": segment_length,
        "direct_attention_span": direct_span,
        "max_theoretical_span": max_theoretical_span,
        "context_multiplier": max_theoretical_span / segment_length,
    }


# Compare different configurations
configs = [
    (6, 128),  # Small model
    (12, 256),  # Medium model
    (24, 512),  # Large model
]

results = [compute_effective_context(n, l) for n, l in configs]

Out[11]:

Console

Effective Context Length in Transformer-XL

Configuration         Direct Span Max Theoretical   Multiplier
--------------------------------------------------------------
6L × 128L                     256             896          7.0x
12L × 256L                    512           3,328         13.0x
24L × 512L                  1,024          12,800         25.0x

A 24-layer model with 512-token segments can theoretically access information from over 13,000 tokens ago, even though it only directly attends to 1,024 tokens per layer. The depth of the network amplifies the effective context. This explains why Transformer-XL's original paper used configurations with many layers: the authors were adding expressiveness and extending reach at the same time.

It is worth being clear about what "theoretical" means here. The numbers above represent the maximum possible reach under the assumption that information propagates perfectly and without loss through each cached representation. In practice, information degrades as it passes through the hidden state bottleneck. We will examine this degradation in the limitations section. The theoretical maximum sets an upper bound; the practical reach depends on the nature of the information and how much the network has learned to preserve it during training.

Out[12]:

Visualization

Line plot showing linear growth of effective context with layer depth, compared to constant direct attention span. — Effective context growth with layer depth in Transformer-XL. The maximum theoretical context (solid line) grows linearly with the number of layers, while direct attention span (dashed line) remains constant at twice the segment length. For a 24-layer model with 512-token segments, the effective context exceeds 12,000 tokens. This shows how network depth multiplies context reach.

Out[13]:

Visualization

Diagram showing expanding context reach at higher transformer layers, with layer 1 seeing 1 previous segment and layer 6 seeing 6 previous segments. — Information reach by layer in Transformer-XL. Each colored row represents a transformer layer, and the width of the colored band shows how many prior segments that layer can access through the recurrence chain. Layer 1 can only see one segment back through direct cached attention. Layer 6 can access information from up to six segments back, because its cached input already contains aggregated information from the prior five segments.

The visualization shows how context reach expands with network depth. Layer 1 can only see the immediately preceding segment through cached states. Layer 6, however, has access to information from 6 segments back because the cached states at layer 5 already incorporated information that propagated through the previous segment's entire network.