Recurrent Memory: Extending Transformer Context

Transformers process sequences in fixed-length segments. When a document exceeds the context window, the standard approach is to truncate or split it, processing each piece independently. But this independence comes at a cost: information from earlier segments vanishes entirely. A pronoun in segment 3 cannot resolve to its antecedent in segment 1 because segment 1 no longer exists in the model's view. The transformer has, in effect, amnesia: it wakes up at the start of each segment with no memory of what came before.

This limitation becomes painfully apparent when working with real-world text. A legal brief references a clause defined pages earlier. A novel's character acts on motivations established chapters back. A scientific paper's conclusion depends on definitions in its introduction. When we split these documents into segments and process each independently, we destroy the long-range connections that give language its meaning. The model sees each fragment in isolation and cannot form the kind of coherent, document-level understanding that these texts demand.

Transformer-XL introduced a solution that seems almost obvious in retrospect: what if we kept the hidden states from the previous segment and let the current segment attend to them? This segment-level recurrence creates a form of memory that extends effective context far beyond the training sequence length. The model processes sequences one segment at a time, but each segment can "remember" what came before through cached hidden states. Think of it like reading a book with a bookmark that marks your place and carries forward a compressed summary of everything you have read. You do not reread from the beginning, but that summary shapes how you understand each new page.

Transformer-XL extends context without recomputing attention over the entire history. Storing and recomputing attention over all past tokens would cost quadratic time in the total sequence length, which is exactly what we wanted to avoid by using segments in the first place. Instead, the cached hidden states serve as a fixed-length, information-dense memory that can be attended to with only linear overhead relative to the segment size. The tradeoff is that information is compressed into a fixed-dimensional representation rather than preserved exactly, but this compression is what makes the approach computationally tractable.

This chapter explores how Transformer-XL implements recurrent memory, why it requires relative positional encodings, and what limitations remain. Understanding this approach illuminates a key tension in long-context modeling: the tradeoff between computational efficiency and true bidirectional context. We will also see that solving the memory problem immediately creates a new problem: positions. When hidden states from one segment are concatenated with hidden states from another, the model must understand where each token lies in the combined sequence. Standard absolute position encodings fail here in a subtle but important way, leading to one of Transformer-XL's most elegant contributions.

Standard transformers suffer from what the Transformer-XL paper calls "context fragmentation." When you split a long document into fixed-length segments, each segment is processed without any knowledge of its neighbors. The model sees each chunk as an independent sequence. This limitation affects predictions directly: the model's predictions for the first few tokens of each new segment are systematically worse than predictions in the middle of a segment, because those early tokens have no context to draw on. The model is perpetually starting cold.

The severity of context fragmentation depends on both the segment length and the nature of the text. For documents with frequent long-range dependencies, every segment boundary is a potential failure point. Consider coreference resolution, the task of linking pronouns to their referents. If "the researcher" appears in segment 1 and "she" appears in segment 2, a model with context fragmentation cannot resolve the pronoun, even if a human reader would find the reference obvious. The same problem arises with discourse connectives ("As mentioned earlier..."), ellipsis resolution, and any form of reasoning that requires holding multiple pieces of information in mind simultaneously.

Think of context fragmentation as reading a story through a narrow window that reveals only one page at a time. Each page you read, you must discard the previous one. When a character on page 15 says "You know what I mean," you cannot, because the context that would let you understand was on page 12, which has already been discarded. The model faces this exact situation at every segment boundary.

The quantitative impact of context fragmentation is stark. Consider a 4096-token document processed with a 512-token window. Only a small fraction of all possible token-to-token dependencies are visible within any single segment. The vast majority of potentially informative connections between tokens are simply cut off.

Consider processing a 4096-token document with a 512-token context window. The naive approach splits this into 8 segments, processing each independently. Token 513 (the first token of segment 2) cannot attend to token 512 (the last token of segment 1). They're as disconnected as tokens from completely different documents.

The attention pattern reveals the fundamental limitation: each segment is an island. No matter how important a reference in segment 1 might be for understanding segment 4, that information cannot flow through the attention mechanism. The red boundaries are walls, not suggestions.

Now that we understand the problem, let's explore how Transformer-XL solves it. The solution is remarkably simple: instead of discarding the previous segment entirely, cache its hidden states and make them available during attention computation for the current segment.

Think of it like this: when you read a new paragraph, you don't forget the previous one. You carry forward a mental summary of what came before. That summary isn't the original words themselves, but your processed understanding of them. Transformer-XL does exactly this, but with hidden states instead of mental summaries. The cached states are not raw token embeddings. They are processed representations that already encode relationships between tokens, contextual information from within that segment, and whatever higher-level abstractions the network has learned to extract.

The key insight is that we do not need to preserve the original tokens from previous segments. We need to preserve the information they contained in a form that is useful for processing the current segment. Hidden states serve this purpose because they encode meaning rather than surface form alone. A token's hidden state at layer 3 is less like "the word 'cat'" and more like "a specific cat being discussed in a context involving a mat and some sitting." That richer representation is what gets cached and carried forward.

This approach also fits naturally with the autoregressive nature of language modeling. When generating text one token at a time, we process segments sequentially anyway. Caching the hidden states of the just-processed segment and feeding them to the next segment adds minimal overhead to the already-sequential processing. There is no need to maintain an exponentially growing memory buffer; we only ever need the most recent segment's hidden states.

The recurrence operates at each layer independently. When processing layer $n$ of segment $\tau$, the model:

1. Retrieves the cached hidden states $\mathbf{h}_{\tau-1}^{n-1}$ from processing the previous segment at layer $n-1$
2. Concatenates these cached states with the current segment's hidden states
3. Computes attention where queries come only from the current segment, but keys and values span both the cached and current states
4. Caches the current segment's output hidden states for use when processing the next segment

This creates an asymmetric attention pattern: current tokens can "look back" at cached tokens, but we never recompute outputs for the cached tokens. They're frozen representations from the previous forward pass. This asymmetry is not a limitation but a design choice. Recomputing the outputs of cached tokens would require backpropagating gradients through an unbounded history, which is exactly the computational cost we are trying to avoid. By treating cached states as read-only context, Transformer-XL achieves information flow across segments at constant computational cost per segment.

Another subtle but important point is that the cached states are from layer $n-1$, not layer $n$. When processing layer $n$, we concatenate the previous segment's layer $n-1$ output with the current segment's layer $n-1$ output and compute attention. This is not an implementation detail but a consequence of how information flows through a multi-layer network. At layer $n$, the current segment's representation already incorporates information from lower layers. The cached representation we need to extend must be at the same "processing depth" to combine coherently with the current representation.

Let's formalize this mechanism. Suppose we're processing segment $\tau$, which contains $L$ tokens. The previous segment's hidden states, also of length $L$ (or some memory length $M$), have been cached. At layer $n$, we want to compute attention over an extended context that includes both segments.

The extended context for attention at layer $n$ concatenates the stop-gradient of the previous segment's hidden states with the current segment's hidden states. Concretely:

where:

• $\tilde{\mathbf{h}}_\tau^{n-1}$: the extended hidden states combining previous and current segments, used as keys and values
• $\mathbf{h}_{\tau-1}^{n-1}$: cached hidden states from the previous segment at layer $n-1$
• $\mathbf{h}_\tau^{n-1}$: current segment's hidden states at layer $n-1$
• $\text{StopGrad}(\cdot)$: stops gradient flow to prevent backpropagating through the cached states
• $\circ$: concatenation along the sequence dimension, resulting in a tensor of length $M + L$

Why does StopGrad matter? If we allowed gradients to flow back through the cached states, we would need to store the computation graph for the previous segment's forward pass, then the segment before that, and so on, going back all the way to the beginning of the document. This would make the memory cost proportional to the total document length, defeating the purpose of segmentation. By stopping gradients at the cached states, each segment's backward pass touches only the current segment's computation, keeping memory usage constant regardless of document length.

The attention computation then becomes:

where:

• $\mathbf{q}_\tau^n \in \mathbb{R}^{L \times d}$: queries from the current segment only
• $\mathbf{k}_\tau^n, \mathbf{v}_\tau^n \in \mathbb{R}^{(L + M) \times d}$: keys and values from the extended context (current + cached)
• $W_q^n, W_k^n, W_v^n$: learnable projection matrices for layer $n$
• $M$: the length of the cached memory (typically equal to segment length $L$)

The critical detail is that queries come only from the current segment while keys and values include the cached previous segment. This asymmetry is intentional: we want current tokens to attend to past context, but we don't want to regenerate outputs for past tokens. The cached states are read-only: they provide context but don't receive updates.

Why does this formula make sense? Notice that the query dimension remains $L$ (only current tokens ask questions) while the key and value dimensions are $L + M$ (both cached and current tokens can be referenced). The attention matrix has shape $(L, L+M)$, which means each current token computes a score against every cached token and every preceding current token. The resulting weighted sum draws from both the cached and current portions of the extended context, exactly what we want.

Let's translate this into code. The core operation is straightforward: concatenate cached and current hidden states, project to queries/keys/values, and compute attention with appropriate masking.

The attention pattern shows the distinctive Transformer-XL signature: a triangular pattern in the current segment (causal self-attention) combined with a rectangular region on the left (attention to cached memory). Every token in the current segment can see the entire cached memory, and each token can additionally see all the preceding tokens in the current segment.

The recurrence mechanism creates a dependency chain across segments. Information from segment 1 flows to segment 2 through the cached hidden states. Segment 2's hidden states then carry forward information to segment 3. This chain means the effective context length grows beyond a single segment, bounded by how far information can propagate through the hidden states.

For an $N$-layer transformer with segment length $L$, the maximum effective context length is $O(N \times L)$. To understand why, consider how information propagates layer by layer:

• Layer 1 of segment $\tau$ receives cached states from layer 0 of segment $\tau - 1$, giving it access to 1 previous segment
• Layer 2 of segment $\tau$ receives cached states from layer 1 of segment $\tau - 1$, which already incorporated information from segment $\tau - 2$ at its own layer 1
• Layer $n$ can potentially access information from $n$ segments back through this chain of cached representations

The depth of the network multiplies the effective context window: the same depth that provides representational power also provides temporal reach. A deeper network can represent more complex patterns and carry information farther back.

Think of this layer-by-layer propagation as a game of telephone played upward through the layers. At layer 1, each segment's representation contains information from at most the immediately preceding segment. At layer 2, each representation has been informed by a layer-1 representation that already incorporated one segment's worth of history. By the time information reaches the top layer, it has been aggregated across $N$ segments, even though no single attention computation ever looked more than one segment back.

The key insight here is that depth and context are not independent in Transformer-XL. Every extra layer you add to the network also extends the effective context window by one segment length. This means increasing model depth is simultaneously an investment in representational capacity and in temporal reach. Conversely, it also means that shallow models will have limited effective context even when using segment-level recurrence.

The visualization shows how context reach expands with network depth. Layer 1 can only see the immediately preceding segment through cached states. Layer 6, however, has access to information from 6 segments back because the cached states at layer 5 already incorporated information that propagated through the previous segment's entire network.

Standard absolute position encodings break under segment-level recurrence. If we use learned or sinusoidal position embeddings based on absolute positions within each segment, we encounter a fundamental inconsistency.

Consider a token at position 5 in segment $\tau$. In the previous segment $\tau - 1$, there was also a token at position 5. Both receive the same absolute position encoding. But from the perspective of the current segment, these tokens are at very different distances: position 5 in the current segment is "here," while position 5 in the cached segment is 8 positions back (if segments have length 8).

The problem is even deeper than this. Absolute positions within a segment reset at the start of each segment. Position 0 in segment 1 and position 0 in segment 2 both have the same position encoding, but they are separated by the entire length of segment 1. When we concatenate their hidden states and ask the attention mechanism to use position information to understand their relationship, we give it the same encoding for tokens that are far apart. The attention mechanism cannot distinguish between "this is my first token" and "this is the first token in some previous segment."

Think of it like a book where every chapter starts its page numbers over from 1. If you're on page 5 of chapter 3 and want to cite page 5 of chapter 1, both references look identical in the page numbering system, but they point to completely different locations. The reader needs additional information (the chapter number) to disambiguate. Transformer-XL solves this analogous problem by replacing absolute position numbers with relative distance measures, so the position encoding always says "this token is $k$ steps before me" rather than "this token is at absolute position $n$."

If absolute position encodings were used, the model would receive conflicting signals. Two tokens with identical position encodings would be at different temporal distances. The attention mechanism, which relies on position information to understand sequence structure, would be confused.

Transformer-XL solves this with relative position encodings. Instead of encoding absolute positions and adding them to token embeddings, the model directly encodes the relative distance between query and key positions in the attention computation itself.

We've established that segment-level recurrence breaks absolute position encodings. A token at position 5 in the cached segment and position 5 in the current segment have the same absolute encoding, yet they are 8 positions apart from the perspective of the current segment. How do we fix this? The answer lies in rethinking what position information attention needs.

Attention computes a score between every query and every key to decide how much information should flow from each key to each query. Position information enters this computation to reflect the fact that two tokens at different positions should interact differently than two tokens at the same position. But the original formulation bakes in a specific assumption: that absolute position matters. Transformer-XL's relative position encoding challenges this assumption and replaces it with a more fundamental one: what matters is not where each token is, but how far apart the tokens are.

This changes the role of position information. The same pair of query and key content at a distance of 3 positions should produce the same position-influenced attention score regardless of whether that query is at absolute position 10 or absolute position 1000. The distance is always 3, so the position contribution to attention should always be the same. This property is called translation equivariance, and it is exactly what relative position encodings provide.

Think of relative position encoding as teaching the model about spatial relationships rather than absolute coordinates. Just as you naturally say "the store is two blocks north of here" rather than "the store is at GPS coordinates 37.7749, -122.4194," relative encodings describe relationships, not locations. Two tokens that are adjacent will always have a relative distance of 1, regardless of where they appear in the document.

When you read a sentence like "The cat sat on the mat because it was tired," you understand that "it" refers to "cat" not because of their absolute positions in the document, but because of their relative proximity. The pronoun comes shortly after its antecedent. This observation is the key insight: attention cares about how far apart tokens are, not where they are in absolute terms.

Consider two identical queries, one at position 10 and one at position 100, both attending to keys that are 3 positions before them. If the tokens involved have the same content, shouldn't these attention computations behave similarly? With absolute position encodings, they don't, because positions 7 and 97 have completely different encodings. With relative position encodings, they do, because "3 positions back" always means the same thing.

The implications for generalization are significant. If a model trained on sequences of length 512 learns that pronouns tend to refer to nouns within the previous 20 tokens, that pattern should generalize to any position in any sequence. Absolute position encodings undermine this generalization by making the same pattern look different at different absolute positions. Relative position encodings make the pattern position-agnostic, which is exactly what we want.

To understand how Transformer-XL achieves relative position encoding, we need to first dissect how position information enters standard attention. In the original transformer, the attention score between a query at position $i$ and a key at position $j$ starts as a simple dot product:

where:

• $\text{score}_{ij}$: the attention score determining how much position $i$ attends to position $j$
• $\mathbf{q}_i$: the query vector at position $i$
• $\mathbf{k}_j$: the key vector at position $j$

But where does position come in? The original transformer adds position embeddings to token embeddings before projecting to queries and keys. So the query at position $i$ is $W_q(\mathbf{x}_i + \mathbf{p}_i)$ and the key at position $j$ is $W_k(\mathbf{x}_j + \mathbf{p}_j)$. Substituting these into the dot product and using the distributive property of matrix multiplication:

where:

• $\mathbf{x}_i, \mathbf{x}_j$: token embeddings at positions $i$ and $j$
• $\mathbf{p}_i, \mathbf{p}_j$: absolute position embeddings for positions $i$ and $j$
• $W_q, W_k$: learnable query and key projection matrices

This is where the magic of algebra reveals hidden structure. When we expand this product using the distributive property, we get four distinct terms:

Each term tells us something different about why one token might attend to another:

• Content-content: "Does this token's meaning relate to that token's meaning?" This is pure semantic matching, independent of where the tokens appear.
• Content-position: "Given what this token is looking for, does that position matter?" For example, a verb might preferentially attend to its subject, which typically precedes it.
• Position-content: "Given where this token is, does that token's content matter more?" Early positions might attend differently than late positions.
• Position-position: "Do these two positions have an inherent affinity?" Adjacent positions might naturally attend to each other.

Why does this decomposition matter? Because it shows us exactly which parts of the attention score use absolute positions. Terms (b), (c), and (d) all depend on $\mathbf{p}_i$ or $\mathbf{p}_j$ (or both), the absolute position embeddings. These are the terms we need to fix. Term (a) is already position-free and can stay as is.

The problem with this decomposition is that all position information uses absolute positions $\mathbf{p}_i$ and $\mathbf{p}_j$. Transformer-XL's insight is that we can rewrite these terms to use relative position instead. The redesign makes two key changes:

1. Replace the key's absolute position with relative distance: Instead of $\mathbf{p}_j$ (the absolute position of the key), use $\mathbf{r}_{i-j}$ (the relative distance from query to key).

2. Replace the query's position with a learned global bias: The query's absolute position $\mathbf{p}_i$ becomes learned vectors $\mathbf{u}$ and $\mathbf{v}$ that don't depend on position at all.

The first change makes sense because the key's absolute position is what we want to replace. We want to say "this key is 5 positions before this query" rather than "this key is at absolute position 47." The second change is more subtle: the query's absolute position would still be ambiguous across segment boundaries even after fixing the key's encoding, so we replace it with a global learned bias that captures the query's general tendency to prefer nearby or distant keys, independent of where in the document the query appears.

The resulting formula is:

where:

• $\mathbf{x}_i, \mathbf{x}_j$: token embeddings at positions $i$ and $j$, unchanged from before
• $\mathbf{r}_{i-j}$: a sinusoidal encoding of the relative distance $i - j$, not the absolute position
• $W_{k,E}$: key projection matrix for content (the "E" stands for embeddings)
• $W_{k,R}$: key projection matrix for relative positions (the "R" stands for relative)
• $\mathbf{u}$: a learned global bias for content attention, shared across all query positions
• $\mathbf{v}$: a learned global bias for position attention, also shared across all positions

Why does this formula make sense? Notice that every term that previously depended on $\mathbf{p}_i$ or $\mathbf{p}_j$ now depends on either a relative distance encoding $\mathbf{r}_{i-j}$ or a global learned constant. The relative distance $i-j$ is the same for any pair of tokens separated by the same gap, regardless of their absolute positions. This means the formula produces the same attention contribution for identical content pairs at identical relative distances, regardless of where in the document those pairs appear.

Let's unpack each component:

1. Term (a): Pure content-based attention. This is unchanged from standard attention. The word "cat" attends to "feline" because of semantic similarity, regardless of position.

2. Term (b): Content-dependent distance preference. The query content determines how much the model cares about distance. A pronoun might strongly prefer nearby tokens, while a discourse marker might look further back.

3. Term (c): Global content importance. Some tokens are just important regardless of the query's position. The beginning-of-sentence token might receive attention from everywhere.

4. Term (d): Global distance preference. The model learns a general preference for certain distances. Typically, nearby tokens receive more attention than distant ones.

Terms (b) and (d) now depend on $i - j$ rather than on $i$ and $j$ separately. This means the position signal is the same whether we're at the start of the document or the end, whether we're attending within the current segment or reaching back into cached memory.

With the theory in place, let's implement relative position encoding step by step. We need two components: (1) a function to generate sinusoidal encodings for each possible relative distance, and (2) a function to compute attention scores using the four-term formula.

The sinusoidal encoding for relative positions works similarly to absolute position encodings, but instead of encoding absolute positions 0, 1, 2, ..., we encode relative distances ..., -2, -1, 0, 1, 2, .... Negative distances mean the key is before the query; positive distances mean the key is after the query (though in causal attention, we only see non-positive distances). The sinusoidal function ensures smooth, gradual variation between nearby distances, so encodings for similar distances are similar vectors.

The similarity matrix shows that encodings for nearby relative positions are similar, gradually diverging for larger distances. This smooth decay allows the attention mechanism to naturally prefer nearby tokens while still accessing distant context when needed. The negative similarities at large distances reflect the oscillatory nature of sinusoidal functions, which ensures that encodings for distant positions become distinct vectors rather than faded versions of nearby ones.

To make the segment-level recurrence concrete, let's trace how information about a specific token propagates across three segments of a simple model. Imagine a 2-layer model with segment length 4 processing the sentence: "The professor lectured. Students took notes. They asked questions."

We have three segments: ["The", "professor", "lectured", "."], ["Students", "took", "notes", "."], and ["They", "asked", "questions", "."]. The token "They" in segment 3 is a pronoun that refers to "Students" in segment 2. Let's trace how the model can resolve this reference.

Segment 1 processing. No cached memory exists. Layer 1 processes ["The", "professor", "lectured", "."] with standard causal attention. The hidden states at layer 1 encode things like "lecturer introducing a topic." Layer 2 receives these layer-1 states as keys and values and produces layer-2 hidden states. These layer-2 states are cached as $\mathbf{h}_1^2$ (segment 1, layer 2 output) and $\mathbf{h}_1^1$ (segment 1, layer 1 output).

Segment 2 processing. Layer 1 of segment 2 receives the layer-1 cached states from segment 1: $\mathbf{h}_1^0$ (the embedding layer outputs). The cache at layer $n$ uses the previous segment's layer $n-1$ output. Thus, layer 1 of segment 2 receives the embedding-level outputs of segment 1 as its extended key-value context. Layer 1 computes attention over ["Students" attended to "The", "professor", "lectured", "."] plus itself. The hidden state for "Students" at layer 1 now encodes "a plural subject, appearing after a lecturing context." Layer 2 of segment 2 receives layer-1 outputs of segment 1 as its extended context. The hidden state for "Students" at layer 2 now carries information about both the current segment and the previous one.

Segment 3 processing. Layer 1 of segment 3 receives the layer-1 outputs of segment 2 as its extended context. "They" at layer 1 can attend to the layer-1 representation of "Students" from segment 2. The layer-1 hidden state for "They" begins to encode a connection to the plural subject from the previous segment. Layer 2 of segment 3 receives the layer-2 outputs of segment 2 as its extended context. "They" at layer 2 now attends to the richer layer-2 representation of "Students," which already encoded the connection between "Students" and the lecturing context from segment 1. The final hidden state of "They" captures, through this chain of attention, a connection back to "Students" and even further to the "professor" from segment 1.

This trace illustrates several important properties. First, the coreference link between "They" and "Students" is resolved by a single attention operation at layer 1 of segment 3, which can directly attend to "Students" in the cached memory. Second, deeper connections (like the link between the students and the professor) are accessible through the layer-2 representations, which already aggregated information from segment 1 when processing segment 2. Third, the information path follows the recurrence chain: segment 3 layer 1 attends to segment 2 layer 0 cached states, and segment 3 layer 2 attends to segment 2 layer 1 cached states, which themselves incorporated segment 1 information.

The numerical details would require running an actual trained model, but the logical structure of the information flow is clear: information about tokens from prior segments is compressed into the hidden states and carried forward one segment at a time, with each layer having access to the corresponding layer's cached representation from the previous segment.

We've now covered the two core innovations of Transformer-XL: segment-level recurrence for extending context across segment boundaries, and relative position encoding for handling positions consistently across segments. Let's bring these pieces together into a complete implementation.

A Transformer-XL layer follows the same structure as a standard transformer layer: attention followed by feed-forward, with residual connections and layer normalization. The key differences are in the attention computation, where we must handle cached memory and compute relative position biases. Transformer-XL's original implementation uses "pre-norm" layering, applying layer normalization before the attention and feed-forward sublayers rather than after. This improves training stability for deep networks.

The implementation below simplifies some aspects for clarity, particularly multi-head attention (which runs this same computation in parallel across multiple heads with smaller dimensions) and the exact handling of the pre-norm versus post-norm choice. The core logic of concatenating cached states, computing extended keys and values, and applying the four-term attention score is faithfully represented.

The visualization reveals how attention is distributed between memory and the current segment. Early positions in the current segment allocate substantial attention to memory because they have limited local context. Later positions can attend more to the growing local context while still accessing memory for longer-range dependencies. This natural transition from memory-heavy to local-heavy attention across positions reflects the model's learned understanding that nearby context is usually more relevant but that distant context is invaluable when local cues are scarce.

How does Transformer-XL's approach compare to other long-context methods? We can evaluate on a synthetic task that explicitly requires long-range dependencies: the copying task. The model must reproduce a sequence of tokens after a long delay filled with noise. This task is a clean test because there is no way to guess the target tokens; the only path to success is maintaining an accurate memory of the original sequence across the delay.

The copying task is deliberately extreme: real language modeling rarely requires verbatim recall of distant tokens. But it is a diagnostic probe that reveals the hard boundary of a model's effective memory. Any model that cannot solve the copying task at a given delay length cannot reliably use information from that far back in any task.

The figure illustrates the fundamental limitation of fixed context windows. As delay length increases, the required context eventually exceeds any fixed segment length. Standard transformers fail in the shaded regions. Transformer-XL extends the failure threshold by a factor proportional to the number of layers, but the memory cache size and information decay still impose practical limits.

While Transformer-XL's segment-level recurrence significantly extends effective context, it comes with important limitations that shape when and how the technique should be applied. These limitations determine when recurrent memory is the right tool and when other approaches such as sparse attention or full long-context models would serve better.

Information decay over segments. Hidden states are finite-dimensional vectors. As information propagates through multiple segments, it inevitably compresses and degrades. A fact stated in segment 1 may be perfectly preserved in segment 2's hidden states, partially preserved in segment 3, and largely lost by segment 10. Unlike attention over the full sequence, recurrence cannot perfectly preserve arbitrary information over arbitrary distances.

The rate of decay depends on how distinctive the information is. A highly unusual word or a rare syntactic structure creates a strong signal in the hidden states that resists compression. Common words and typical sentence structures blend into the background representation more quickly. This means the model's effective memory is not uniform: it tends to remember remarkable events for longer and forget mundane details sooner. For language modeling, this selective decay can be useful because remarkable events are also more likely to be referenced later. For tasks requiring precise recall of any specific token regardless of distinctiveness, this selective decay can be problematic.

Training and inference divergence. During training, the memory cache is populated from training data, maintaining realistic statistics. During inference on new text, the cache may be empty at the start, creating a "cold start" problem where initial segments lack the memory context the model learned to expect. Some implementations address this by processing a warmup prefix that isn't used for generation, but this adds latency. The cold start problem is especially pronounced for tasks where the very beginning of a document is critical, such as generating a response to a prompt that constitutes the first segment.

Gradient computation complexity. Although the cached hidden states don't receive gradients (the StopGrad operation), backpropagation still needs to flow through the attention computation over the extended sequence. This slightly increases training complexity compared to pure segment-independent processing, though it's far less expensive than full sequence backpropagation. In practice, the StopGrad decision also means the model cannot learn to produce hidden states that are optimally informative as memory: the gradient signal for improving the quality of what gets cached is indirect, coming only through the representations that the next segment learns to extract from the memory.

Limited to one segment of direct memory. In the standard Transformer-XL formulation, the model only caches one previous segment's hidden states at each layer. Information from earlier segments is only accessible through the layered recurrence chain, and even then only up to $N$ segments back for an $N$-layer model. If you need to access information from 100 segments ago in a 12-layer model, you simply cannot, regardless of how much memory you have available. Some variants extend this by caching multiple previous segments, but this multiplies memory requirements and attention cost proportionally.

Transformer-XL's approach shines in specific scenarios where its properties are well-matched to the task. Understanding these scenarios helps you decide when recurrent memory is the right choice and when other long-context approaches would serve better.

Recurrent memory works best in situations where information flows naturally in one direction and nearby context matters more than distant context. Language modeling on long documents is the paradigmatic use case: you are always generating the next token based on everything that came before, and the most recent context is almost always the most relevant. Streaming applications where text arrives incrementally have a similar structure: you process each new piece as it arrives, carrying forward a memory of what came before. Document summarization, long-form text generation, and code completion in large files all fit this profile.

Memory-constrained settings are another natural fit for recurrent memory. When the alternative would be either truncating context or paying the quadratic cost of full attention over long sequences, caching a fixed-size memory provides a middle ground. The memory cost is linear in sequence length rather than quadratic, and the cached states compress a segment's information into a fixed-size buffer rather than growing without bound.

Transformer-XL's approach is less suitable when the task requires looking at context that may be far in the past, or when the relevant information is unpredictably distributed across the document. Question answering over long documents is a prime example: the answer may appear anywhere in the document, and the question text, which appears at the end, cannot inform how the earlier document was processed. Bidirectional tasks like natural language inference, which require comparing two pieces of text that may not be adjacent, also do not map well to unidirectional recurrence. For these tasks, either full bidirectional attention or retrieval-based approaches tend to work better.

Modern alternatives like FlashAttention and sparse attention patterns have reduced the cost of longer context windows, somewhat diminishing the need for recurrent approaches. However, when truly long sequences must be processed incrementally, segment-level recurrence remains a powerful tool. The recurrent approach also has the appealing property of being constant-cost per token at inference time: no matter how long the document is, processing each new segment costs the same amount. This makes it particularly well-suited for streaming inference where latency per token is a constraint.

The key tradeoff to keep in mind when choosing recurrent memory is specificity versus scale. Recurrent memory can handle very long documents with modest computational cost, but it trades precision for scale. If your task requires precise access to information from many segments ago, recurrent memory will likely disappoint. If your task involves gradual accumulation of context where the recent past is most important and the distant past provides soft background influence, recurrent memory is an excellent choice.

This chapter explored Transformer-XL's recurrent memory mechanism, which extends effective context beyond the fixed segment length by caching and reusing hidden states from previous segments.

Context fragmentation, where fixed-length segment processing breaks cross-segment dependencies, fundamentally limits standard transformers. Transformer-XL addresses this by caching the hidden states from the previous segment and concatenating them with the current segment's keys and values. Queries still come only from the current segment, creating an asymmetric attention pattern that enables information flow from past to present while keeping computational cost constant per segment.

The recurrence mechanism requires relative position encodings because absolute positions would be ambiguous across segments. Transformer-XL redesigns the attention score computation to depend on the relative distance between query and key positions rather than their absolute locations. This involves decomposing the standard attention score into four terms and replacing the absolute-position-dependent terms with relative distance encodings and global learned biases. The result is an attention mechanism that behaves consistently regardless of where in the document a pair of tokens appears.

The effective context length grows with network depth. Information propagates one segment further back at each layer, so an $N$-layer model has a theoretical reach of $N$ segments beyond the directly attended memory. In practice, information decay limits this reach, but the extension is still substantial, and the same depth that provides expressiveness also provides temporal reach.

Key implementation considerations include:

• Memory cache storage scales with memory length, hidden dimension, number of layers, and batch size
• The StopGrad operation prevents backpropagation through cached states, limiting training signal for long-range learning
• Cold start at inference time may require warmup prefixes to populate the cache
• Unidirectional information flow limits applicability to bidirectional tasks
• Information decay across segments means recurrent memory works better for statistical dependencies than precise recall

Recurrent memory represents a principled approach to the long-context problem: accept that we cannot attend to everything at once, but ensure that information can flow across the boundaries we impose. While modern advances in attention efficiency have expanded what "at once" can mean, the core insight, that hidden states can carry forward context without explicit attention, remains valuable for streaming and memory-efficient processing of long sequences.

When implementing Transformer-XL or similar recurrent memory mechanisms, the following parameters have the greatest impact on model behavior:

• segment_length: The number of tokens processed in each forward pass. Larger segments capture more local context but increase memory usage quadratically (due to attention). Typical values range from 128 to 512 tokens.

• memory_length: The number of tokens cached from the previous segment. Usually set equal to segment_length, but can be larger to extend context reach at the cost of increased memory and computation.

• num_layers: Deeper networks extend effective context linearly. A 24-layer model can theoretically access 24x more context than a single layer, though information decay limits practical gains.

• d_model: The hidden dimension affects both model capacity and memory requirements. Cache memory scales linearly with this parameter. Common values are 768 (BERT-base) to 1024 (GPT-2).

• max_rel_dist: The maximum relative distance for position encodings. Should be at least segment_length + memory_length to cover all possible query-key distances. Setting this too small causes position information to saturate for distant tokens.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about Transformer-XL and recurrent memory mechanisms.