Recurrent Memory › Recurrent Memory

Transformers process sequences in fixed-length segments. When a document exceeds the context window, the standard approach is to truncate or split it, processing each piece independently. But this independence comes at a cost: information from earlier segments vanishes entirely. A pronoun in segment 3 cannot resolve to its antecedent in segment 1 because segment 1 no longer exists in the model's view. The transformer has, in effect, amnesia: it wakes up at the start of each segment with no memory of what came before.

This limitation becomes painfully apparent when working with real-world text. A legal brief references a clause defined pages earlier. A novel's character acts on motivations established chapters back. A scientific paper's conclusion depends on definitions in its introduction. When we split these documents into segments and process each independently, we destroy the long-range connections that give language its meaning. The model sees each fragment in isolation and cannot form the kind of coherent, document-level understanding that these texts demand.

Transformer-XL introduced a solution that seems almost obvious in retrospect: what if we kept the hidden states from the previous segment and let the current segment attend to them? This segment-level recurrence creates a form of memory that extends effective context far beyond the training sequence length. The model processes sequences one segment at a time, but each segment can "remember" what came before through cached hidden states. Think of it like reading a book with a bookmark that marks your place and carries forward a compressed summary of everything you have read. You do not reread from the beginning, but that summary shapes how you understand each new page.

Transformer-XL extends context without recomputing attention over the entire history. Storing and recomputing attention over all past tokens would cost quadratic time in the total sequence length, which is exactly what we wanted to avoid by using segments in the first place. Instead, the cached hidden states serve as a fixed-length, information-dense memory that can be attended to with only linear overhead relative to the segment size. The tradeoff is that information is compressed into a fixed-dimensional representation rather than preserved exactly, but this compression is what makes the approach computationally tractable.

This chapter explores how Transformer-XL implements recurrent memory, why it requires relative positional encodings, and what limitations remain. Understanding this approach illuminates a key tension in long-context modeling: the tradeoff between computational efficiency and true bidirectional context. We will also see that solving the memory problem immediately creates a new problem: positions. When hidden states from one segment are concatenated with hidden states from another, the model must understand where each token lies in the combined sequence. Standard absolute position encodings fail here in a subtle but important way, leading to one of Transformer-XL's most elegant contributions.