Recurrent Memory › Summary

This chapter explored Transformer-XL's recurrent memory mechanism, which extends effective context beyond the fixed segment length by caching and reusing hidden states from previous segments.

Context fragmentation, where fixed-length segment processing breaks cross-segment dependencies, fundamentally limits standard transformers. Transformer-XL addresses this by caching the hidden states from the previous segment and concatenating them with the current segment's keys and values. Queries still come only from the current segment, creating an asymmetric attention pattern that enables information flow from past to present while keeping computational cost constant per segment.

The recurrence mechanism requires relative position encodings because absolute positions would be ambiguous across segments. Transformer-XL redesigns the attention score computation to depend on the relative distance between query and key positions rather than their absolute locations. This involves decomposing the standard attention score into four terms and replacing the absolute-position-dependent terms with relative distance encodings and global learned biases. The result is an attention mechanism that behaves consistently regardless of where in the document a pair of tokens appears.

The effective context length grows with network depth. Information propagates one segment further back at each layer, so an $N$-layer model has a theoretical reach of $N$ segments beyond the directly attended memory. In practice, information decay limits this reach, but the extension is still substantial, and the same depth that provides expressiveness also provides temporal reach.

Key implementation considerations include:

• Memory cache storage scales with memory length, hidden dimension, number of layers, and batch size
• The StopGrad operation prevents backpropagation through cached states, limiting training signal for long-range learning
• Cold start at inference time may require warmup prefixes to populate the cache
• Unidirectional information flow limits applicability to bidirectional tasks
• Information decay across segments means recurrent memory works better for statistical dependencies than precise recall

Recurrent memory represents a principled approach to the long-context problem: accept that we cannot attend to everything at once, but ensure that information can flow across the boundaries we impose. While modern advances in attention efficiency have expanded what "at once" can mean, the core insight, that hidden states can carry forward context without explicit attention, remains valuable for streaming and memory-efficient processing of long sequences.