Recurrent Memory › When to Use Recurrent Memory

Transformer-XL's approach shines in specific scenarios where its properties are well-matched to the task. Understanding these scenarios helps you decide when recurrent memory is the right choice and when other long-context approaches would serve better.

Recurrent memory works best in situations where information flows naturally in one direction and nearby context matters more than distant context. Language modeling on long documents is the paradigmatic use case: you are always generating the next token based on everything that came before, and the most recent context is almost always the most relevant. Streaming applications where text arrives incrementally have a similar structure: you process each new piece as it arrives, carrying forward a memory of what came before. Document summarization, long-form text generation, and code completion in large files all fit this profile.

Memory-constrained settings are another natural fit for recurrent memory. When the alternative would be either truncating context or paying the quadratic cost of full attention over long sequences, caching a fixed-size memory provides a middle ground. The memory cost is linear in sequence length rather than quadratic, and the cached states compress a segment's information into a fixed-size buffer rather than growing without bound.

Transformer-XL's approach is less suitable when the task requires looking at context that may be far in the past, or when the relevant information is unpredictably distributed across the document. Question answering over long documents is a prime example: the answer may appear anywhere in the document, and the question text, which appears at the end, cannot inform how the earlier document was processed. Bidirectional tasks like natural language inference, which require comparing two pieces of text that may not be adjacent, also do not map well to unidirectional recurrence. For these tasks, either full bidirectional attention or retrieval-based approaches tend to work better.

Modern alternatives like FlashAttention and sparse attention patterns have reduced the cost of longer context windows, somewhat diminishing the need for recurrent approaches. However, when truly long sequences must be processed incrementally, segment-level recurrence remains a powerful tool. The recurrent approach also has the appealing property of being constant-cost per token at inference time: no matter how long the document is, processing each new segment costs the same amount. This makes it particularly well-suited for streaming inference where latency per token is a constraint.

The key tradeoff to keep in mind when choosing recurrent memory is specificity versus scale. Recurrent memory can handle very long documents with modest computational cost, but it trades precision for scale. If your task requires precise access to information from many segments ago, recurrent memory will likely disappoint. If your task involves gradual accumulation of context where the recent past is most important and the distant past provides soft background influence, recurrent memory is an excellent choice.