Transformer-XL (2019) › 2019: Transformer-XL
History of Language AIAll chapters Transformer-XL (2019) › 2019: Transformer-XLOne section · Sections in this chapter1 / 7 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Transformer language models were commonly trained on fixed-length pieces of text. A piece started without the hidden states computed for the preceding piece, even when both came from the same document. This created context fragmentation: a dependency crossing the boundary disappeared from the model's input.
Transformer-XL, published in 2019 by researchers at Carnegie Mellon University and Google Brain, connected those pieces. Each layer cached activations from earlier segments and exposed them as memory to the next segment. A new relative-position formulation made those reused activations positionally consistent with the current query.
The architecture did not give attention unlimited or linear-cost memory. It processed one segment at a time against a bounded cache. This reused computation and extended the effective context while retaining standard attention within that working set.