Transformer-XL (2019) › Applications and Impact
History of Language AIAll chapters Transformer-XL (2019) › Applications and ImpactOne section · Sections in this chapter4 / 7 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Transformer-XL was introduced and tested as an autoregressive language model. Its most direct application was generation or scoring of a continuous stream while carrying a bounded history between segments. Claims about document classification, coreference, or code generation require separate task-specific evidence and were not results of the original paper.
The released code and checkpoints made the recurrence mechanism reusable. XLNet, published later in 2019 by overlapping authors, adopted Transformer-XL's recurrence and relative attention while changing the pretraining objective.
Transformer-XL also made cached activations a visible design choice for long-context models. Later memory models explored compression or retrieval, while sparse-attention models took a different route by changing which token pairs could interact. These are alternative engineering strategies, not interchangeable forms of recurrence.