Transformer-XL (2019) › Legacy

Transformer-XL supplied a concrete answer to context fragmentation: retain earlier layer activations and reuse them without backpropagating through the full history. XLNet is the clearest direct continuation of that design.

Its relative-attention formulation also helped establish relative position as an alternative to adding absolute vectors at the input. Later systems used several distinct methods, including relative biases and rotary embeddings. Those methods share a design concern with Transformer-XL but should not all be described as implementations of its exact scheme. GPT-3, for example, used learned absolute position embeddings.

The chapter's main lesson is the tradeoff, not a claim of unlimited context. A fixed cache preserves recent hidden states and avoids recomputation. Extending that cache increases attention cost, and frozen memories cannot be rewritten by later evidence.