Transformer-XL (2019) › The Solution

Let $\mathbf{h}_{\tau}^{n}$ denote the hidden states at layer $n$ for segment $\tau$. When processing the next segment, layer $n$ receives a memory made from earlier layer activations and the current segment:

$$\widetilde{\mathbf{h}}_{\tau+1}^{n-1} = \left[ \operatorname{SG}\!\left(\mathbf{h}_{\tau}^{n-1}\right) \,\Vert\, \mathbf{h}_{\tau+1}^{n-1} \right],$$

where $\operatorname{SG}$ stops gradients through the cached segment and $\Vert$ denotes concatenation. Queries come from the current segment. Keys and values come from the concatenated memory, so a current token can attend to earlier cached states.

Stopping the gradient is an important boundary. Information from the cache affects the forward pass, but backpropagation does not continue through the entire document. Training remains segment-based rather than becoming full-document backpropagation through time.

The memory length is a configuration choice. With current length $L$ and memory length $M$, attention for one layer is proportional to $L(L+M)$. Reusing cached states avoids recomputing them, but attending to a larger memory still costs time and storage. Production implementations normally keep a fixed-size cache and discard older states.

Transformer-XL decomposes an attention score into content and position terms. The position term depends on the distance $i-j$ between a current query at $i$ and a key at $j$. It uses a sinusoidal relative-position vector together with learned projection and bias parameters. It is not a learned lookup embedding for every possible distance.

This formulation solves the cache mismatch because a stored state has no single new absolute index. Its positional contribution is recomputed from its distance to each current query. The model can also evaluate a longer attention length than the segment length used during training, although performance at unseen distances remains an empirical question rather than a guarantee.

The paper reported new results on five language-modeling datasets, including perplexity 18.3 on WikiText-103 and 21.8 on One Billion Word. Its relative effective context length was measured as 450% longer than the vanilla Transformer baseline in the paper's setup.

The reported evaluation speedup of up to 1,800 times compared Transformer-XL's cached generation with a vanilla Transformer that recomputed a long context for each prediction. It was not a claim that Transformer-XL attention was 1,800 times cheaper for every workload.