Transformer-XL (2019) › Limitations

The cache contains fixed hidden states. A later token can read them, but new evidence does not revise their representations. This asymmetry suits causal language modeling; it is less suitable when a task needs bidirectional reinterpretation of the entire document.

A bounded cache still forgets. Increasing $M$ lengthens the available history and raises the $L(L+M)$ attention cost. Segment recurrence reuses old computation, but it does not remove the memory-compute tradeoff.

Gradients stop at the segment boundary. The model can learn to use cached states because they are present in the forward pass, but it cannot assign credit through an arbitrarily long chain of prior segments during one update.

Relative position does not make length extrapolation automatic. Queries at evaluation time may attend over distances that were rare during training. The formulation permits those distances to be represented; model quality still depends on the learned behavior.

Finally, recurrence imposes sequential processing across segments. Tokens within a segment remain parallel, but segment $\tau+1$ needs the cache produced for segment $\tau$. That dependency limits document-level training parallelism.