Recurrent Memory › Evaluation: Comparing Context Approaches
Language AI HandbookAll chapters Recurrent Memory › Evaluation: Comparing Context ApproachesOne section · Sections in this chapter9 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
How does Transformer-XL's approach compare to other long-context methods? We can evaluate on a synthetic task that explicitly requires long-range dependencies: the copying task. The model must reproduce a sequence of tokens after a long delay filled with noise. This task is a clean test because there is no way to guess the target tokens; the only path to success is maintaining an accurate memory of the original sequence across the delay.
The copying task is deliberately extreme: real language modeling rarely requires verbatim recall of distant tokens. But it is a diagnostic probe that reveals the hard boundary of a model's effective memory. Any model that cannot solve the copying task at a given delay length cannot reliably use information from that far back in any task.
def create_copying_task(
seq_length: int, copy_length: int, delay_length: int, vocab_size: int = 10
) -> tuple[np.ndarray, np.ndarray]:
"""
Create a copying task instance.
The input consists of:
1. A sequence of tokens to remember (copy_length tokens)
2. A delay period filled with blanks (delay_length tokens)
3. A signal token indicating the model should start reproducing
The target is the original sequence of tokens.
"""
rng = np.random.default_rng(42)
# Tokens to copy (1 to vocab_size-2, reserve 0 for blank, vocab_size-1 for signal)
to_copy = rng.integers(1, vocab_size - 1, size=copy_length)
# Build input sequence
blank_token = 0
signal_token = vocab_size - 1
input_seq = np.concatenate(
[
to_copy,
np.full(delay_length, blank_token),
[signal_token],
np.full(
copy_length - 1, blank_token
), # Positions where model outputs
]
)
# Target: just the copied tokens at the end
target = np.concatenate(
[
np.full(copy_length + delay_length + 1, -1), # -1 = ignore
to_copy[:-1], # Predict each token in the copy
]
)
return input_seq, target, to_copy
# Create examples with different delay lengths
delays = [50, 100, 200, 400]
copy_len = 10
examples = [create_copying_task(500, copy_len, d) for d in delays]Copying Task Examples Task: Remember the first 10 tokens, output them after a delay Delay length: 50 Tokens to copy: [1 7 6 4 4 7 1 6 2 1] Input length: 70 Required context: 60 positions Delay length: 100 Tokens to copy: [1 7 6 4 4 7 1 6 2 1] Input length: 120 Required context: 110 positions Delay length: 200 Tokens to copy: [1 7 6 4 4 7 1 6 2 1] Input length: 220 Required context: 210 positions Delay length: 400 Tokens to copy: [1 7 6 4 4 7 1 6 2 1] Input length: 420 Required context: 410 positions
This copying task becomes impossible for models without sufficient context. If the segment length is 100 and the delay is 200, a standard transformer with context fragmentation can never succeed because the tokens to copy fall outside any segment that needs to reproduce them. Transformer-XL can succeed as long as the required context falls within its effective reach of tokens.
The figure illustrates the fundamental limitation of fixed context windows. As delay length increases, the required context eventually exceeds any fixed segment length. Standard transformers fail in the shaded regions. Transformer-XL extends the failure threshold by a factor proportional to the number of layers, but the memory cache size and information decay still impose practical limits.