Recurrent Memory › The Segment Boundary Problem
Language AI HandbookAll chapters Recurrent Memory › The Segment Boundary ProblemOne section · Sections in this chapter2 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Standard transformers suffer from what the Transformer-XL paper calls "context fragmentation." When you split a long document into fixed-length segments, each segment is processed without any knowledge of its neighbors. The model sees each chunk as an independent sequence. This limitation affects predictions directly: the model's predictions for the first few tokens of each new segment are systematically worse than predictions in the middle of a segment, because those early tokens have no context to draw on. The model is perpetually starting cold.
The severity of context fragmentation depends on both the segment length and the nature of the text. For documents with frequent long-range dependencies, every segment boundary is a potential failure point. Consider coreference resolution, the task of linking pronouns to their referents. If "the researcher" appears in segment 1 and "she" appears in segment 2, a model with context fragmentation cannot resolve the pronoun, even if a human reader would find the reference obvious. The same problem arises with discourse connectives ("As mentioned earlier..."), ellipsis resolution, and any form of reasoning that requires holding multiple pieces of information in mind simultaneously.
Think of context fragmentation as reading a story through a narrow window that reveals only one page at a time. Each page you read, you must discard the previous one. When a character on page 15 says "You know what I mean," you cannot, because the context that would let you understand was on page 12, which has already been discarded. The model faces this exact situation at every segment boundary.
The quantitative impact of context fragmentation is stark. Consider a 4096-token document processed with a 512-token window. Only a small fraction of all possible token-to-token dependencies are visible within any single segment. The vast majority of potentially informative connections between tokens are simply cut off.
Context fragmentation occurs when a transformer processes a long sequence in fixed-length segments without information flow between them. Each segment starts fresh, losing all context from previous segments regardless of semantic continuity.
Consider processing a 4096-token document with a 512-token context window. The naive approach splits this into 8 segments, processing each independently. Token 513 (the first token of segment 2) cannot attend to token 512 (the last token of segment 1). They're as disconnected as tokens from completely different documents.
# Simulate context fragmentation
document_length = 4096
segment_length = 512
num_segments = document_length // segment_length
# For each segment, the maximum dependency distance is limited
# to within-segment connections only
max_within_segment_distance = segment_length - 1
max_possible_distance = document_length - 1
# Calculate the fraction of potential dependencies that are visible
visible_dependencies_per_segment = segment_length * (segment_length - 1) // 2
total_possible_dependencies = document_length * (document_length - 1) // 2
visibility_fraction = (
visible_dependencies_per_segment * num_segments
) / total_possible_dependenciesDocument length: 4,096 tokens Segment length: 512 tokens Number of segments: 8 Within each segment: Maximum attention distance: 511 tokens Visible dependencies per segment: 130,816 Across entire document: Total possible dependencies: 8,386,560 Fraction visible with fragmentation: 12.48%
Less than 13% of potential token-to-token dependencies are visible when processing with context fragmentation. Cross-segment dependencies, which may carry information such as coreference chains or long-range discourse structure, are completely invisible.
The attention pattern reveals the fundamental limitation: each segment is an island. No matter how important a reference in segment 1 might be for understanding segment 4, that information cannot flow through the attention mechanism. The red boundaries are walls, not suggestions.