Global Tokens › The Information Bottleneck in Local Attention
Language AI HandbookAll chapters Global Tokens › The Information Bottleneck in Local AttentionOne section · Sections in this chapter2 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Sliding window attention limits each token to a local neighborhood. While efficient, this creates a path length problem reminiscent of RNNs. To transfer information from the start of a document to the end, data must flow through overlapping windows one hop at a time.
The problem appears in practice. A document understanding model reading a legal contract needs tokens near the end of the document, where the signature block appears, to be influenced by the definitions established at the beginning. A summarization model processing a news article needs the concluding paragraph to be shaped by the topic sentence in the lede. A question answering model reading a scientific paper needs the answer token, which may appear in the methods section, to be directly aware of the question about experimental design. In all these cases, the long-range dependency is not optional: it is the core of the task.
Consider a sequence of 4096 tokens with a window size of 512. A token at position 0 can directly see tokens up to position 256 (half the window). To reach position 4000, information must traverse roughly 16 hops through successive windows. Each hop introduces potential for information loss or distortion.
This 16-hop path can cause problems in neural networks. Each attention layer transforms representations, and there is no guarantee that a specific piece of information introduced at position 0 will survive intact through 16 layers of transformation as it propagates toward position 4000. In an RNN, this was called the vanishing gradient problem in a different guise. In sliding window attention, it manifests as effective forgetting: the further the semantic distance, the more attenuated the information signal becomes across successive layers.
Path length measures the number of attention operations needed for one position to influence another. Standard self-attention has path length 1 (direct connection). Sliding window attention has path length proportional to sequence length divided by window size. The formal bound is:
where is the distance between positions and is the window size. For a 4096-token sequence with window size 512, two positions at opposite ends of the sequence have a path length of 16.
Global tokens restore direct connections by serving as intermediaries. Any token can communicate with any other token by passing through a global token, reducing the maximum path length to 2.
The path length of 2 deserves careful attention, because it provides a stronger guarantee than it first appears. With full self-attention, every pair of tokens communicates in exactly 1 hop: direct attention. With sliding window attention alone, the maximum path is , which grows with sequence length. With global tokens added, the bound becomes 2 regardless of sequence length. This means that even for a million-token sequence, any two tokens are always just one global token hop away from each other. The global token architecture achieves a property that is qualitatively closer to full attention than to pure sliding window attention, even though its complexity profile resembles the latter.
import numpy as np
def compute_path_lengths(seq_len, window_size, num_global=0):
"""
Compute maximum path lengths for different attention patterns.
Path length = minimum number of attention operations for info to flow
between any two positions.
"""
# Sliding window: path length is ceil(distance / (window_size / 2))
max_distance = seq_len - 1
window_path = int(np.ceil(max_distance / (window_size // 2)))
# With global tokens: max path is 2 (token -> global -> token)
global_path = 2 if num_global > 0 else window_path
return {
"full_attention": 1,
"sliding_window": window_path,
"with_global_tokens": global_path,
}
# Example: 4096 token sequence
seq_len = 4096
window_size = 512
paths = compute_path_lengths(seq_len, window_size, num_global=1)
The difference is stark. Sliding window attention requires 16 hops to connect distant positions, while adding just one global token reduces this to 2. This improvement in connectivity comes with minimal computational overhead since only a few tokens become global.
What the chart also shows is that the three approaches divide naturally into two qualitative tiers. Full attention and global-token attention both achieve constant-bounded path lengths (1 and 2 respectively), while pure sliding window attention has a path length that grows with the sequence. That qualitative difference has real consequences during training: models with bounded path lengths can learn long-range dependencies through gradient backpropagation much more reliably than models where the gradient signal must travel through 16 or more attention layers to propagate between distant positions.