Global Tokens › Global-Local Attention Mixing
Language AI HandbookAll chapters Global Tokens › Global-Local Attention MixingOne section · Sections in this chapter8 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The interplay between global and local attention creates interesting information flow patterns. In a single attention layer, information moves in two streams. The local stream propagates through sliding windows, carrying fine-grained positional information about neighboring context. The global stream broadcasts through designated tokens. This provides sequence-wide context without the positional specificity of local attention.
These two streams are not independent. Local tokens contribute to the global token's representation at each layer, and the global token's representation then flows back to local tokens. Over multiple layers, this creates a feedback loop: local context informs the global summary, which then enriches local representations, which then further refine the global summary. The process converges to a state where both local and global representations are mutually informed by each other, a property that pure sliding window models cannot achieve.
The information flow through layers is cumulative in a precise sense. A token's effective receptive field, meaning the set of all positions that can influence its representation, grows with each layer. After one layer with a window of size and a single global token, a token's receptive field includes its -neighborhood plus the global token's information (which encompasses the full sequence). After two layers, the token can also see through the global token's two-layer receptive field, and through the local neighbors' one-layer receptive fields. The receptive field does not grow linearly: it jumps to near-full coverage after the first layer due to the global token's presence, then deepens in quality over subsequent layers.
def simulate_information_flow(
seq_len, window_size, global_positions, num_layers
):
"""
Simulate how information spreads through global-local attention.
Returns reachability matrix: which positions can influence which after L layers.
"""
# Start with direct connections from attention mask
mask = create_longformer_attention_mask(
seq_len, window_size, global_positions
)
reachable = mask.copy()
layer_snapshots = [reachable.copy()]
# Propagate through layers
for _ in range(num_layers - 1):
# Position j can reach position i if there's any intermediate k
# where j reaches k and k reaches i
new_reachable = reachable @ reachable > 0
reachable = reachable | new_reachable
layer_snapshots.append(reachable.copy())
return layer_snapshots
# Simulate: 32 tokens, window 8, CLS at 0
seq_len = 32
window_size = 8
global_positions = [0]
snapshots = simulate_information_flow(
seq_len, window_size, global_positions, num_layers=3
)
The visualization reveals how quickly information propagates. After one layer, connectivity is limited to local neighborhoods plus the global token. By layer 2, the bidirectional global token has served as a relay and every position can reach every other position. Layer 3 remains fully connected; its value is representational refinement, not additional graph reachability.
This suggests that even with aggressive sparsity, a few layers of global-local attention can match the connectivity of full attention. The key insight is that global tokens provide "shortcuts" that reduce the effective diameter of the attention graph.
The jump between layer 1 and layer 2 connectivity is particularly striking. After one layer, only about 34% of position pairs can reach each other. After two layers, that number reaches 100%. This discontinuous jump is a direct consequence of the bidirectional global token. Without it, the window would expand only incrementally and still miss distant pairs. The global token converts slow growth in connectivity into full two-hop coverage.