Global Tokens › Summary

Global tokens bridge efficient local attention and effective long-range modeling. By designating certain positions as communication hubs with full attention span, they reduce the maximum path length between any two positions to just 2 hops, achieving near-full-attention connectivity at linear computational cost.

Key takeaways:

• The bottleneck problem: Sliding window attention limits direct communication to local neighborhoods, requiring many hops for long-range information flow.
• Global tokens as hubs: Designated positions that can attend to and be attended by all positions, serving as relay points for sequence-wide communication.
• CLS token attention: BERT's classification token naturally fits the global attention role, aggregating sequence information for downstream tasks.
• Task-specific globals: Question tokens in QA, separator tokens in sentence pairs, or any semantically meaningful positions can serve as global tokens.
• Learned latents: Trainable global tokens that discover what information to aggregate. This gives flexibility at the cost of additional parameters.
• Efficient scaling: Global token count scales linearly, so even many global tokens preserve the efficiency gains of sparse attention.
• Reduced path length: With global tokens, maximum path length drops from $O(n/w)$ to 2, where $n$ is the sequence length and $w$ is the window size. This matches full attention's connectivity properties.
• Deployment considerations: Production use requires sparse attention kernels for true linear memory, careful gradient management for global token parameters, and thoughtful decisions about global token placement and count for the specific task.

The next chapter examines Longformer, which combines sliding window attention with global tokens into a complete architecture. We will see how these components work together to achieve strong performance on document-level NLP tasks while maintaining linear complexity in sequence length.