Global Tokens › Comparison: Different Global Token Strategies
Language AI HandbookAll chapters Global Tokens › Comparison: Different Global Token StrategiesOne section · Sections in this chapter10 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Different architectures use global tokens in distinct ways. Understanding these variations helps when choosing or designing efficient attention mechanisms for a specific task. The variety of approaches reflects the fact that "which tokens should be globally visible" is fundamentally a domain knowledge question, and different tasks come with different structural priors about which positions carry globally relevant information.
The design space can be organized along two axes: whether the global token set is determined by position (positional strategies) or by semantics (semantic strategies), and whether it is fixed before training (static strategies) or discovered during training (learned strategies). CLS-only is positional and static. Task-specific tokens like question tokens are semantic and static. Periodic tokens are positional and static. Learned latents are semantic and dynamic, because their effective content, though not their positions, is determined by training.
| Strategy | Global Tokens | Use Case | Advantages |
|---|---|---|---|
| CLS-only (Longformer) | First token | Classification | Simple, single aggregation point |
| Task-specific (QA) | Question tokens | Extractive tasks | Semantic alignment with task structure |
| Periodic (BigBird) | Every k-th token | General | Uniform coverage, no special tokens needed |
| Learned latents (Perceiver) | Separate buffer | Multimodal, long sequences | Flexible, task-agnostic |
Each strategy makes trade-offs. CLS-only is simplest but creates a single bottleneck. Task-specific requires knowing the task structure. Periodic global tokens offer even coverage but may waste capacity on uninformative positions. Learned latents are most flexible but add trainable parameters and require the model to discover the global structure from scratch, which can slow convergence on small datasets.
The periodic strategy, used in BigBird, deserves particular attention because it offers a theoretical guarantee absent from the others. BigBird proved that its combination of random and local attention, along with global attention, is a universal approximator of sequence functions in the sense that it can approximate any function on sequences that a full-attention transformer can compute. The periodic global tokens in BigBird act as the "global" component in this theoretical framework. This ensures that there are always positions that can see the full sequence regardless of where the random attention heads happen to land.
def compare_global_strategies(seq_len, window_size):
"""Compare connectivity of different global token strategies."""
strategies = {
"CLS only": [0],
"CLS + SEP": [0, seq_len - 1],
"Periodic (every 8)": list(range(0, seq_len, 8)),
"First 4 tokens": list(range(4)),
}
results = []
for name, global_pos in strategies.items():
mask = create_longformer_attention_mask(
seq_len, window_size, global_pos
)
density = mask.sum() / (seq_len * seq_len)
complexity = compute_attention_complexity(
seq_len, window_size, len(global_pos)
)
results.append(
{
"strategy": name,
"num_global": len(global_pos),
"density": density * 100,
"savings": complexity["savings_percent"],
}
)
return results
# Compare strategies
seq_len = 64
window_size = 8
comparison = compare_global_strategies(seq_len, window_size)Comparison: seq_len=64, window_size=8 Strategy # Global Density Savings -------------------------------------------------------- CLS only 1 16.5% 84.4% CLS + SEP 2 19.3% 81.3% Periodic (every 8) 8 33.9% 64.1% First 4 tokens 4 24.8% 75.4%
All strategies maintain significant savings compared to full attention. The periodic strategy with 8 global tokens has higher density (more connections) but still achieves over 80% savings. The choice depends on whether you need uniform global coverage (periodic) or task-aligned global positions (CLS, question tokens).
The density column is the most informative for understanding the attention pattern's sparsity. A density of 15% means that only 15% of all possible token pairs attend to each other. This sparsity is what makes the attention computable in linear time: the 85% of pairs that do not attend need not be computed at all. Notice that even the densest strategy in the comparison (Periodic with every-8th token as global) achieves over 80% savings, confirming that the linear cost of global tokens is small relative to the quadratic cost of full attention.