Global Tokens › Limitations and Impact

Global tokens solve the long-range dependency problem in sparse attention, but they come with limitations that become more pronounced as sequence lengths and task complexities grow.

The primary limitation is the potential information bottleneck. A single CLS token must compress an entire document's worth of information into one vector of dimension $d$, typically 768 or 1024 in standard BERT-scale models. For a short paragraph, this is entirely feasible: the CLS token can capture the main topic and sentiment, plus key entities, without strain. For a 10,000-word document covering multiple topics, the same fixed-size vector must somehow encode everything that might be globally relevant, and something will inevitably be lost.

This bottleneck manifests in a specific failure mode: global tokens tend to saturate on frequent, high-salience features and lose less prominent information. In a legal document, the CLS token might learn to encode the parties, the date, and the type of agreement, but lose track of specific clause exceptions buried in the body text. Those exceptions may be precisely what a downstream model needs to answer a specific legal question. The solution, adding more global tokens, works but increases the number of parameters receiving full-sequence gradients, which in turn increases the effective computation.

Global tokens also introduce asymmetry in the attention pattern. The CLS token sees everything, while other tokens see only windows plus global positions. This asymmetry can lead to uneven gradient flow during training, with global token parameters receiving disproportionately large updates. In a transformer with many layers, this can cause the global token's representation to drift toward patterns that are easy to learn but not maximally informative for the task. Careful initialization and learning rate scheduling help mitigate this issue, but they add complexity to the training pipeline.

A third limitation is that global tokens require the practitioner to make a design decision before training: how many global tokens, and where? For well-studied tasks like classification and extractive QA, the answer is relatively clear. For novel tasks or tasks where the relevant global context is not known in advance, choosing the wrong global token configuration can hurt performance. Learned global tokens side-step this problem but require more data and longer training to discover good configurations.

Despite these limitations, global tokens work remarkably well in practice. Longformer achieved state-of-the-art results on long-document tasks like WikiHop and TriviaQA when it was introduced in 2020. The ability to process 4096 tokens efficiently, compared to BERT's maximum of 512, opened new applications in document understanding, long-form question answering, and summarization. BigBird extended this further to 4096 and beyond, with theoretical guarantees about universal approximation that provided a principled foundation for the empirical successes of sparse global-local attention.

The conceptual impact may be even more significant than the benchmark numbers. Global tokens demonstrate that carefully designed sparse patterns can match or exceed the performance of full attention while being much more efficient. This principle has influenced subsequent architectures, from Perceiver's latent arrays to modern retrieval-augmented models that retrieve relevant context rather than attending to everything. The insight that "not everything needs to attend to everything, but some things need to attend to everything" is simple to state but took the field several years and multiple architectures to fully internalize and exploit. It continues to shape how researchers design attention mechanisms for long-context language models today.