Global Tokens › Global Tokens

Sparse attention patterns like sliding windows solve the quadratic complexity problem but introduce a new challenge: how does information travel across the full sequence? If each token only attends to nearby neighbors, a token at position 0 cannot directly communicate with a token at position 1000. Information must hop through many intermediate windows, risking degradation along the way. Global tokens solve this by designating certain positions as communication hubs that can attend to and be attended by every position in the sequence.

To understand why this matters, think about what attention is really doing. In standard self-attention, every token can directly ask "what information anywhere in the sequence is relevant to me?" That direct query-answer relationship is precisely what gives transformers their power over recurrent models, which had to pass information through a long chain of hidden states. Sliding window attention sacrifices that directness in exchange for efficiency, and global tokens are the mechanism that wins it back, at least partially.

The key insight is that you do not need every pair of tokens to communicate directly. You need every token to be reachable from every other token within a small, fixed number of steps. If you can guarantee that, the model can still reason about the full sequence across multiple layers, even if no single layer has global receptive fields for all tokens. Global tokens provide exactly those shortcuts. Think of them as express stops on a subway network: you might have to take a local train one stop to reach the nearest express station, but from there you can travel anywhere in the system in one more hop.

The mathematical elegance of this approach is that the cost grows linearly rather than quadratically. Adding $g$ global tokens to a sequence of length $n$ introduces $O(g \cdot n)$ additional attention connections on top of the $O(n \cdot w)$ sliding window connections, where $w$ is the window size. As long as $g$ stays small relative to $n$, the overall complexity remains linear. In practice, even a single global token is often sufficient for many tasks, and even using 64 global tokens for a sequence of 4096 tokens adds negligible overhead.

This chapter builds directly on the sliding window and sparse attention mechanisms covered in earlier chapters. If you have been following the progression of efficient attention techniques, you will recognize global tokens as a targeted surgical fix rather than a wholesale redesign. The underlying sliding window machinery is left intact. Global tokens are added on top, requiring only a modest change to the attention mask and the computation that handles global positions separately from local ones.

We will explore how global tokens evolved from BERT's classification token, how different tasks motivate different placement strategies, how learned global tokens discovered by training differ from task-designated ones, and how all of this interacts with computational complexity. By the end, you will understand the mechanics of global tokens and why they represent one of the cleanest solutions to the long-range problem in efficient transformers.