Global Tokens › CLS Token as Global Attention
Language AI HandbookAll chapters Global Tokens › CLS Token as Global AttentionOne section · Sections in this chapter3 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The idea of global tokens predates efficient transformers. BERT's [CLS] token, placed at the start of every sequence, was designed to aggregate sentence-level information. During fine-tuning, the [CLS] representation feeds into classification heads because it has learned to summarize the entire input.
The [CLS] token was introduced in the original BERT paper (Devlin et al., 2018) primarily as a bookkeeping convenience. Because BERT was trained on pairs of sentences, the model needed a position whose representation could be used to classify whether the two sentences were consecutive in the original text (the next sentence prediction task). The [CLS] token, attending to both sentences through standard full self-attention, naturally aggregated information from both halves of the input. Researchers later discovered that this aggregation property made it ideal as a sentence embedding: the final-layer [CLS] representation often captured the overall semantic content of the input well enough to drive downstream classification tasks with a simple linear layer on top. Longformer's designers, faced with the problem of extending BERT to long documents, recognized that the [CLS] token's existing role as a sequence-level aggregator mapped perfectly onto the global attention hub concept. Rather than inventing a new mechanism, they preserved the familiar token but gave it the architectural privilege it had always semantically deserved.
Longformer and BigBird formalized this intuition by giving the [CLS] token bidirectional global attention. In standard BERT, [CLS] only receives information through the normal attention mechanism. In Longformer, [CLS] can attend to every token and every token can attend to [CLS].
The distinction between unidirectional and bidirectional global attention is worth dwelling on. In standard BERT's self-attention, [CLS] does attend to all tokens in the sequence, because BERT uses full quadratic attention. What changes in Longformer is the context: when using sliding window attention for all other tokens, [CLS] would normally only see a small local window if treated like any other position. Making it globally attending means [CLS] retains its BERT-like property of seeing the full sequence even in a sparse attention framework. The bidirectional aspect, that every token also attends back to [CLS], is what maps it from a mere aggregator into a true communication hub. Without bidirectionality, local tokens could push information to [CLS] but could not pull information back from the global context it has accumulated.
Think of it this way: a non-bidirectional [CLS] is like a town bulletin board where everyone can post notices, but nobody reads them. A bidirectional global [CLS] is like a town hall meeting where everyone speaks and everyone listens, concentrated into a single representative who then broadcasts the collective understanding back to the whole community.
def create_longformer_attention_mask(seq_len, window_size, global_positions):
"""
Create an attention mask for Longformer-style attention.
Returns:
mask: Boolean array where True means position i can attend to position j
"""
mask = np.zeros((seq_len, seq_len), dtype=bool)
# Local window attention for all positions
for i in range(seq_len):
start = max(0, i - window_size // 2)
end = min(seq_len, i + window_size // 2 + 1)
mask[i, start:end] = True
# Global positions can attend to everything and be attended by everything
for g in global_positions:
mask[g, :] = True # Global attends to all
mask[:, g] = True # All attend to global
return mask
# Example: 16 tokens, window of 4, CLS token at position 0
seq_len = 16
window_size = 4
global_positions = [0] # CLS token
mask = create_longformer_attention_mask(seq_len, window_size, global_positions)
The attention mask reveals the structure clearly. The cross pattern emanating from position 0 shows the CLS token's global connectivity. The diagonal band represents local window attention for all other positions. This combination ensures that every token is at most 2 hops from any other token: local token to CLS, then CLS to distant local token.
In practice, this mask is sparse enough to provide major computational savings while being connected enough to preserve the essential long-range reasoning capabilities. The fraction of enabled attention connections in this 16-token example is small, and for realistic sequences of thousands of tokens, the fraction enabled by the global token grows even smaller relative to the baseline of full attention.