Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Global Tokens › Global Token Count

Michael BrenndoerferUpdated June 23, 20256 min read

Language AI HandbookAll chapters

Part 17 of 67Efficient Attention

Chapter 120Global Tokens › Global Token Count

Reading section 6 of 15 · Global Token Count · 6 minSection 6 of 15 · 6 min

Section 6 of 15 · 6 min read

Global Tokens › Global Token CountOne section · Sections in this chapter6 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

How many global tokens do you need? This depends on the task and sequence length. Too few global tokens create an information bottleneck. Too many defeat the purpose of sparse attention by reintroducing quadratic interactions.

The right way to think about this trade-off is in terms of information-theoretic capacity. A single global token with embedding dimension $d$ has at most $d$ real-valued numbers to represent the global context of a sequence that may contain thousands of tokens. For a simple classification task where the label depends on one or two facts from the document, a single global token may be sufficient: it just needs to encode those facts and suppress everything else. For a complex task like multi-hop question answering, where the answer requires combining three or four facts distributed across the document, a single global token becomes a bottleneck. Each fact competes for the limited representational capacity, and some may be lost.

The operational rule of thumb is to start with one global token and add more only when task-specific evaluation metrics show that performance has plateaued. A useful diagnostic is to inspect the attention patterns of the global token after training: if it shows very diffuse, nearly uniform attention over the entire sequence, it is probably overloaded and struggling to concentrate on the most important information. If it shows sharply peaked attention at a few positions, it may have sufficient capacity and adding more tokens may not help much.

For classification tasks with a single CLS token, one global token often suffices. The CLS token aggregates a summary representation, and the classification head only needs this single vector.

For tasks requiring fine-grained outputs, such as question answering, named entity recognition, or span extraction, more global tokens help. Making all question tokens global in QA provides richer context for every answer span candidate. Each question token carries part of the question's semantics, so different context tokens can attend to whichever question token is most relevant to their local content.

In[11]:

Code

def compute_attention_complexity(seq_len, window_size, num_global):
    """
    Compute number of attention connections for different configurations.
    """
    # Full attention: n^2 connections
    full = seq_len * seq_len

    # Sliding window: approximately n * window_size
    local = seq_len * window_size

    # Global token connections: each global attends to all (n)
    # and is attended by all (n), minus overlap
    # Approximately: 2 * num_global * seq_len
    global_connections = 2 * num_global * seq_len - num_global * num_global

    # Total sparse: local + global (with some overlap correction)
    sparse = local + global_connections

    return {
        "full_attention": full,
        "local_only": local,
        "local_plus_global": sparse,
        "savings_percent": 100 * (1 - sparse / full),
    }


# Test with different global token counts
seq_len = 4096
window_size = 512

results = []
for num_global in [1, 4, 16, 64]:
    complexity = compute_attention_complexity(seq_len, window_size, num_global)
    results.append(
        {
            "num_global": num_global,
            "connections": complexity["local_plus_global"],
            "savings": complexity["savings_percent"],
        }
    )

Out[12]:

Visualization

Bar chart showing attention connections for 1, 4, 16, and 64 global tokens, all far below the full attention baseline. — Total attention connections for 1, 4, 16, and 64 global tokens in a 4096-token sequence with window size 512. All configurations remain far below full attention (red dashed line at 16.8M connections). This shows that global token overhead is negligible relative to the quadratic baseline.

Bar chart showing savings percentage above 84% for all global token counts tested. — Efficiency retained (percentage savings over full attention) as global token count increases from 1 to 64. Even with 64 global tokens, over 84% of the full-attention connections are eliminated, showing that the linear cost of global tokens does not substantially erode the efficiency gains of sparse attention.

Even with 64 global tokens, we maintain over 84% savings compared to full attention. The marginal cost of additional global tokens is linear in sequence length, not quadratic. This means you can afford to be generous with global tokens for complex tasks without sacrificing efficiency.

To understand why this holds, recall that the total number of connections in global-local attention is approximately:

C(n, w, g) = n \cdot w + 2 \cdot g \cdot n

where:

$n$ : sequence length
$w$ : window size
$g$ : number of global tokens

The first term, $n \cdot w$ , comes from the sliding window: each of the $n$ positions attends to $w$ neighbors. The second term, $2 \cdot g \cdot n$ , comes from global tokens: each global token attends to all $n$ positions (adding $g \cdot n$ connections) and each of the $n$ positions attends to all $g$ global tokens (adding another $g \cdot n$ connections). Both terms are linear in $n$ , while full attention costs $n^2$ . So the savings relative to full attention are:

\text{savings} = 1 - \frac{n \cdot w + 2 \cdot g \cdot n}{n^2} = 1 - \frac{w + 2g}{n}

For $n = 4096$ , $w = 512$ , and $g = 64$ : savings $= 1 - (512 + 128)/4096 \approx 84\%$ . As $n$ grows, the savings percentage approaches 100% for any fixed $w$ and $g$ .