Global Tokens › Key Parameters

When implementing global-local attention, several parameters control the trade-off between efficiency and expressiveness:

• window_size: The number of tokens each position can attend to locally. Larger windows capture more local context but increase computation. Typical values range from 256 to 512 for long-document models. The window should be large enough to capture phrase-level dependencies. Setting the window too small forces the model to rely heavily on global tokens for even moderately long-range dependencies, which can overwhelm the global token capacity.

• global_positions: A list of token indices designated as global. Common strategies include:

  • First position only (CLS token) for classification
  • First and last positions (CLS + SEP) for sequence-pair tasks
  • All question tokens for extractive QA
  • Every k-th position for uniform coverage

• num_global: The count of global tokens. More global tokens reduce the information bottleneck but add $O(g \cdot n)$ connections where $g$ is the global count and $n$ is sequence length. Start with 1-4 global tokens and increase if task performance plateaus.

• d_k: The dimension used for scaling attention scores. Standard practice sets this equal to the model's head dimension (typically 64). Scaling by $\sqrt{d_k}$ keeps attention logits in a stable range regardless of dimensionality.

The relationship between window size and number of global tokens defines the fundamental trade-off in global-local attention design. A large window with few global tokens provides rich local context and minimal global routing capacity. A small window with many global tokens provides tight local focus and rich global communication bandwidth. Most practical architectures land somewhere in the middle: a moderately large window (256-512) with a small number of global tokens (1-8), sized to cover the task's specific global information requirements.