Longformer › Configuring Global Attention for Different Tasks
Language AI HandbookAll chapters Longformer › Configuring Global Attention for Different TasksOne section · Sections in this chapter10 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The power of Longformer comes from its flexibility in configuring global attention. Different tasks benefit from different global attention patterns. This configurability is not a minor implementation detail; it is a fundamental design philosophy. Rather than hardcoding a fixed sparse attention structure, Longformer treats the global attention pattern as a task-specific hyperparameter that the practitioner controls. Getting this right has a significant impact on downstream performance.
The guiding principle for choosing global tokens is to ask: which tokens in my input absolutely must have access to the entire document to perform their role in the task? These are the tokens that should receive global attention. For tasks where the answer is "no specific tokens need full-sequence access," you can rely on sliding window attention alone and use Longformer as a more efficient BERT replacement. For tasks where the answer is "many tokens need full-sequence access," the computational advantages of Longformer diminish as global attention becomes more prevalent. The sweet spot is tasks where a small number of designated tokens (typically 1 to 20) need global reach while the majority of the sequence processes locally.
Document Classification
For classification, you typically only need global attention on the [CLS] token:
def create_classification_global_mask(input_ids, cls_token_id):
"""Global attention only on [CLS] token."""
global_mask = torch.zeros_like(input_ids)
global_mask[:, 0] = 1 # [CLS] is always at position 0
return global_mask
# Example usage
cls_mask = create_classification_global_mask(
inputs["input_ids"], tokenizer.cls_token_id
)Question Answering
For question answering, the question tokens need to see the entire context to find the answer:
def create_qa_global_mask(input_ids, question_end_position):
"""Global attention on question tokens (positions 0 to question_end)."""
global_mask = torch.zeros_like(input_ids)
global_mask[:, :question_end_position] = 1
return global_mask
# Example: question ends at position 20
qa_mask = create_qa_global_mask(inputs["input_ids"], question_end_position=20)Global attention configurations: Classification: 1 global token ([CLS]) Question Answering: 20 global tokens (question) Memory comparison at 4,096 tokens: Full attention: 16,777,216 attention weights Classification Longformer: ~2,105,344 weights (12.5%) QA Longformer: ~2,260,992 weights (13.5%)
The difference in global token count between classification and QA tasks has minimal impact on memory efficiency. Classification uses just 12.5% of full attention memory, while QA with 20 global question tokens uses only 13.5%. Both represent massive savings compared to the 16.7 million attention weights required by full attention at 4,096 tokens.
Named Entity Recognition
For token-level tasks like NER, you might want global attention on sentence boundaries or special delimiter tokens:
def create_ner_global_mask(input_ids, sep_token_id):
"""Global attention on [SEP] tokens (sentence boundaries)."""
global_mask = (input_ids == sep_token_id).long()
# Also include [CLS]
global_mask[:, 0] = 1
return global_maskThe flexibility to configure global attention per task is one of Longformer's key advantages. You control exactly which tokens have global reach, optimizing the trade-off between computational cost and model capability.
Beyond the three task patterns shown here, there are more advanced configurations worth knowing. For multi-label document classification where you predict multiple categories simultaneously, you might use one global token per category label, with each global token learning to aggregate evidence for its associated category from the full document. For structured prediction tasks like relation extraction, you might assign global attention to entity spans so that each entity can directly compare itself with every other entity in the document, regardless of distance. For dialogue systems processing multi-turn conversation histories, you might use global attention on turn separators to allow the model to track conversational state transitions across arbitrarily long histories.
One practical caution: if you add global attention to tokens that do not semantically need it, you typically see minor performance improvements while increasing memory usage substantially. The model can always choose to assign near-zero attention weights to irrelevant global tokens, so the expressiveness never hurts performance, but the memory cost of storing and computing the additional full-sequence attention rows is real. Always benchmark the minimal global attention configuration before adding more global tokens.