Longformer › The Core Insight: Local + Global Attention

Longformer's key innovation is recognizing that not every token needs to attend to every other token. Most language understanding happens locally. When reading a sentence, the meaning of a word primarily depends on its immediate neighbors. The subject of a sentence matters for its verb, but a word in paragraph 15 rarely affects the interpretation of a word in paragraph 3.

Think of it this way: when you read the word "therefore" in a paragraph, you look left to understand the cause and right to understand the conclusion. You do not simultaneously scan back to the title page or forward to the appendix. Human reading is overwhelmingly local. Longformer formalizes this observation into an attention pattern where each token has a fixed-width spotlight that illuminates only its immediate neighborhood, rather than broadcasting attention to every other token in the document.

However, some tokens need global reach. A question token in question answering must see the entire context to find the answer. A classification token needs to aggregate information from the whole document. Longformer solves this by treating these global tokens differently from the rest of the sequence. Rather than giving all tokens equal capabilities, it partitions them into two classes with different attention behaviors tailored to their roles.

Notice that this design closely mirrors how humans process documents. When reading a research paper, you read each sentence in context with its neighbors. But certain anchoring concepts, like the research question stated in the abstract, remain globally relevant throughout. Longformer's global tokens play the same role as these anchoring concepts: they provide document-level context that every local token can access.

The architecture defines two types of attention:

• Sliding window attention: Each token attends only to tokens within a fixed window around it. With window size $w$, token $i$ attends to positions $[i - w/2, i + w/2]$
• Global attention: Selected tokens attend to all tokens in the sequence, and all tokens attend back to them

This combination allows Longformer to handle documents that would overwhelm standard transformers, while still capturing the relationships that matter for understanding. The elegance of the design is that neither component requires the other to function: you could use sliding window attention alone for tasks with only local dependencies, or use global attention alone for tasks requiring full-sequence reasoning. The hybrid approach earns the best of both worlds at a cost close to the cheaper option.