Longformer › Summary

Longformer tackles the quadratic attention bottleneck through an elegant combination of local and global attention patterns. The key ideas are:

• Sliding window attention reduces per-token complexity from $O(n)$ to $O(w)$, where $w$ is the window size. Most tokens only need local context to understand their meaning.

• Global attention preserves the ability to aggregate information across the entire sequence. By designating specific tokens as global, you maintain long-range dependencies where they matter most.

• Separate projections for local and global attention allow the model to learn distinct representations for different types of context aggregation.

• Linear complexity enables processing sequences of 4,096 tokens or more on hardware that would fail with standard attention at 512 tokens.

The practical impact is substantial. Research papers, legal documents, and book chapters that were previously too long for transformer processing are now accessible. Tasks requiring reasoning across thousands of tokens become feasible without the memory explosion of full attention.

Longformer represents a broader trend in efficient attention: recognizing that the dense attention pattern of standard transformers is often more than necessary. By designing sparse patterns that match the actual information flow needed for a task, we can dramatically reduce computational costs while maintaining model quality.

When you use Longformer, the most important decisions you make are not about the architecture itself but about how you configure the global attention mask. The model architecture is fixed after pretraining, but the global attention pattern is a task-specific parameter you control at inference time. Investing time in understanding which tokens need full-sequence access for your task will pay dividends in both performance and efficiency. Use the [CLS] token for classification, the question tokens for question answering, and entity or structural markers for tasks requiring cross-document reasoning.

Looking ahead, the ideas in Longformer influenced a wave of subsequent work on efficient attention for long documents. BigBird extended the sparse attention idea by adding random attention alongside the local and global patterns. This provides theoretical guarantees that the resulting attention graph is a connected expander with favorable information-mixing properties. Hierarchical approaches like Longformer Encoder-Decoder (LED) adapted the Longformer attention mechanism for generation tasks like summarization, where the decoder must attend to a long encoded document. The design choices made in Longformer, particularly the task-specific global attention configuration and the separate projection matrices, became templates that later efficient attention models either adopted or explicitly chose to deviate from. Understanding Longformer deeply gives you the vocabulary to evaluate these successor models and understand why they make the choices they do.