Longformer › Longformer

Standard transformer attention scales quadratically with sequence length, making it prohibitively expensive for documents that span thousands of tokens. If you want to process an entire research paper, a legal contract, or a book chapter, vanilla attention becomes a memory and compute bottleneck long before you reach the interesting parts of the text.

To see why this is a problem in practice, consider what happens when you increase the sequence length from 512 tokens to 4,096 tokens. With full attention, each of the 4,096 tokens must compute an attention score against every other token, producing over 16 million attention weights per layer. On a GPU with 16 GB of memory, this easily saturates available resources before you even add the residual connections, feedforward layers, and batch dimension. Standard BERT was trained with 512-token sequences for exactly this reason: it was the longest sequence that remained feasible given typical hardware constraints in 2018.

This limitation matters enormously for real-world text. A legal contract averages 10,000 words. A scientific paper runs 8,000 to 12,000 words. A court opinion might span 20,000 words. When you truncate these documents to 512 tokens, you throw away roughly 90% of the content. A model analyzing whether a contract clause conflicts with another clause mentioned 8,000 tokens earlier simply cannot make that connection if both sections are chopped off.

Longformer addresses this by combining two complementary attention patterns: local sliding window attention for capturing nearby context, and global attention for aggregating information across the entire sequence. This hybrid approach reduces complexity from $O(n^2)$ to $O(n)$, where $n$ is the sequence length. In practical terms, doubling the sequence length with standard attention quadruples the cost, but with Longformer it only doubles the cost. The result is a transformer that can handle sequences of 4,096 tokens or more without the memory explosion that would occur with full attention.

The key insight is that the quadratic cost of full attention is largely wasted. Not every pair of tokens needs a direct attention connection. Language has structure: meaning flows locally through sentences and paragraphs, and only a small number of special tokens (like document-level question markers or classification anchors) need to see the entire sequence. By designing an attention pattern that reflects this structure, Longformer recovers efficiency without sacrificing the representational power needed for long-document understanding.