Longformer › Limitations and Trade-offs

Longformer represents a significant advance for long document processing, but it comes with trade-offs you should understand before adopting it.

The sliding window creates an information bottleneck for tokens far from any global token. If important information lies 1,000 positions away from the nearest global token and outside any token's window, the model must rely on the stacking of attention layers to propagate that information. Deep transformer stacks help mitigate this through repeated local interactions that gradually spread information, but the effective receptive field is still limited compared to full attention. For tasks where any token might need to attend to any other token equally, Longformer's sparse pattern might lose important connections.

This bottleneck becomes apparent in tasks involving fine-grained comparisons between distant tokens. Consider a coreference resolution task where "the defendant" in paragraph 1 must be resolved to "John Smith, age 34" in paragraph 6, which is 800 tokens away. Without a global token near both mentions, the model must propagate this connection through many layers of local attention. If the transformer has only 6 layers and the window size is 256, the effective receptive field at layer 6 is only 1,536 tokens, which might be sufficient. But for a 12-layer model processing a 16,000-token document with no global tokens near the relevant mentions, the connection may simply not form reliably.

The separate projection matrices for global attention increase parameter count by approximately 50% for the attention layers. For models with billions of parameters, this overhead is significant. You are trading memory during inference for additional model weights that must be stored and loaded. The global projections also add complexity to fine-tuning: you need to carefully initialize them, and learning dynamics can differ between local and global components.

In practice, the parameter overhead is manageable for most applications. A Longformer-base model has 149 million parameters compared to BERT-base's 110 million, an increase of about 35%. This is small compared to the functional gains for long-document tasks. However, if you are deploying on resource-constrained hardware or trying to maximize performance per parameter, the additional weights are a real cost. One mitigation is to initialize the global projection matrices as copies of the local projection matrices at the start of fine-tuning, so the model begins from a reasonable starting point before the two sets of projections diverge to serve their distinct roles.

Choosing which tokens should have global attention requires task-specific knowledge. For classification, the [CLS] token is an obvious choice. For question answering, the question tokens work well. But for open-ended tasks like summarization or dialogue, the optimal global attention configuration is less clear. Getting this wrong can significantly hurt performance, and finding the right configuration often requires experimentation.

The ambiguity around global token selection is especially challenging when you are applying Longformer to a new domain without existing research to guide you. You may find yourself running ablations to compare single-CLS-global against multi-token-global configurations, or testing whether global attention on domain-specific markers (like section headings in legal documents) improves over CLS-only. This experimentation cost is real and should be factored into project timelines when adopting Longformer for novel tasks.

Longformer's linear complexity assumes the window size and number of global tokens remain constant as sequence length grows. If your task requires global attention on a growing fraction of tokens (say, one global token per paragraph), complexity can approach $O(n^2)$ again. The linear scaling benefit only materializes when global attention is sparse.

There is also a subtle inference-time consideration: while Longformer's memory footprint scales linearly with sequence length, it does not match the raw speed of full attention on shorter sequences. The sparse attention implementation has higher constant-factor overhead compared to the dense matrix multiplication used by full attention, which is heavily optimized in standard linear algebra libraries. For sequences under 1,024 tokens, standard BERT with full attention is often faster in wall-clock time than Longformer with sparse attention. Longformer's efficiency advantages become meaningful starting around 2,048 tokens, and grow substantially beyond that. If your workload is dominated by short documents, Longformer may not be the right tool.

Finally, Longformer was pretrained on relatively long documents from scientific literature, web text, and books. If your target domain involves a very different kind of long-form text, such as code, tabular data embedded in prose, or structured reports with consistent formatting, the pretrained representations may not transfer as effectively as they do for narrative prose. In such cases, domain-adaptive pretraining on your target corpus before fine-tuning can improve results.