Longformer › Key Parameters

When working with Longformer, the following parameters have the greatest impact on model behavior and performance:

• window_size: Controls how many neighboring tokens each position can attend to. Larger windows capture more context but increase memory usage linearly. The default of 512 works well for most document tasks, but you might reduce it to 256 for memory-constrained environments or increase it for tasks requiring broader local context.

• global_attention_mask: A binary tensor indicating which tokens have global attention. Set to 1 for tokens that need to see the entire sequence (e.g., [CLS] for classification, question tokens for QA). Keep the number of global tokens small to maintain linear complexity.

• max_length: Maximum sequence length the model can process. Longformer-base supports 4,096 tokens by default. Longer sequences require more memory and compute, but the linear scaling makes lengths up to 16,384 feasible.

• attention_mode (in custom implementations): Determines whether to use sliding window only, global only, or the hybrid pattern. The hybrid pattern is standard for most tasks.

When fine-tuning Longformer, pay special attention to the global attention configuration. The pretrained model learns both local and global projection matrices, so changing which tokens receive global attention during fine-tuning is straightforward. However, adding too many global tokens can negate the efficiency benefits that make Longformer attractive for long documents.

One aspect of the window size parameter that is easy to overlook is its relationship to the number of transformer layers. A model with 12 layers and window size 512 has an effective receptive field of $12 \times 512 = 6{,}144$ positions, comfortably covering a 4,096-token sequence even without any global tokens. However, a model with only 6 layers and window size 256 has an effective receptive field of just 1,536 positions, meaning that tokens near the end of a 4,096-token document would have a strongly impoverished view of the document's beginning even after all layers. When choosing window size, factor in the number of layers in your model and the maximum sequence length you intend to process. The rule of thumb is that the product of layers times window size should comfortably exceed the maximum sequence length for purely local tasks. If it does not, you need more global tokens to bridge the remaining gap.

The attention_window parameter in Hugging Face's Longformer implementation allows you to specify a different window size per layer. This is an advanced configuration used in some research work where lower layers benefit from smaller windows (capturing syntactic structure) while higher layers use larger windows (capturing broader semantic context). In most practical applications, a uniform window size across all layers performs well and is much simpler to reason about.