Longformer › Using Longformer from Hugging Face
Language AI HandbookAll chapters Longformer › Using Longformer from Hugging FaceOne section · Sections in this chapter9 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
For practical applications, you'll want to use the optimized Longformer implementation from Hugging Face. The production implementation differs substantially from our educational implementation in one critical way: it uses a dedicated CUDA kernel for sliding window attention that operates on chunked, diagonal-band sparse matrices rather than a Python loop over positions. This kernel achieves attention in time proportional to the number of non-zero entries rather than the full grid, which delivers the speed and memory benefits in practice. Our loop-based implementation demonstrates the correct logic but would not be fast enough for real workloads.
Let's see how to apply the Hugging Face Longformer to a document processing task. The interface closely mirrors standard transformer models, with one additional parameter: the global_attention_mask that specifies which tokens receive global attention.
from transformers import LongformerModel, LongformerTokenizer
# Load pre-trained Longformer
tokenizer = LongformerTokenizer.from_pretrained("allenai/longformer-base-4096")
model = LongformerModel.from_pretrained("allenai/longformer-base-4096")
# Sample long document (concatenated for demonstration)
document = (
"""
Machine learning has transformed how we process and understand text.
Traditional approaches relied on hand-crafted features and statistical methods.
Modern neural networks learn representations directly from data, enabling
unprecedented performance on tasks from translation to summarization.
The transformer architecture, introduced in 2017, revolutionized the field.
Self-attention allows models to capture relationships between any positions
in a sequence, overcoming the sequential limitations of recurrent networks.
However, the quadratic complexity of attention limited sequence lengths.
Longformer addresses this limitation through a hybrid attention pattern.
By combining local sliding window attention with strategic global attention,
it achieves linear complexity while maintaining the ability to model
long-range dependencies essential for document understanding.
"""
* 10
) # Repeat to create a longer document# Tokenize the document
inputs = tokenizer(
document,
return_tensors="pt",
max_length=4096,
truncation=True,
padding="max_length",
)
# Create global attention mask - set first token ([CLS]) to have global attention
global_attention_mask = torch.zeros_like(inputs["input_ids"])
global_attention_mask[:, 0] = 1 # [CLS] token has global attention
# Forward pass
with torch.no_grad():
outputs = model(
input_ids=inputs["input_ids"],
attention_mask=inputs["attention_mask"],
global_attention_mask=global_attention_mask,
)
# Get the [CLS] token representation for classification
cls_representation = outputs.last_hidden_state[:, 0, :]Document tokens: 1,522 Max sequence length: 4,096 [CLS] representation dimension: 768 Global attention positions: 1 token(s)
The document contains over 1,000 tokens, padded to Longformer's maximum sequence length of 4,096. Despite processing this long sequence, only a single token (the [CLS] token at position 0) has global attention. This token's 768-dimensional representation now contains information aggregated from the entire document, enabling downstream tasks like classification or similarity computation without the quadratic memory cost of full attention.