BigBird › A Worked Example: Long Document Classification
Language AI HandbookAll chapters BigBird › A Worked Example: Long Document ClassificationOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's trace through how BigBird processes a long document classification task:
def simulate_document_processing(doc_length, num_classes=5):
"""
Simulate BigBird processing a long document for classification.
Args:
doc_length: Number of tokens in the document
num_classes: Number of classification categories
Returns:
Dictionary with processing statistics
"""
d_model = 768
window_size = 64
num_global = 8
num_random = 5
# Simulate input embeddings
X = np.random.randn(doc_length, d_model) * 0.1
# Create attention mask
mask = create_bigbird_attention_pattern(
doc_length, window_size, num_global, num_random, seed=42
)
# Calculate attention statistics
attended_per_token = mask.sum(axis=1)
total_attended = mask.sum()
sparsity = 1 - (total_attended / (doc_length * doc_length))
# Memory savings
full_memory_mb = (doc_length * doc_length * 4) / (1024 * 1024) # float32
sparse_memory_mb = (total_attended * 4) / (1024 * 1024)
return {
"doc_length": doc_length,
"avg_attended": attended_per_token.mean(),
"sparsity": sparsity,
"full_memory_mb": full_memory_mb,
"sparse_memory_mb": sparse_memory_mb,
"memory_savings": full_memory_mb / sparse_memory_mb,
}
# Process documents of different lengths
doc_lengths = [1024, 2048, 4096, 8192]
results = [simulate_document_processing(n) for n in doc_lengths]BigBird document processing statistics:
-----------------------------------------------------------------
Doc Length Avg Attend Sparsity Full (MB) Sparse (MB)
-----------------------------------------------------------------
1,024 144.8 85.9% 4.0 0.57
2,048 147.4 92.8% 16.0 1.15
4,096 148.7 96.4% 64.0 2.32
8,192 149.4 98.2% 256.0 4.67The table shows that as document length increases, the sparsity remains roughly constant (determined by window size, global tokens, and random connections), while the memory savings grow dramatically. An 8,192-token document that would require 256 MB for full attention needs only about 5 MB with BigBird's sparse pattern.
The left panel shows that sparsity increases steadily as documents get longer, reaching over 98% for 8K-token documents. The right panel shows the corresponding memory savings: because full attention grows quadratically while BigBird grows linearly, the savings ratio itself grows linearly with sequence length. This means longer documents benefit even more from BigBird's sparse pattern.
