Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

BigBird › Comparing BigBird and Longformer

Michael BrenndoerferUpdated June 24, 20254 min read

Language AI HandbookAll chapters

Part 17 of 67Efficient Attention

Chapter 122BigBird › Comparing BigBird and Longformer

Reading section 5 of 12 · Comparing BigBird and Longformer · 4 minSection 5 of 12 · 4 min

Section 5 of 12 · 4 min read

BigBird › Comparing BigBird and LongformerOne section · Sections in this chapter5 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

BigBird and Longformer are both sparse attention models designed for long documents, but they differ in their use of random attention. Let's compare their attention patterns and understand the trade-offs.

In[15]:

Code

def create_longformer_attention_pattern(seq_len, window_size, num_global):
    """
    Create a Longformer attention pattern (local + global, no random).
    """
    mask = np.zeros((seq_len, seq_len), dtype=np.float32)

    # Local sliding window
    for i in range(seq_len):
        start = max(0, i - window_size)
        end = min(seq_len, i + window_size + 1)
        mask[i, start:end] = 1.0

    # Global tokens
    mask[:num_global, :] = 1.0
    mask[:, :num_global] = 1.0

    return mask


# Create both patterns for comparison
seq_len_compare = 64
window = 3
num_global = 2
num_random = 3

longformer_mask = create_longformer_attention_pattern(
    seq_len_compare, window, num_global
)
bigbird_mask_compare = create_bigbird_attention_pattern(
    seq_len_compare, window, num_global, num_random, seed=42
)

# Calculate differences
difference = bigbird_mask_compare - longformer_mask
random_additions = (difference > 0).sum()

Out[16]:

Console

Pattern comparison (n=64):
  Longformer attended positions: 674
  BigBird attended positions:    860
  Random attention additions:    186

Overhead from random attention: 27.6%

The random attention component adds only about 30% more connections compared to Longformer. This modest overhead is the price for BigBird's stronger theoretical guarantees. In practice, both patterns achieve similar downstream performance on most tasks, but BigBird's random connections ensure the model can theoretically approximate any full-attention computation.

Out[17]:

Visualization

Heatmap highlighting the random attention positions that BigBird adds beyond Longformer's local and global attention. — Random attention additions in BigBird compared to Longformer. The heatmap shows positions where BigBird adds random connections that Longformer lacks. These scattered connections, while sparse, are essential for reducing graph diameter and providing theoretical guarantees.

The visualization highlights exactly where BigBird differs from Longformer. Each green point represents a random attention connection, the shortcuts that reduce graph diameter from linear to logarithmic. Note how these connections are distributed across the entire sequence. This ensures that no region is isolated.

Out[18]:

Visualization

Heatmap showing Longformer attention with diagonal band and global tokens. — Longformer attention pattern with sliding window and global tokens. The pattern is entirely deterministic, with attention concentrated along the diagonal and at global token positions.

Heatmap showing BigBird attention with diagonal band, global tokens, and scattered random connections. — BigBird attention pattern adds random connections (visible as scattered points). These seemingly minor additions dramatically reduce the graph diameter and improve theoretical properties.

The visual difference is subtle, as the random connections are sparse. However, the theoretical implications are significant:

Comparison of Longformer and BigBird attention patterns. BigBird's random connections provide theoretical guarantees that Longformer lacks.
Property	Longformer	BigBird
Local attention	Yes	Yes
Global tokens	Yes	Yes
Random attention	No	Yes
Graph diameter	$O(n/w)$	$O(\log n)$
Turing complete	Not proven	Proven
Universal approximator	Not proven	Proven

The graph diameter difference is particularly important: $O(n/w)$ means that for a sequence of 4,096 tokens with window size 64, Longformer needs roughly 64 attention hops to propagate information end-to-end. BigBird with random connections needs only about $\log_2(4096) \approx 12$ hops.

In practice, both models perform well on long-document tasks, and the choice often depends on the specific application. Longformer's deterministic pattern is easier to optimize with custom CUDA kernels, while BigBird's random connections provide stronger theoretical guarantees.