Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

BigBird › The BigBird Attention Pattern

Michael BrenndoerferUpdated June 24, 202511 min read

Language AI HandbookAll chapters

Part 17 of 67Efficient Attention

Chapter 122BigBird › The BigBird Attention Pattern

Reading section 2 of 12 · The BigBird Attention Pattern · 11 minSection 2 of 12 · 11 min

Section 2 of 12 · 11 min read

BigBird › The BigBird Attention PatternOne section · Sections in this chapter2 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To understand BigBird, we need to ask a fundamental question: what does each token need to see? Full self-attention assumes every token might need information from every other token, leading to the $O(n^2)$ bottleneck. But this assumption is wasteful. In practice, tokens draw information from three distinct sources: their immediate neighbors (for local syntax and semantics), a few special positions that summarize global context, and distant positions that happen to be relevant. BigBird formalizes this intuition into three complementary attention components.

BigBird Attention

BigBird attention is a sparse attention mechanism that combines local sliding window attention, global tokens, and random attention connections. Each token attends to: (1) its nearby neighbors within a fixed window, (2) a small set of global tokens that see the entire sequence, and (3) a random subset of other tokens. This pattern achieves linear complexity while maintaining strong theoretical guarantees.

The power of BigBird comes from how the three components complement each other. Local attention captures the dense, predictable dependencies between adjacent words. Global tokens create information highways that span the entire sequence. And random connections provide the theoretical backbone that guarantees the sparse pattern can approximate any full attention computation.

Let's examine each component in detail, building up to the complete attention pattern.

Component 1: Sliding Window AttentionLink Copied

The first component addresses the most common type of dependency in natural language: relationships between nearby tokens. Consider the sentence "The quick brown fox jumps over the lazy dog." Understanding "jumps" requires knowing its subject ("fox"), which is just two positions away. Understanding "lazy" requires knowing what it modifies ("dog"), which is adjacent. These local dependencies are pervasive in language.

Sliding window attention captures these patterns efficiently. For a given query position $i$ , the set of key positions it can attend to is defined as:

\text{LocalNeighbors}(i) = \{j : \max(0, i - w) \leq j \leq \min(n-1, i + w)\}

where:

$i$ : the query position (the token asking "what should I attend to?")
$j$ : candidate key positions that token $i$ might attend to
$w$ : the one-sided window size (number of neighbors on each side)
$n$ : the total sequence length

This formulation means each token sees at most $2w + 1$ neighbors: $w$ positions to the left, $w$ positions to the right, and itself. Tokens near the sequence boundaries see fewer neighbors due to the $\max$ and $\min$ clipping.

This local attention captures the syntactic and semantic relationships that typically exist between nearby words. In natural language, adjacent tokens often have strong dependencies: subjects precede verbs, adjectives precede nouns, and pronouns refer to nearby antecedents. The sliding window efficiently captures these patterns with just $O(n \cdot w)$ attention operations, since each of the $n$ tokens attends to at most $2w + 1$ positions.

Component 2: Global TokensLink Copied

Local attention alone has a critical limitation: information can only propagate a fixed distance per layer. If a pronoun at position 500 refers to an entity at position 50, and your window size is 64, you would need at least $\lceil 450/64 \rceil = 8$ layers just to connect these positions. This is inefficient and forces deep architectures even for simple long-range dependencies.

Global tokens solve this by creating information hubs that can see and be seen by every position in the sequence. Think of them as central routers in a network: instead of messages hopping through many intermediate nodes, they can travel through a central hub in just two hops (source → hub → destination).

BigBird supports two configurations for global tokens:

BigBird-ITC (Internal Transformer Construction): Existing tokens (like [CLS] or sentence-start tokens) are designated as global. This reuses tokens that already have semantic meaning.
BigBird-ETC (Extended Transformer Construction): Additional learnable tokens are prepended to the sequence. These tokens have no predetermined meaning but learn to aggregate and broadcast information.

The global component adds $O(n \cdot g)$ attention operations, where $g$ is the number of global tokens and $n$ is the sequence length. Since $g$ is typically small (2-8 tokens), this remains linear in sequence length. The total attention operations from global tokens is $2 \cdot n \cdot g$ : each global token attends to all $n$ positions, and each of the $n$ positions attends to the $g$ global tokens.

Component 3: Random AttentionLink Copied

At this point, you might think we have everything we need: local attention for nearby dependencies, global tokens for long-range communication. Longformer stops here, and it works well in practice. So why does BigBird add a third component?

The answer lies in a subtle but important limitation of the local-plus-global pattern. Global tokens provide shortcuts, but they create a bottleneck: all long-range information must flow through the same small set of hub tokens. If different parts of the document need to exchange information simultaneously, the global tokens can become saturated. More importantly, the local-plus-global pattern lacks certain theoretical properties that guarantee it can approximate arbitrary attention patterns.

Random attention addresses both issues. Each token attends to a fixed number $r$ of randomly selected positions, creating shortcuts that bypass both the local neighborhood and the global hubs. These connections might seem arbitrary, but they have deep theoretical significance rooted in graph theory.

The key insight comes from the study of small-world networks: even a sparse random graph has small diameter. Adding just a few random edges to a regular lattice (our sliding window) dramatically reduces the maximum distance between any two nodes. This means information can flow between any two positions in logarithmic rather than linear steps, without requiring all traffic to route through global tokens.

In[3]:

Code

import numpy as np


def create_bigbird_attention_pattern(
    seq_len, window_size, num_global, num_random, seed=42
):
    """
    Create a BigBird attention pattern combining local, global, and random attention.

    Args:
        seq_len: Length of the sequence
        window_size: Size of the sliding window (one-sided)
        num_global: Number of global tokens at the start
        num_random: Number of random attention connections per token
        seed: Random seed for reproducibility

    Returns:
        attention_mask: Binary mask of shape (seq_len, seq_len)
    """
    rng = np.random.default_rng(seed)
    mask = np.zeros((seq_len, seq_len), dtype=np.float32)

    # Component 1: Sliding window attention
    for i in range(seq_len):
        start = max(0, i - window_size)
        end = min(seq_len, i + window_size + 1)
        mask[i, start:end] = 1.0

    # Component 2: Global tokens (first num_global tokens)
    mask[:num_global, :] = 1.0  # Global tokens attend to all
    mask[:, :num_global] = 1.0  # All tokens attend to global

    # Component 3: Random attention
    for i in range(num_global, seq_len):
        # Get positions not already attended to
        local_start = max(0, i - window_size)
        local_end = min(seq_len, i + window_size + 1)

        # Candidate positions for random attention
        candidates = []
        for j in range(num_global, seq_len):
            if j < local_start or j >= local_end:
                candidates.append(j)

        # Sample random positions
        if len(candidates) > 0:
            num_to_sample = min(num_random, len(candidates))
            random_positions = rng.choice(
                candidates, size=num_to_sample, replace=False
            )
            mask[i, random_positions] = 1.0

    return mask

In[4]:

Code

# Create a BigBird attention pattern for visualization
seq_len = 64
window_size = 3
num_global = 2
num_random = 3

bigbird_mask = create_bigbird_attention_pattern(
    seq_len, window_size, num_global, num_random
)

# Calculate sparsity
total_positions = seq_len * seq_len
attended_positions = bigbird_mask.sum()
sparsity = 1 - (attended_positions / total_positions)

Out[5]:

Console

BigBird attention pattern statistics:
  Sequence length: 64
  Window size: 3 (each side)
  Global tokens: 2
  Random connections per token: 3

  Total positions: 4,096
  Attended positions: 860
  Sparsity: 79.0%

With a 64-token sequence and these parameters, BigBird attends to only about 15% of possible positions. This sparse pattern dramatically reduces memory usage while preserving the model's ability to capture both local and long-range dependencies. As sequence length grows, the sparsity increases further, making the efficiency gains even more pronounced.

The sparsity calculation reveals the efficiency gain. Instead of computing all $n^2$ attention scores, BigBird computes approximately:

\text{TotalConnections} \approx n \cdot (2w + g + r)

where:

$n$ : the sequence length
$w$ : the one-sided window size (so $2w$ accounts for neighbors on both sides)
$g$ : the number of global tokens
$r$ : the number of random connections per token

This is linear in sequence length when $w$ , $g$ , and $r$ are constants. The key insight is that the number of attended positions per token is bounded by a constant, making the total computation scale as $O(n)$ rather than $O(n^2)$ .

Out[6]:

Visualization

Heatmap showing BigBird sparse attention pattern with diagonal band, global token rows and columns, and random scattered connections. — BigBird attention pattern combining three components. Yellow squares near the diagonal represent sliding window attention. The vertical and horizontal yellow bars at the top and left show global tokens that see and are seen by the entire sequence. Scattered yellow points throughout the matrix represent random attention connections.

The visualization shows the three distinct components of BigBird attention. The diagonal band represents the sliding window, capturing local dependencies. The rows and columns at the edges correspond to global tokens. The scattered points outside the diagonal band are random attention connections, on which the model's theoretical properties depend.

Out[7]:

Visualization

Heatmap with three colors showing local attention in blue along the diagonal, global tokens in orange at edges, and random connections as green scattered points. — BigBird attention pattern with components color-coded. Blue represents sliding window (local) attention forming the diagonal band. Orange shows global tokens that attend to and are attended by all positions. Green dots mark random attention connections that create shortcuts across the sequence. This decomposition reveals how each component contributes to the overall pattern.

The color-coded visualization makes each component's contribution clear. The blue diagonal band captures the sliding window pattern. This ensures every token sees its immediate neighbors. The orange bars at the top and left edges represent the global tokens, which can communicate with any position in the sequence. The scattered green dots are the random connections, sparse but strategically placed to reduce the graph diameter.