BigBird › Key Parameters

When configuring BigBird attention, several parameters control the trade-off between efficiency and model expressiveness:

• window_size: The one-sided sliding window size, determining how many neighbors each token attends to on each side. Larger windows capture more local context but increase computation. Typical values range from 64 to 256. For most NLP tasks, 128 provides a good balance.

• num_global: The number of global tokens that attend to and are attended by all other tokens. These serve as information hubs for long-range communication. Common choices are 2-8 tokens. Using the [CLS] token plus sentence-start markers is a practical approach.

• num_random: The number of random attention connections per non-global token. BigBird's theoretical guarantees depend on this parameter. Values of 3-5 are typically sufficient to achieve logarithmic graph diameter. Increasing beyond 5 has diminishing returns.

• block_size: In optimized implementations, attention is computed in blocks for efficiency. Common values are 64 or 128. This should divide evenly into your sequence length.

• attention_type: BigBird supports two modes: "original_full" computes full attention for short sequences (faster when $n < 1024$), while "block_sparse" uses the sparse pattern. Most libraries automatically select based on sequence length.

The interplay between these parameters determines the effective receptive field and computational cost. For a sequence of length $n$ with window size $w$, $g$ global tokens, and $r$ random connections, the total attention operations scale as $O(n \cdot (2w + g + r))$, which remains linear as long as these hyperparameters are constants.