BERT Architecture › Bidirectional Attention Patterns
Language AI HandbookAll chapters BERT Architecture › Bidirectional Attention PatternsOne section · Sections in this chapter5 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
BERT's attention is bidirectional: every token can attend to every other token in the sequence. This contrasts sharply with the causal (left-to-right) attention used in GPT and other autoregressive models. The difference changes which patterns BERT can learn.
To appreciate the difference concretely, consider the sentence "The animal didn't cross the street because it was too tired." A language model reading left to right must decide what "it" refers to based only on "The animal didn't cross the street because." The causal model might correctly infer that "it" refers to "animal" based on the prior context, but it is working against itself: the word "tired" that resolves any ambiguity appears after "it" and is invisible during the processing of "it". BERT, by contrast, has access to the entire sentence simultaneously when computing the representation of "it", including the word "tired" that strongly suggests "animal" is the antecedent. This access to both preceding and following context is what makes BERT dramatically better at coreference resolution, semantic role labeling, and other tasks that require integrating distant context.
The bidirectionality also enables more powerful contextual disambiguation. The word "bank" means different things in "river bank" and "bank account". A left-to-right model processing "bank account" knows the word before "bank" is "the" and perhaps several earlier words about finance. A bidirectional model processing "bank" in either phrase sees the disambiguating word on both sides, making disambiguation significantly easier. Research on probing BERT's representations has confirmed that BERT develops rich, context-sensitive word senses that outperform previous static word embedding approaches by a wide margin.
Visualizing Attention
Let's examine what attention patterns look like in practice. We'll create sample attention weights and visualize how different heads might specialize:
Research on BERT's attention patterns reveals consistent specialization:
- Early layers tend to attend broadly, with some heads focusing on the [CLS] token
- Middle layers develop syntactic patterns, with heads tracking subject-verb relationships, dependency arcs, and constituent boundaries
- Later layers show more semantic attention, with heads focusing on semantically related tokens
This layered specialization is not explicitly programmed. It emerges from pretraining on the MLM and NSP objectives. The fact that syntactic patterns appear in middle layers and semantic patterns in later layers mirrors how linguists think about the relationship between syntax and semantics: syntax provides the structural scaffolding on which semantic interpretation is built. BERT appears to recapitulate this hierarchy through gradient descent alone, which is one of the most striking findings from early BERT interpretability research.
The specialization of individual heads within layers is also notable. Clark et al. (2019) analyzed BERT-Base's 144 attention heads and found that different heads specialize in different syntactic relationships. One head tracks direct objects, another tracks possessives, another tracks coreference links. Some heads have a "previous token" pattern, attending primarily to the immediately preceding token. Others have a "first token" pattern, routing most attention to [CLS]. This suggests that multi-head attention decomposes contextual processing into simpler, specialized operations.
Attention Mask for Padding
When processing batches of variable-length sequences, BERT uses attention masks to prevent attending to padding tokens. The mask is applied as a large negative value before softmax, effectively zeroing out attention to padded positions:
def create_attention_mask(
input_ids: torch.Tensor, pad_token_id: int = 0
) -> torch.Tensor:
"""
Create attention mask for padded sequences.
Args:
input_ids: Token indices, shape (batch, seq_len)
pad_token_id: ID of the padding token
Returns:
Attention mask, shape (batch, 1, 1, seq_len) for broadcasting
"""
# 1 for real tokens, 0 for padding
mask = (input_ids != pad_token_id).float()
# Reshape for broadcasting with attention scores (batch, heads, seq, seq)
mask = mask.unsqueeze(1).unsqueeze(2)
# Convert to additive mask: 0 for attend, -inf for ignore
mask = (1.0 - mask) * -10000.0
return maskInput tokens (0 = padding): tensor([[ 101, 7592, 2088, 102, 0, 0, 0, 0]]) Attention mask shape: torch.Size([1, 1, 1, 8]) Mask values (0 = attend, -10000 = ignore): tensor([ -0., -0., -0., -0., -10000., -10000., -10000., -10000.])
The large negative value (-10000) becomes approximately zero after softmax. This ensures padded positions receive no attention weight.