Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Encoder Architecture › Bidirectional Self-Attention

Michael BrenndoerferUpdated June 15, 20258 min read

Language AI HandbookAll chapters

Part 16 of 67Transformer Architectures

Chapter 111Encoder Architecture › Bidirectional Self-Attention

Reading section 3 of 13 · Bidirectional Self-Attention · 8 minSection 3 of 13 · 8 min

Section 3 of 13 · 8 min read

Encoder Architecture › Bidirectional Self-AttentionOne section · Sections in this chapter3 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The defining feature of encoders is bidirectional self-attention. Every token can attend to every other token in the sequence, including tokens that appear after it. This contrasts sharply with the causal (masked) attention used in decoders, where each token can only attend to previous tokens.

Consider processing the sentence "The bank approved the loan quickly." When computing the representation for "bank," bidirectional attention sees:

Before: "The"
After: "approved the loan quickly"

The word "loan" in the future context helps disambiguate that "bank" refers to a financial institution rather than a river bank. Causal attention would miss this signal, seeing only "The" before "bank."

In[3]:

Code

import numpy as np

np.random.seed(42)


def softmax(x, axis=-1):
    """Numerically stable softmax."""
    x_max = np.max(x, axis=axis, keepdims=True)
    exp_x = np.exp(x - x_max)
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)


def create_attention_pattern(seq_len, pattern_type="bidirectional"):
    """
    Create attention patterns for visualization.

    Args:
        seq_len: Length of the sequence
        pattern_type: "bidirectional" or "causal"

    Returns:
        Attention mask (0 for allowed, -inf for blocked)
    """
    if pattern_type == "bidirectional":
        # All positions can attend to all positions
        return np.zeros((seq_len, seq_len))
    else:
        # Causal: can only attend to current and previous positions
        mask = np.triu(np.ones((seq_len, seq_len)) * float("-inf"), k=1)
        return mask


# Example sentence
tokens = ["The", "bank", "approved", "the", "loan", "quickly"]
seq_len = len(tokens)

# Create random similarity scores (simulating Q @ K.T)
np.random.seed(42)
raw_scores = np.random.randn(seq_len, seq_len)

# Apply masks
bidirectional_mask = create_attention_pattern(seq_len, "bidirectional")
causal_mask = create_attention_pattern(seq_len, "causal")

# Compute attention weights
bidirectional_weights = softmax(raw_scores + bidirectional_mask, axis=-1)
causal_weights = softmax(raw_scores + causal_mask, axis=-1)

Out[4]:

Visualization

Heatmap showing full attention matrix where all positions attend to all other positions. — Bidirectional attention in encoders. Every token attends to every other token, including future context. When processing 'bank', the model sees 'loan' which helps disambiguate the meaning.

Heatmap showing lower-triangular attention pattern where positions only attend to previous tokens. — Causal attention in decoders. Each token can only attend to itself and previous tokens. 'Bank' cannot see 'loan' and must rely solely on left context.

The visualizations reveal the fundamental difference. In bidirectional attention (left), the "bank" row shows non-zero weights for all six positions, including "loan" and "quickly" that appear later in the sequence. In causal attention (right), the "bank" row shows weights only for "The" and "bank" itself. The future context that would help disambiguate "bank" is completely invisible.

The Attention Mechanism Without MaskingLink Copied

To understand how bidirectional attention works mathematically, we need to build up from a simple question: how does a token decide which other tokens to pay attention to?

The answer involves three learned projections that give each token different "roles" in the attention computation. Think of it as a information retrieval system where tokens can both ask questions and provide answers:

Queries represent what a token is looking for. When computing the representation for "bank," its query vector encodes the question "what information do I need to understand my meaning?"
Keys represent what a token offers. Each token advertises its content through a key vector, saying "this is what I can tell you about."
Values contain the actual information to be gathered. Once attention decides which tokens matter, their value vectors provide the content.

Starting from the input sequence $X \in \mathbb{R}^{n \times d}$ containing $n$ tokens with $d$ -dimensional embeddings, we project into these three spaces:

Q = XW^Q, \quad K = XW^K, \quad V = XW^V

where:

$X$ : the input matrix with one row per token
$W^Q, W^K, W^V \in \mathbb{R}^{d \times d_k}$ : learned projection matrices
$Q, K, V$ : the resulting QKV matrices

With these projections in hand, we measure relevance by computing dot products between queries and keys. A high dot product between query $i$ and key $j$ means token $i$ should attend strongly to token $j$ . We arrange all these dot products into a single matrix multiplication:

\text{Scores} = QK^T

This produces an $n \times n$ matrix where entry $(i, j)$ measures how much token $i$ should attend to token $j$ . Encoders compute all $n^2$ entries, including the upper triangle. Position 2 can attend to position 5, and position 5 can attend to position 2.

Raw dot products can grow large as the dimension $d_k$ increases, which would push softmax into regions with vanishing gradients. To stabilize training, we scale by $\sqrt{d_k}$ :

\text{Scaled Scores} = \frac{QK^T}{\sqrt{d_k}}

The softmax function converts these scores into a probability distribution. For each token (each row), softmax ensures the attention weights across all positions sum to 1:

\text{Attention Weights} = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)

Finally, we use these weights to compute a weighted average of value vectors. If token $i$ assigns weight 0.4 to token $j$ , then 40% of $j$ 's value vector contributes to $i$ 's output:

\text{Output} = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

Putting it all together, the complete attention formula is:

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

where:

$Q = XW^Q$ : query matrix, each row is a token asking "what should I pay attention to?"
$K = XW^K$ : key matrix, each row is a token advertising "this is what I contain"
$V = XW^V$ : value matrix, each row is a token's actual content to be gathered
$d_k$ : dimension of query/key vectors, used for scaling to prevent gradient issues
The softmax normalizes attention weights to sum to 1 for each query position

The important difference from decoder attention: no mask is applied before the softmax. Every query-key pair contributes to the attention weights. This allows information to flow in both directions. This is the simplest form of self-attention, with no architectural constraints on which positions can interact.

In[5]:

Code

def scaled_dot_product_attention(Q, K, V, mask=None):
    """
    Compute scaled dot-product attention.

    Args:
        Q: Query matrix of shape (seq_len, d_k)
        K: Key matrix of shape (seq_len, d_k)
        V: Value matrix of shape (seq_len, d_v)
        mask: Optional mask (0 for allowed, -inf for blocked)

    Returns:
        Output of shape (seq_len, d_v) and attention weights
    """
    d_k = Q.shape[-1]

    # Compute attention scores
    scores = Q @ K.T / np.sqrt(d_k)

    # Apply mask if provided
    if mask is not None:
        scores = scores + mask

    # Convert to probabilities
    weights = softmax(scores, axis=-1)

    # Compute weighted sum of values
    output = weights @ V

    return output, weights


# Demonstrate with a simple example
d_model = 64
d_k = 16

# Random input embeddings
X = np.random.randn(seq_len, d_model)

# Random projections (in practice, these are learned)
W_q = np.random.randn(d_model, d_k) * 0.1
W_k = np.random.randn(d_model, d_k) * 0.1
W_v = np.random.randn(d_model, d_k) * 0.1

Q = X @ W_q
K = X @ W_k
V = X @ W_v

# Bidirectional attention (no mask)
output_bidir, weights_bidir = scaled_dot_product_attention(Q, K, V, mask=None)

# Causal attention (with mask)
causal = np.triu(np.ones((seq_len, seq_len)) * float("-inf"), k=1)
output_causal, weights_causal = scaled_dot_product_attention(
    Q, K, V, mask=causal
)

Out[6]:

Visualization

Grouped bar chart comparing attention weights for bidirectional vs causal attention at each token position. — Attention distribution for the word 'bank' (position 1). Bidirectional attention distributes weight across all tokens, including future context like 'loan' that disambiguates meaning. Causal attention concentrates entirely on the token itself and 'The', missing crucial context.

The visualization quantifies what bidirectional attention enables. With bidirectional attention, "bank" draws information from all tokens, including future tokens like "loan" and "quickly" that appear later in the sequence. With causal attention, these future tokens contribute zero weight. The representation of "bank" differs fundamentally depending on which attention pattern is used.