Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Encoder-Decoder Architecture › Cross-Attention: The Bridge Between Encoder and Decoder

Michael BrenndoerferUpdated June 20, 202510 min read

Language AI HandbookAll chapters

Part 16 of 67Transformer Architectures

Chapter 113Encoder-Decoder Architecture › Cross-Attention: The Bridge Between Encoder and Decoder

Reading section 3 of 13 · Cross-Attention: The Bridge Between Encoder and Decoder · 10 minSection 3 of 13 · 10 min

Section 3 of 13 · 10 min read

Encoder-Decoder Architecture › Cross-Attention: The Bridge Between Encoder and DecoderOne section · Sections in this chapter3 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Consider the challenge of translation. When generating the French word "chat," how does the decoder know to look at the English word "cat" rather than "the" or "sat"? The encoder has produced rich representations of every source word, but the decoder needs a way to selectively access the right information at each generation step.

This is precisely what cross-attention accomplishes. It creates a dynamic, learned connection between the decoder's current state and the encoder's representations, allowing each generated token to "query" the source sequence and retrieve relevant information.

Cross-Attention vs Self-Attention

In self-attention, QKV representations all come from the same sequence, with tokens attending to other tokens within their own context. Cross-attention breaks this symmetry: queries come from the decoder (what information am I looking for?), while keys and values come from the encoder (what information is available in the source?). This asymmetry is what enables the decoder to search for and retrieve relevant source information at each generation step.

The Cross-Attention MechanismLink Copied

To understand cross-attention mathematically, we need to track two distinct sequences flowing through the architecture. The decoder maintains hidden states $\mathbf{H}_{\text{dec}} \in \mathbb{R}^{m \times d}$ for the $m$ target positions generated so far. The encoder has already produced its output $\mathbf{H}_{\text{enc}} \in \mathbb{R}^{n \times d}$ for all $n$ source positions. The key insight is that these sequences can have completely different lengths. We might translate a 10-word English sentence into a 15-word French sentence, or summarize a 1000-token document into 50 tokens.

Cross-attention connects these two sequences through three projections, each serving a distinct purpose:

\mathbf{Q} = \mathbf{H}_{\text{dec}} \mathbf{W}^Q, \quad \mathbf{K} = \mathbf{H}_{\text{enc}} \mathbf{W}^K, \quad \mathbf{V} = \mathbf{H}_{\text{enc}} \mathbf{W}^V

where:

$\mathbf{H}_{\text{dec}} \in \mathbb{R}^{m \times d}$ : decoder hidden states, one per target position
$\mathbf{H}_{\text{enc}} \in \mathbb{R}^{n \times d}$ : encoder output, one per source position
$\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V \in \mathbb{R}^{d \times d_k}$ : learned projection matrices
$\mathbf{Q} \in \mathbb{R}^{m \times d_k}$ : query matrix with $m$ queries from decoder
$\mathbf{K} \in \mathbb{R}^{n \times d_k}$ : key matrix with $n$ keys from encoder
$\mathbf{V} \in \mathbb{R}^{n \times d_v}$ : value matrix with $n$ values from encoder

Cross-attention uses different sources for these projections: queries come from the decoder, while keys and values come from the encoder. Think of it as the decoder asking questions (queries) and the encoder providing a searchable index (keys) along with the actual information to retrieve (values).

With these projections in place, the attention computation proceeds through three steps. First, we measure compatibility between every decoder query and every encoder key. Then we convert these compatibility scores into attention weights. Finally, we use these weights to compute a weighted sum of encoder values:

\text{CrossAttention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}}\right)\mathbf{V}

Let's unpack each component:

Compatibility scores $\mathbf{Q}\mathbf{K}^T \in \mathbb{R}^{m \times n}$ : This matrix multiplication compares each decoder query against all encoder keys. Entry $(i, j)$ measures how strongly decoder position $i$ should attend to encoder position $j$ . Higher values indicate stronger relevance.
Scaling by $\sqrt{d_k}$ : As dimension $d_k$ grows, dot products tend to grow in magnitude, pushing softmax toward extreme values (near 0 or 1). Dividing by $\sqrt{d_k}$ keeps gradients stable during training.

Out[5]:

Visualization

Histogram showing unscaled dot products spreading wider for larger dimensions. — Without scaling, dot products grow with dimension d, concentrating softmax outputs near 0 or 1.

Histogram showing scaled dot products remaining consistent across dimensions. — With square root scaling, scores remain well-distributed regardless of dimension, enabling stable gradients.

Softmax normalization: Converts each row of scores into a probability distribution. After softmax, each decoder position has attention weights over the source that sum to 1. These weights represent "how much" of each source position to incorporate.
Weighted value sum: The final matrix multiplication retrieves information from the encoder. Each decoder position gets a weighted combination of encoder values, where the weights come from the attention distribution.

The key geometric insight is that $\mathbf{Q}\mathbf{K}^T$ produces an $m \times n$ rectangular matrix, not a square matrix. Each of the $m$ decoder positions computes attention weights over all $n$ encoder positions. This rectangular attention matrix is what distinguishes cross-attention from self-attention and enables sequence-to-sequence transformation between sequences of different lengths.

In[6]:

Code

def softmax(x, axis=-1):
    """Numerically stable softmax."""
    x_max = np.max(x, axis=axis, keepdims=True)
    exp_x = np.exp(x - x_max)
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)


def cross_attention(Q, K, V):
    """
    Compute cross-attention between decoder queries and encoder keys/values.

    Args:
        Q: Decoder queries of shape (m, d_k) - one per target position
        K: Encoder keys of shape (n, d_k) - one per source position
        V: Encoder values of shape (n, d_v) - one per source position

    Returns:
        Output of shape (m, d_v) and attention weights of shape (m, n)
    """
    d_k = Q.shape[-1]

    # Compute attention scores: (m, n)
    scores = Q @ K.T / np.sqrt(d_k)

    # No masking needed - decoder can attend to all encoder positions
    weights = softmax(scores, axis=-1)

    # Weighted sum of values: (m, d_v)
    output = weights @ V

    return output, weights


# Example: 4 decoder positions attending to 6 encoder positions
m, n, d_k = 4, 6, 64

# Simulate decoder queries (from current decoder hidden states)
H_dec = np.random.randn(m, d_k) * 0.1
W_q = np.random.randn(d_k, d_k) * 0.1
Q = H_dec @ W_q

# Simulate encoder outputs (already computed, fixed)
H_enc = np.random.randn(n, d_k) * 0.1
W_k = np.random.randn(d_k, d_k) * 0.1
W_v = np.random.randn(d_k, d_k) * 0.1
K = H_enc @ W_k
V = H_enc @ W_v

output, attn_weights = cross_attention(Q, K, V)

Out[7]:

Console

Cross-Attention Dimensions
---------------------------------------------
Decoder hidden states (H_dec): (4, 64)
Encoder hidden states (H_enc): (6, 64)

Queries from decoder (Q): (4, 64)
Keys from encoder (K): (6, 64)
Values from encoder (V): (6, 64)

Attention weights (m × n): (4, 6)
Output (m × d_v): (4, 64)

Each decoder position attends to all encoder positions.
Row sums of attention weights: [1. 1. 1. 1.]

The output confirms cross-attention produces a rectangular $4 \times 6$ attention weight matrix rather than a square one. Each decoder position computes a probability distribution over all 6 encoder positions, which is why the row sums equal exactly 1.0. This demonstrates the key insight: queries from 4 target positions attend to keys from 6 source positions, creating asymmetric attention that lets each generated word "look at" the entire input sequence.

Out[8]:

Visualization

Heatmap showing cross-attention weights between 4 decoder and 6 encoder positions. — Cross-attention weight matrix showing how each decoder position (row) attends to encoder positions (column). Unlike self-attention, this matrix is rectangular when source and target lengths differ. Each row sums to 1, representing where each decoder position looks in the source sequence.

What Cross-Attention LearnsLink Copied

The remarkable property of cross-attention is that it learns meaningful alignments entirely from the end task, without any explicit supervision about which source words correspond to which target words. During training, gradients flow backward from the translation loss, gradually shaping the attention weights to focus on the most relevant source positions.

When translating "The cat sat" to "Le chat était assis," a trained model learns attention patterns that reflect linguistic structure:

When generating "chat" (French for cat), attend strongly to "cat" in the source, a direct lexical correspondence
When generating "était assis" (was sitting), attend primarily to "sat", capturing a one-to-many mapping where one English word expands to two French words
When generating "Le," attend to context that determines grammatical gender, since the model learns that "cat" requires the masculine article

This learned alignment emerges from the training signal alone. The model discovers through thousands of examples that attending to the right source positions helps produce correct translations. In effect, cross-attention replaces the explicit alignment models used in pre-neural statistical machine translation, while being more flexible since it can capture soft, distributed alignments rather than hard one-to-one correspondences.

Out[9]:

Visualization

Diagram showing alignment arrows from target tokens to source tokens based on attention weights. — Illustration of how cross-attention creates soft alignment between source and target tokens. When generating each target word, the decoder attends to relevant source words. This alignment is learned end-to-end without explicit supervision.