Cross-Attention: Connecting Encoder and Decoder

In the previous chapters, we explored encoder-only and decoder-only transformer architectures, each using self-attention to let tokens within a sequence attend to each other. These architectures are powerful for their respective tasks: encoders excel at building rich contextual representations of an input sequence, while decoders excel at generating fluent text one token at a time. But many of the most important tasks in language AI require something different: they require reading one sequence and writing another. Translating English to French, summarizing a long document into a short paragraph, answering a question given a passage of text, converting speech to transcription, or transforming a piece of code into a docstring. All of these tasks share a fundamental structure. You have a source sequence you must understand, and a target sequence you must produce. Cross-attention is the mechanism that makes this possible.

Think of cross-attention as a bridge. On one side stands the encoder, which has carefully read and processed the entire source sequence. On the other side stands the decoder, which is in the middle of generating the target sequence one token at a time. At each step of generation, the decoder needs to look back across the bridge and ask: "Which parts of the source should I pay attention to right now?" Cross-attention is the formal mechanism for answering that question. The decoder formulates a query expressing what it currently needs, the encoder's representations provide the keys and values that answer that query, and the result is a rich, dynamically computed fusion of source and target information.

Understanding cross-attention requires seeing it clearly in contrast to self-attention. In self-attention, a sequence talks to itself: each token in the sequence asks "which other tokens in this same sequence are relevant to me?" The attention matrix is square, and both the questions and the answers come from the same pool of tokens. Cross-attention breaks this symmetry deliberately. Questions still come from the decoder, but the answers come from an entirely different sequence processed by the encoder. This asymmetry is not a complication; it is the core design feature. It is what allows the decoder to incorporate external information while generating text, rather than being confined to what it has already produced.

The mathematical machinery of cross-attention is identical to scaled dot-product attention: QKV representations combined via a softmax-normalized dot product. What differs is only the origin of those three components. This elegance is part of what makes the transformer architecture so modular and extensible. The same attention primitive, applied in different configurations, enables intra-sequence modeling (self-attention), inter-sequence modeling (cross-attention), and constrained generation (masked self-attention). In this chapter, we will understand each aspect of cross-attention in depth: why it is designed the way it is, how the mathematics work, where it lives inside the decoder, how padding and caching affect it, and what its real-world limitations look like.

By the end of this chapter, you will have a complete understanding of cross-attention as both a mathematical object and a practical engineering component. You will be able to implement it from scratch, reason about its behavior during training and inference, and understand why encoder-decoder models like T5, BART, and mBART rely on it as their central information-routing mechanism.

Self-attention and cross-attention share the same mathematical formulation: QKV representations combined with scaled dot-product attention. The critical difference lies in where these components come from. To understand this distinction precisely, we need to see both forms side by side and appreciate what changes when you move from one to the other.

In self-attention, all three projections derive from the same sequence. The intuition is that every token in the sequence can act as a questioner (query), an advertiser (key), and a content provider (value) simultaneously. A token might ask "what context surrounds me?" while also answering the questions of other tokens by advertising its own content. This simultaneous questioning and answering within a single sequence is what lets self-attention capture rich intra-sequence dependencies: subject-verb agreement, pronoun resolution, long-range semantic coherence.

Given an input matrix $X$ containing token representations, self-attention computes QKV representations as:

where:

• $X \in \mathbb{R}^{n \times d}$: the input sequence matrix, with $n$ tokens each represented as a $d$-dimensional vector
• $W^Q \in \mathbb{R}^{d \times d_k}$: the learned query projection matrix
• $W^K \in \mathbb{R}^{d \times d_k}$: the learned key projection matrix
• $W^V \in \mathbb{R}^{d \times d_v}$: the learned value projection matrix
• $Q, K \in \mathbb{R}^{n \times d_k}$: the resulting query and key matrices
• $V \in \mathbb{R}^{n \times d_v}$: the resulting value matrix

Every token attends to every other token (and itself) within the same sequence. The attention matrix is square with shape $(n \times n)$, and each entry $(i, j)$ describes how much token $i$ attends to token $j$.

In cross-attention, the queries come from one sequence while keys and values come from a different sequence. This separation is the key distinction: the decoder generates queries to ask "what do I need?" while the encoder provides keys and values to answer "here's what I have." The decoder cannot answer its own questions by consulting only its own partial output. It needs access to the encoded representation of the source sequence. Cross-attention provides exactly that access, in a learnable and differentiable way.

where:

• $X_{\text{dec}} \in \mathbb{R}^{n_{\text{dec}} \times d}$: the decoder's current representations (what's been generated so far)
• $X_{\text{enc}} \in \mathbb{R}^{n_{\text{enc}} \times d}$: the encoder's output representations (the processed source sequence)
• $W^Q, W^K \in \mathbb{R}^{d \times d_k}$: the learned projection matrices for queries and keys
• $W^V \in \mathbb{R}^{d \times d_v}$: the learned projection matrix for values
• $n_{\text{dec}}$: the number of tokens in the decoder sequence (target length so far)
• $n_{\text{enc}}$: the number of tokens in the encoder sequence (source length)
• $d$: the model dimension (size of each token's representation)

The key insight is that the encoder output appears twice: once for the keys and once for the values. The keys tell the decoder "this is what I offer," and the values tell the decoder "this is what I'll send you if you attend to me." These two roles are separated because the model benefits from being able to project the encoder's information into two different spaces: one for matching (keys against queries) and one for content transmission (values).

This asymmetry is what makes cross-attention powerful for tasks like translation. The decoder asks questions (queries) based on what it has generated so far, and the encoder's representations provide the answers (keys and values). The learned projection matrices $W^Q$, $W^K$, and $W^V$ are trained end-to-end by gradient descent, so the model learns to ask good questions and provide informative answers entirely from the supervision signal (for translation, the signal is simply "produce the correct target sentence given the source").

To understand cross-attention deeply, we need to think about what problem it solves. Imagine you're translating "The cat sat on the mat" into French. You've already generated "Le chat" (the cat), and now you need to produce the next word. Which part of the English sentence should you focus on? The answer is "sat," because that's the verb that follows the subject. The model doesn't know this in advance; it has to figure it out by examining the decoder's current state (which encodes the fact that you've generated a subject) and comparing it against every position in the encoder output to find the best match.

This is precisely what cross-attention computes: for each position in the target sequence, it determines which positions in the source sequence are most relevant, then gathers information from those positions. The mechanism needs to answer two questions simultaneously:

1. Where should I look? Each decoder position needs to identify which encoder positions contain relevant information.
2. What should I extract? Once the relevant positions are identified, the decoder needs to pull out the appropriate information.

The query-key-value framework elegantly separates these concerns. Queries encode what the decoder is looking for, keys encode what each encoder position offers, and values encode the actual information to transmit. The separation between keys and values is conceptually similar to the separation between an index and the data it points to: the key is a compact description used for matching, while the value is the full content that gets transmitted once a match is found.

Let's construct the cross-attention formula piece by piece, understanding why each component is necessary.

Step 1: Create queries from the decoder. Each decoder position needs to express what information it's seeking. We project the decoder representations into a "query space." Think of each query as a search query that the decoder sends out: "I'm looking for information about X." We compute this with a linear projection:

where $X_{\text{dec}} \in \mathbb{R}^{n_{\text{dec}} \times d}$ contains the decoder's current representations and $W^Q \in \mathbb{R}^{d \times d_k}$ is a learned projection matrix. The resulting $Q \in \mathbb{R}^{n_{\text{dec}} \times d_k}$ has one row per decoder position, each encoding "what am I looking for?" The projection $W^Q$ is learned: during training, the model learns to map decoder states into a query space where semantically or syntactically related queries and keys produce high dot products.

Step 2: Create keys from the encoder. Each encoder position needs to advertise what information it contains. We project the encoder output into a "key space" using a different projection:

where $X_{\text{enc}} \in \mathbb{R}^{n_{\text{enc}} \times d}$ is the encoder output and $W^K \in \mathbb{R}^{d \times d_k}$ is another learned matrix. The resulting $K \in \mathbb{R}^{n_{\text{enc}} \times d_k}$ has one row per encoder position, each encoding "here's what I offer." The key space is designed to match the query space: a query that's looking for a verb should produce a high dot product with keys that correspond to verb positions in the source. Critically, the projection $W^K$ is separate from $W^Q$: queries and keys live in the same $d_k$-dimensional space, but they are projected there by different matrices, giving the model flexibility to learn asymmetric matching.

Step 3: Create values from the encoder. When attention flows to an encoder position, we need to specify what information transfers. The value projection captures this:

where $W^V \in \mathbb{R}^{d \times d_v}$ projects into a "value space." The resulting $V \in \mathbb{R}^{n_{\text{enc}} \times d_v}$ contains the content that will be aggregated. The value space is separate from the key space: the key is what you use to find a match, and the value is what you receive once the match is made. This separation allows the model to learn different representations for retrieval (keys) and content transmission (values).

Step 4: Compute similarity scores. Now we need to measure how well each query matches each key. The dot product is ideal for this: when two vectors point in similar directions (when they are highly correlated in the embedding space), their dot product is large. We compute all pairwise scores with a single matrix multiplication:

The resulting score matrix $S \in \mathbb{R}^{n_{\text{dec}} \times n_{\text{enc}}}$ has entry $(i, j)$ equal to the dot product between decoder query $i$ and encoder key $j$. This is where the rectangular shape emerges: we're comparing $n_{\text{dec}}$ queries against $n_{\text{enc}}$ keys. Entry $(i,j)$ being large means decoder position $i$ is likely to attend strongly to encoder position $j$, because their query and key representations are well-aligned.

Step 5: Scale to prevent saturation. In high dimensions, dot products tend to have large magnitudes, which would push softmax into saturation. When a softmax input contains very large values, the output becomes nearly one-hot: almost all probability mass concentrates on the maximum entry, and gradients for non-maximum entries vanish. This makes learning very slow or unstable. Dividing by $\sqrt{d_k}$ normalizes the scores:

Why $\sqrt{d_k}$ specifically? The reasoning comes from the expected magnitude of a dot product between two random unit vectors in $d_k$ dimensions. If the components of $Q$ and $K$ are independent with mean zero and unit variance, the dot product $Q_i \cdot K_j$ has variance equal to $d_k$, so its standard deviation is $\sqrt{d_k}$. Dividing by $\sqrt{d_k}$ normalizes this to unit variance, keeping the scores in a range where softmax gradients remain healthy throughout training. The original "Attention Is All You Need" paper noted this instability explicitly and proposed the scaling factor as the fix.

The histograms illustrate why scaling matters. With $d_k = 64$, raw dot products have a standard deviation around 8, producing scores that can easily reach $\pm 20$ or more. After scaling by $\sqrt{64} = 8$, the standard deviation drops to approximately 1, keeping scores in a range where softmax produces meaningful gradients. This is not a minor detail: without the scaling factor, training deep transformer models becomes significantly more difficult. The gradients from the softmax layer become nearly zero for most attention entries, meaning the model barely learns to adjust its attention patterns. The fix is elegantly simple, and it works.

Step 6: Convert scores to attention weights. We apply softmax row-wise to convert each row of scores into a probability distribution over encoder positions. Each row of the score matrix represents one decoder position asking questions of all encoder positions. The softmax converts these raw scores into a probability distribution, interpreting the scores as un-normalized log-probabilities:

Each row of $A$ sums to 1.0, representing how one decoder position distributes its attention across all encoder positions. An entry $A_{ij} = 0.7$ means decoder position $i$ is directing 70% of its "attention budget" to encoder position $j$. The softmax is applied row-wise, so each decoder position independently computes its own distribution over the encoder. This means different decoder positions can attend to completely different parts of the encoder simultaneously.

Step 7: Aggregate values. Finally, we use the attention weights to compute a weighted sum of encoder values for each decoder position. This is the step where information flows from encoder to decoder:

The output has shape $(n_{\text{dec}} \times d_v)$: one row per decoder position, each containing information gathered from the encoder. If decoder position $i$ attends strongly to encoder position $j$, then the value vector $V_j$ contributes heavily to the output for position $i$. The weighted average captures the decoder's selective reading of the encoder: the output is not a copy of any single encoder representation, but a blend computed according to the decoder's current needs.

Why does this formula make sense? Notice that the softmax attention weights $A$ are non-negative and sum to 1 along each row, so the output $AV$ is literally a convex combination of the encoder's value vectors. Position $i$ in the decoder output is a weighted mixture of all encoder values, with weights determined by how well each encoder key matched that decoder's query. The model learns to assign high weights to relevant encoder positions and low weights to irrelevant ones, entirely through gradient descent on the end task.

Combining all steps, we arrive at the cross-attention formula:

where:

• $Q = X_{\text{dec}} W^Q \in \mathbb{R}^{n_{\text{dec}} \times d_k}$: queries from the decoder, encoding "what information am I looking for?"
• $K = X_{\text{enc}} W^K \in \mathbb{R}^{n_{\text{enc}} \times d_k}$: keys from the encoder, encoding "what information do I have to offer?"
• $V = X_{\text{enc}} W^V \in \mathbb{R}^{n_{\text{enc}} \times d_v}$: values from the encoder, encoding "what content should I contribute?"
• $Q K^T \in \mathbb{R}^{n_{\text{dec}} \times n_{\text{enc}}}$: raw similarity scores measuring alignment between decoder queries and encoder keys
• $\sqrt{d_k}$: scaling factor that maintains healthy gradients during training
• $\text{softmax}(\cdot)$: applied row-wise to produce probability distributions
• $d_k$: query/key dimension (must match for the dot product)
• $d_v$: value dimension (determines output size)

The attention weight matrix has shape $(n_{\text{dec}} \times n_{\text{enc}})$, fundamentally different from self-attention's square $(n \times n)$ matrix. This rectangular shape reflects the asymmetry of sequence-to-sequence tasks: we have $n_{\text{dec}}$ positions that need to attend to $n_{\text{enc}}$ positions, and these lengths are typically different. For translating a 10-word English sentence into a 12-word French sentence when 8 French words have been generated, the attention matrix is $8 \times 10$: eight rows for the eight generated target positions, ten columns for the ten source positions.

Let's make this concrete by tracing the shapes through a realistic example. We'll simulate translating a 6-word English sentence into French, where we've generated 4 words so far.

The heatmap reveals how each decoder token distributes its attention across the source sequence. Row sums equal 1.0 (each row is a probability distribution), but column sums can vary: some source tokens receive more total attention than others. In a trained model, you would often see diagonal-ish patterns for monotonic languages (where words tend to translate in order) or more complex patterns for languages with different word orders.

The choice of where $Q$, $K$, and $V$ come from is not arbitrary. It reflects the fundamental asymmetry of sequence-to-sequence tasks and a carefully designed information flow. Understanding this design choice deeply will help you appreciate why cross-attention is structured the way it is, and why alternatives do not work as well.

Queries represent what you're looking for. At each decoder position, the model is trying to generate the next token. The query encodes "what information do I need from the source to make this prediction?" The decoder has access to its own context (previous tokens, positional information) and uses this to formulate questions. Think of queries as the decoder's "search intent": they are derived from the decoder's current state, which encodes everything the decoder knows so far about what it has generated. A decoder that has generated "The cat" in French might formulate a query that implicitly asks "what verb follows the cat in the English sentence?"

Keys represent what's available. The encoder has processed the entire source sequence and built representations that capture its meaning. Keys advertise "here's what I know about this position in the source." The encoder output is fixed once computed, so keys remain constant throughout decoding. Each key is a compact descriptor of an encoder position's content, learned to be easily comparable to decoder queries in the shared $d_k$-dimensional space. Because queries and keys live in the same space (they are both projected to $\mathbb{R}^{d_k}$), the dot product $Q_i \cdot K_j$ measures how well the decoder's current need matches the encoder's offering at position $j$.

Values represent what gets transmitted. When attention flows from decoder to encoder, the values determine what information transfers. If the decoder strongly attends to a particular encoder position, that position's value vector contributes heavily to the decoder's output. The value representation can be different from the key representation (they use different projection matrices $W^V$ vs $W^K$), allowing the model to learn that "what identifies a position" and "what information that position transmits" are distinct concerns.

Consider translation from English to French. When generating the French word for "cat," the decoder's query might encode "I need information about an animal noun." The encoder's keys for the position containing "cat" would encode "animal, noun, subject." The high dot product between these creates a strong attention weight, and the encoder's value for "cat" (containing semantic features about cats) flows into the decoder's representation. The separation of key and value means the model can learn to index positions (via keys) and transmit content (via values) in ways that are not necessarily the same representation. This flexibility has proven beneficial in practice.

The alternative design, where the decoder queries its own representations rather than the encoder's, would not accomplish the task. A decoder attending only to itself can only look backward at what it has already generated. It cannot read the source sentence. The key innovation of cross-attention is precisely that it routes a query from one sequence to an answer from a completely different sequence, enabling the integration of source and target information in a fully differentiable way that can be optimized end-to-end.

The visualization shows "chat" (French for "cat") attending strongly to "cat" in the encoder. This alignment emerges naturally from the learned query and key projections, which encode semantic relationships between source and target tokens. In a trained model, these patterns can become surprisingly interpretable: different heads learn to track word alignment, positional correspondence, syntactic relationships, and semantic similarity simultaneously.

Unlike causal self-attention in decoders, cross-attention typically does not require causal masking. The decoder can attend to any position in the encoder output because the encoder sequence is fully processed before decoding begins. There's no "future information" in the encoder to hide. The encoder has already seen the entire source sentence, so its output at every position already incorporates global context. When the decoder queries the encoder, it can freely look at any position, including positions corresponding to the end of the source sentence, without any information leak.

This contrasts sharply with decoder self-attention, where causal masking is essential. In decoder self-attention, positions can only attend to earlier positions because the decoder is generating tokens sequentially: if it could attend to future tokens, it would be "cheating" by seeing answers before they are generated. Cross-attention has no such concern because the encoder output is not being generated; it is a fixed input representation.

However, cross-attention does require padding masking when processing batches with variable-length source sequences. Real-world datasets contain sentences of many different lengths. If you want to process a batch of several sentences simultaneously, you need to pad the shorter sequences to match the length of the longest sequence in the batch. These padding tokens carry no meaningful information, and the decoder should not attend to them. If it did, the padding would inject noise into the decoder's representations, causing the model to incorporate garbage information when computing its attention-weighted average of the encoder.

The solution is to set the attention scores for padding positions to a very large negative number (like $-10^9$) before applying softmax. Since $e^{-10^9} \approx 0$, these positions receive neededly zero weight after softmax, effectively masking them out. The probability mass that would have gone to padding positions instead concentrates on the valid encoder positions. This ensures the decoder only attends to meaningful content.

The comparison makes the effect of masking clear. Without masking, attention bleeds into padding positions, corrupting the decoder's representations with meaningless information. With masking, all attention concentrates on the valid tokens, and the model ignores padding entirely. The right-hand heatmap shows darker cells in the first three columns (valid positions) and empty cells in the last three columns (padding), which is exactly the behavior we want.

In the original transformer architecture from "Attention Is All You Need," each decoder layer contains three sub-layers arranged in a specific order, and that order is not accidental. Understanding why these sub-layers are ordered the way they are illuminates the information flow that makes the decoder powerful.

The three sub-layers in each decoder layer are:

1. Masked self-attention: Decoder tokens attend to previous decoder tokens
2. Cross-attention: Decoder tokens attend to encoder output
3. Feed-forward network: Position-wise transformation

The order matters deeply. Masked self-attention comes first, allowing each decoder position to incorporate information from previously generated tokens and build a rich representation of "what has been generated so far and what the context implies." This representation then becomes the query for cross-attention. By processing self-attention first, the decoder ensures that its queries to the encoder are informed by its own context. A decoder that has just generated "Le chat" (the cat) will produce different queries than one that has generated "La maison" (the house), because the self-attention layer has already encoded the contextual implications of the generated sequence.

Then cross-attention brings in information from the source sequence, using the self-attention-enriched decoder state as queries. This means the decoder can ask the encoder highly context-sensitive questions: "Given that I've generated 'the cat' in French so far, what verb did the English sentence use?" Finally, the feed-forward network performs a position-wise nonlinear transformation on the combined representation, giving the model additional capacity to process and transform the fused source-target information before producing the next prediction.

Each sub-layer is wrapped with a residual connection and layer normalization, typically as:

The residual connection ensures gradients flow cleanly through the network during training, and layer normalization stabilizes the activations across positions. Together, these allow very deep decoder stacks (12, 24, or more layers in large models) to train effectively.

The cross-attention layer is the only point where information flows from encoder to decoder. The encoder output is computed once and then used identically in every decoder layer. This architectural choice has an important implication: the encoder keys and values are the same regardless of how much the decoder has generated. Every decoder layer in a 12-layer decoder, for example, attends to the same set of encoder representations. This means the encoder representations must be rich enough to answer a wide variety of questions from 12 different decoder layers, each with different learned attention weights.

The fact that the same encoder output feeds every decoder layer is also an efficiency advantage. The encoder runs once. Its computation scales with the source sequence length, and the result is then cached and accessed repeatedly across all decoder steps and all decoder layers. This is fundamentally different from the decoder's self-attention, which grows as more tokens are generated.

During autoregressive generation, the cross-attention mechanism operates in a precise sequence at each decoding step. Understanding this sequence helps you reason about both correctness and performance.

First, the encoder processes the entire source sequence once at the beginning. It runs all its encoder layers, and the output (a matrix of shape $(n_{\text{enc}} \times d_{\text{model}})$) is stored in memory. This is a one-time cost proportional to the source length.

Then, for each decoder step $t$:

• The decoder's self-attention sees positions $1, 2, \ldots, t$ (previous outputs plus current). It computes intra-sequence attention over everything generated so far, building a rich context representation. The causal mask ensures position $t$ cannot see positions $t+1$ onward.
• Cross-attention takes the self-attention output as queries and uses the cached encoder output to compute keys and values. The decoder queries the encoder: "given what I've generated and my current context, which parts of the source are relevant to my next prediction?"
• The gathered information from cross-attention, combined with the self-attention output via residual connection and normalization, feeds into the feed-forward network for further processing.
• This process repeats through all decoder layers.
• Finally, the output of the last decoder layer is projected through a vocabulary classifier (a linear layer with softmax), producing a probability distribution over the vocabulary for the next token.

The encoder representations stay fixed, while the decoder's queries evolve as it generates more tokens. Early in generation, the decoder might ask broad questions about the source: "what is the main subject?" Later, as more context accumulates, it might ask increasingly specific questions: "what preposition connects the action to the location in the source?" This dynamic querying is what allows the decoder to dynamically allocate its attention over the source sequence as generation proceeds.

Let's implement a complete cross-attention module following the patterns used in production transformers. We'll build it step by step, starting with single-head attention and working up to the full multi-head version.

The implementation below follows the structure of the previous mathematical derivation closely, making it straightforward to verify that the code matches the formulas. In production code, you would use batched matrix operations via PyTorch or JAX, but the single-sequence NumPy implementation here makes the shapes and operations maximally transparent.

Just like self-attention, cross-attention benefits enormously from multiple attention heads. Each head runs an independent cross-attention computation with its own set of QKV projection matrices, and the results from all heads are concatenated and projected back to the model dimension. Multiple heads let the model add capacity while specializing each head in different aspects of source-target alignment.

Think of multi-head attention as asking multiple questions simultaneously. One head might specialize in semantic alignment (which source word has the same meaning as the current target word?), another in syntactic alignment (which source word plays the same grammatical role?), and a third in positional proximity (which source word is near the corresponding position in the sequence?). Because each head has its own independent projection matrices, these specializations emerge naturally from training. The final concatenation and projection combine all these perspectives into a single rich representation.

The mathematical formulation of multi-head attention applies cross-attention $h$ times in parallel, then combines the results:

where each head computes:

and:

• $W^Q_i \in \mathbb{R}^{d \times d_k}$: query projection for head $i$
• $W^K_i \in \mathbb{R}^{d \times d_k}$: key projection for head $i$
• $W^V_i \in \mathbb{R}^{d \times d_v}$: value projection for head $i$
• $W^O \in \mathbb{R}^{h d_v \times d}$: output projection combining all heads

In practice, we set $d_k = d_v = d / h$, so each head operates on a $d/h$-dimensional subspace, and concatenating $h$ heads of dimension $d/h$ gives back $d$ dimensions. This keeps the total computation comparable to single-head attention while providing the diversity of perspectives.

Each head develops its own attention pattern. In trained translation models, researchers have observed heads that focus on positional alignment (source position $i$ attends to target position $i$), heads that track syntactic relationships, and heads that capture semantic similarities. The diversity across heads allows the model to capture multiple aspects of source-target alignment simultaneously. Research by Michel et al. (2019) found that many heads in trained models are prunable without significant performance loss, suggesting that the learned specialization tends to be concentrated in a subset of heads, though which heads are important varies by task.

During autoregressive generation, cross-attention has a remarkable computational advantage over decoder self-attention: the keys and values from the encoder can be computed once and reused for every decoding step. This is the key insight behind KV caching for cross-attention.

To understand why this works, recall that cross-attention keys and values come entirely from the encoder output: $K = X_{\text{enc}} W^K$ and $V = X_{\text{enc}} W^V$. The encoder output $X_{\text{enc}}$ is fixed throughout decoding. It does not change as the decoder generates tokens. Therefore, $K$ and $V$ are the same matrices at every decoding step. Computing them repeatedly would be wasteful. Instead, we compute them once after the encoder finishes and cache them for reuse.

Contrast this with decoder self-attention, where caching is more complex. The decoder's self-attention keys and values depend on all previously generated tokens, so as the decoder generates more tokens, the key and value matrices grow. This is the KV cache that grows during generation and is the primary memory bottleneck for long generations. Cross-attention's cache, by contrast, has a fixed size equal to the encoder length from the very first decoding step.

This caching is particularly important for long source sequences. Without caching, generating 100 tokens from a 1000-token source would require recomputing the encoder's KV projections 100 times. With caching, we compute them once, and each decoding step only needs to compute the query projection for the new token and multiply it against the fixed, cached keys. This reduces the per-step computational cost of cross-attention from $O(n_{\text{enc}} \cdot d)$ matrix multiplications to just the query projection (one multiplication of size $d \times d_k$) plus the attention computation.

In memory terms, the cross-attention KV cache occupies $2 \cdot n_{\text{enc}} \cdot d_v \cdot n_{\text{layers}}$ values (two matrices of size $n_{\text{enc}} \times d_v$ per decoder layer). For a model with $d_v = 64$, $n_{\text{enc}} = 512$, and 12 layers, this is about 800 thousand values, which is very manageable. The decoder self-attention KV cache, by contrast, grows linearly with the number of generated tokens and can become a significant memory constraint for very long outputs.

Let's trace through cross-attention during a complete translation step to see how all the pieces fit together. We'll follow a single decoding step of translating "I love cats" into French, where we've already generated "J'" and need to decide on the next token.

In this example, "aime" (love) should ideally attend strongly to "love" in the source, because these words are semantic translations of each other. While our random weights don't show this pattern (since we haven't trained the model), a trained model would learn to align related words across languages. The gradient signal during training continuously adjusts $W^Q$, $W^K$, and $W^V$ so that semantically corresponding positions produce high query-key dot products, causing the attention to flow appropriately.

The cross-attention output for each target position now contains a weighted mixture of encoder information. For "aime," ideally most of the weight goes to the "love" encoder representation, so the cross-attention output for the "aime" position is dominated by features encoding the concept of loving. This representation then flows into the feed-forward layer, and ultimately helps the model predict the next target token: "les" (the, plural), which follows "aime" in the French translation.

Cross-attention is the mechanism that makes encoder-decoder transformers work. It provides a direct, differentiable connection between source and target sequences, enabling end-to-end training for sequence-to-sequence systems used in translation and summarization. Understanding its limitations is equally important for building systems that work well in practice.

The computational complexity of cross-attention is $O(n_{\text{dec}} \times n_{\text{enc}} \times d)$, where $n_{\text{dec}}$ is the number of decoder tokens, $n_{\text{enc}}$ is the number of encoder tokens, and $d$ is the model dimension. The attention score matrix alone has $n_{\text{dec}} \times n_{\text{enc}}$ entries, and computing it requires $n_{\text{dec}} \times n_{\text{enc}}$ dot products of $d_k$-dimensional vectors. For long source documents such as books, legal contracts, or scientific papers, $n_{\text{enc}}$ can reach thousands or tens of thousands of tokens, making this cost substantial. If you are summarizing a 10,000-word document into a 200-word summary, the cross-attention at each decoder step must score each of the 200 generated tokens against each of approximately 7,500 source tokens. Techniques like sparse cross-attention (attending to only a selected subset of encoder positions) or retrieval-augmented generation (retrieving relevant chunks before encoding) address this by reducing the effective encoder length.

Cross-attention assumes the entire encoder output is available before decoding begins. This makes it unsuitable for streaming applications where source and target are produced simultaneously. Think of real-time speech-to-speech translation: the encoder would need to wait until the speaker finishes an entire sentence before the decoder can start translating, introducing noticeable latency. For such cases, architectures like streaming transformers or monotonic attention (which constrains the decoder to attend to source positions in a monotonically increasing order, allowing it to start decoding before the source is complete) provide alternatives. These architectures trade some quality for lower latency, which is an acceptable tradeoff in many real-time applications.

The fixed encoder output means the decoder cannot "ask follow-up questions" that change how the source is encoded. Each decoder layer sees the same encoder representations, regardless of what the decoder has generated so far. This is a form of information bottleneck: all the relevant information about the source must be captured in the initial encoding, before the decoder has any opportunity to influence how the source is represented. Some architectures address this by adding encoder layers that receive decoder feedback, creating a more interactive encoder-decoder loop. However, such architectures are more complex, harder to train, and sacrifice the efficiency benefit of computing the encoder once.

Another subtle limitation is that cross-attention is inherently a soft alignment mechanism: every decoder position attends to all encoder positions, just with different weights. This is expressive but sometimes misaligned with the actual structure of the task. For tasks with strict one-to-one word correspondences (like many types of technical translation), the soft attention can sometimes spread probability mass too broadly, attending to many irrelevant positions with small but nonzero weights. Hard attention mechanisms, which make discrete attend/not-attend decisions, are more computationally challenging to train (they require reinforcement learning or approximations) but can be more appropriate for tasks with sharp alignment structure.

Despite these considerations, cross-attention has proven remarkably effective across an enormous range of tasks. It underlies the success of T5 (which treats all NLP tasks as text-to-text problems), BART (which learns to reconstruct corrupted text and generalizes to summarization and translation), mBART (multilingual BART for cross-lingual generation), Whisper (for speech-to-text transcription), and CodeT5 (for code generation and summarization). The mechanism's elegance, matching the same scaled dot-product attention used in self-attention, makes it easy to implement, easy to optimize on modern hardware, and easy to analyze via attention visualization. It remains one of the foundational building blocks of modern language AI, and understanding it deeply is essential for working with any encoder-decoder architecture.

Cross-attention bridges encoder and decoder in sequence-to-sequence transformers, enabling the decoder to gather information from the encoded source sequence while generating output tokens. This single mechanism solves the fundamental problem of sequence-to-sequence modeling: how does the decoder know what to say without being able to read the source?

Key takeaways from this chapter:

• Q from decoder, K and V from encoder: This asymmetry defines cross-attention. Queries represent what the decoder is looking for; keys and values represent what the encoder offers. The separation between keys (used for matching) and values (used for content transmission) gives the model flexibility to learn different representations for retrieval and content transfer.

• Rectangular attention matrix: Unlike self-attention's $(n \times n)$ square matrix, cross-attention produces a $(n_{\text{dec}} \times n_{\text{enc}})$ rectangular weight matrix, where $n_{\text{dec}}$ is the target sequence length and $n_{\text{enc}}$ is the source sequence length. Each row represents how one decoder position distributes attention across all encoder positions, forming a probability distribution that sums to 1.

• Padding masks, not causal masks: Cross-attention masks padding tokens in the encoder but doesn't need causal masking. The encoder sequence is fully available because it is a fixed input processed entirely before decoding begins. Padding masks are essential for batched training with variable-length source sequences.

• Fixed encoder output and KV caching: The encoder is computed once, and its keys and values can be cached for efficient inference across all decoder steps and all decoder layers. This makes cross-attention significantly more efficient than decoder self-attention during autoregressive generation.

• Placement in decoder layers: Cross-attention appears between masked self-attention and the feed-forward network in each decoder layer. This ordering ensures the decoder first processes its own context (via self-attention) before querying the encoder (via cross-attention), allowing the queries to be informed by the decoder's generated context.

• Multi-head diversity: Like self-attention, cross-attention benefits from multiple heads that can specialize in different alignment patterns, simultaneously capturing semantic equivalences, syntactic correspondences, and positional relationships between source and target sequences.

• Broad applicability: Cross-attention extends well beyond translation to summarization, question answering, speech recognition, code generation, and any task where you need to generate a sequence conditioned on a different input sequence. The same mathematical machinery applies in all these settings.

The next chapter explores weight tying, a technique for sharing parameters between embedding layers and output projections, reducing model size while maintaining performance.

When implementing cross-attention in encoder-decoder transformers, several parameters control the mechanism's behavior and capacity. Choosing these parameters well is important for model quality and training efficiency.

• d_model: The model dimension, which is the size of input token representations. Both encoder outputs and decoder states should have this dimension. Common values range from 256 to 1024, with larger models using 2048 or more. The model dimension is the primary parameter controlling the overall expressiveness of the attention computation.

• d_k (query/key dimension): The dimension of the projected queries and keys. This controls the capacity of the attention scoring mechanism. Typically set to d_model // n_heads in multi-head attention, giving each head a portion of the full dimensionality. Smaller $d_k$ produces lower-variance attention scores even without the scaling factor, while larger $d_k$ gives more capacity but requires scaling more aggressively.

• d_v (value dimension): The dimension of the projected values. Often equal to $d_k$, but can differ. This determines the size of information transmitted when attention flows from encoder to decoder. In practice, $d_v = d_k = d_{\text{model}} / n_{\text{heads}}$ is the standard choice.

• n_heads: The number of parallel attention heads in multi-head cross-attention. More heads allow the model to attend to different aspects of the source sequence simultaneously. Typical values are 8, 12, or 16 heads. The original transformer used 8 heads with $d_{\text{model}} = 512$, giving $d_k = d_v = 64$ per head.

• encoder_mask: A boolean mask indicating which encoder positions are valid (True) versus padding (False). Essential for batched processing of variable-length source sequences to prevent attending to meaningless padding tokens. Without this mask, shorter sequences in a batch would receive spurious attention from the padding positions.

• scale factor: The scaling factor $1/\sqrt{d_k}$ applied before softmax. This is automatically determined by $d_k$ and prevents attention scores from becoming too large in high dimensions, which would cause softmax to saturate and gradients to vanish. This simple fix has a large practical impact on training stability.

Ready to test your understanding of cross-attention? Take this quick quiz to reinforce what you've learned about connecting encoder and decoder in sequence-to-sequence transformers.