Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Cross-Attention › Placement in the Decoder

Michael BrenndoerferUpdated June 18, 20257 min read

Language AI HandbookAll chapters

Part 16 of 67Transformer Architectures

Chapter 114Cross-Attention › Placement in the Decoder

Reading section 6 of 14 · Placement in the Decoder · 7 minSection 6 of 14 · 7 min

Section 6 of 14 · 7 min read

Cross-Attention › Placement in the DecoderOne section · Sections in this chapter6 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

In the original transformer architecture from "Attention Is All You Need," each decoder layer contains three sub-layers arranged in a specific order, and that order is not accidental. Understanding why these sub-layers are ordered the way they are illuminates the information flow that makes the decoder powerful.

The three sub-layers in each decoder layer are:

Masked self-attention: Decoder tokens attend to previous decoder tokens
Cross-attention: Decoder tokens attend to encoder output
Feed-forward network: Position-wise transformation

The order matters deeply. Masked self-attention comes first, allowing each decoder position to incorporate information from previously generated tokens and build a rich representation of "what has been generated so far and what the context implies." This representation then becomes the query for cross-attention. By processing self-attention first, the decoder ensures that its queries to the encoder are informed by its own context. A decoder that has just generated "Le chat" (the cat) will produce different queries than one that has generated "La maison" (the house), because the self-attention layer has already encoded the contextual implications of the generated sequence.

Then cross-attention brings in information from the source sequence, using the self-attention-enriched decoder state as queries. This means the decoder can ask the encoder highly context-sensitive questions: "Given that I've generated 'the cat' in French so far, what verb did the English sentence use?" Finally, the feed-forward network performs a position-wise nonlinear transformation on the combined representation, giving the model additional capacity to process and transform the fused source-target information before producing the next prediction.

Each sub-layer is wrapped with a residual connection and layer normalization, typically as:

\text{output} = \text{LayerNorm}(x + \text{Sublayer}(x))

The residual connection ensures gradients flow cleanly through the network during training, and layer normalization stabilizes the activations across positions. Together, these allow very deep decoder stacks (12, 24, or more layers in large models) to train effectively.

Out[16]:

Visualization

Vertical diagram showing three sub-layers in a decoder block: masked self-attention, cross-attention, and feed-forward, each with residual connections and layer normalization. — Structure of a transformer decoder layer. Each position first attends to previous decoder positions (masked self-attention), then attends to encoder outputs (cross-attention), and finally passes through a feed-forward network. Add and Norm layers wrap each sub-layer with residual connections and layer normalization, enabling stable training of deep decoder stacks.

The cross-attention layer is the only point where information flows from encoder to decoder. The encoder output is computed once and then used identically in every decoder layer. This architectural choice has an important implication: the encoder keys and values are the same regardless of how much the decoder has generated. Every decoder layer in a 12-layer decoder, for example, attends to the same set of encoder representations. This means the encoder representations must be rich enough to answer a wide variety of questions from 12 different decoder layers, each with different learned attention weights.

The fact that the same encoder output feeds every decoder layer is also an efficiency advantage. The encoder runs once. Its computation scales with the source sequence length, and the result is then cached and accessed repeatedly across all decoder steps and all decoder layers. This is fundamentally different from the decoder's self-attention, which grows as more tokens are generated.

Information Flow During GenerationLink Copied

During autoregressive generation, the cross-attention mechanism operates in a precise sequence at each decoding step. Understanding this sequence helps you reason about both correctness and performance.

First, the encoder processes the entire source sequence once at the beginning. It runs all its encoder layers, and the output (a matrix of shape $(n_{\text{enc}} \times d_{\text{model}})$ ) is stored in memory. This is a one-time cost proportional to the source length.

Then, for each decoder step $t$ :

The decoder's self-attention sees positions $1, 2, \ldots, t$ (previous outputs plus current). It computes intra-sequence attention over everything generated so far, building a rich context representation. The causal mask ensures position $t$ cannot see positions $t+1$ onward.
Cross-attention takes the self-attention output as queries and uses the cached encoder output to compute keys and values. The decoder queries the encoder: "given what I've generated and my current context, which parts of the source are relevant to my next prediction?"
The gathered information from cross-attention, combined with the self-attention output via residual connection and normalization, feeds into the feed-forward network for further processing.
This process repeats through all decoder layers.
Finally, the output of the last decoder layer is projected through a vocabulary classifier (a linear layer with softmax), producing a probability distribution over the vocabulary for the next token.

The encoder representations stay fixed, while the decoder's queries evolve as it generates more tokens. Early in generation, the decoder might ask broad questions about the source: "what is the main subject?" Later, as more context accumulates, it might ask increasingly specific questions: "what preposition connects the action to the location in the source?" This dynamic querying is what allows the decoder to dynamically allocate its attention over the source sequence as generation proceeds.