Cross-Attention › A Worked Example: Translation Step by Step
Language AI HandbookAll chapters Cross-Attention › A Worked Example: Translation Step by StepOne section · Sections in this chapter10 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's trace through cross-attention during a complete translation step to see how all the pieces fit together. We'll follow a single decoding step of translating "I love cats" into French, where we've already generated "J'" and need to decide on the next token.
# Translation example: "I love cats" → "J'aime les chats"
source_sentence = ["I", "love", "cats"]
target_prefix = ["J'", "aime"] # Already generated
n_src = len(source_sentence)
n_tgt = len(target_prefix)
d_model = 8
# Simulated encoder output (in practice, from transformer encoder)
encoder_out = np.random.randn(n_src, d_model)
# Simulated decoder state after self-attention
decoder_state = np.random.randn(n_tgt, d_model)
# Cross-attention
cross_attn = CrossAttention(d_model, d_k=8, d_v=8)
output, weights = cross_attn(decoder_state, encoder_out)Translation: 'I love cats' → 'J'aime les chats'
Source tokens: ['I', 'love', 'cats']
Target prefix: ["J'", 'aime']
Cross-attention weights:
I love cats
J' [0.338 0.351 0.312]
aime [0.381 0.456 0.163]In this example, "aime" (love) should ideally attend strongly to "love" in the source, because these words are semantic translations of each other. While our random weights don't show this pattern (since we haven't trained the model), a trained model would learn to align related words across languages. The gradient signal during training continuously adjusts , , and so that semantically corresponding positions produce high query-key dot products, causing the attention to flow appropriately.
The cross-attention output for each target position now contains a weighted mixture of encoder information. For "aime," ideally most of the weight goes to the "love" encoder representation, so the cross-attention output for the "aime" position is dominated by features encoding the concept of loving. This representation then flows into the feed-forward layer, and ultimately helps the model predict the next target token: "les" (the, plural), which follows "aime" in the French translation.
Cross-attention as used in modern transformers descends from a lineage of attention mechanisms in neural machine translation. The earliest influential neural translation systems, like the models of Sutskever et al. (2014), used recurrent encoder-decoder architectures. The encoder compressed the entire source sentence into a fixed-length vector, and the decoder then generated the translation from this vector. The fundamental problem was that a fixed-length vector could not adequately represent a long, complex sentence.
Bahdanau et al. (2015) introduced additive attention, which allowed the decoder to attend to different encoder hidden states at each decoding step rather than relying on a single compressed vector. This was the first practical cross-attention mechanism, though it used a learned additive compatibility function rather than the dot product used by transformers. Luong et al. (2015) shortly afterward proposed multiplicative (dot-product) attention for sequence-to-sequence models, which is computationally simpler and more similar to transformer cross-attention.
Vaswani et al. (2017), in "Attention Is All You Need," extended these ideas to the full transformer architecture, using multi-head scaled dot-product attention for all three types of attention in the model: encoder self-attention, decoder self-attention, and cross-attention. The introduction of the scaling factor , multi-head attention, and the parallel computation of attention across all positions (rather than one RNN step at a time) were the key innovations that made transformers dramatically more powerful and efficient than RNN-based attention models. The cross-attention mechanism in modern models is essentially the Vaswani et al. formulation, extended to larger scales and diverse applications far beyond the original machine translation use case.