Cross-Attention › Summary

Cross-attention bridges encoder and decoder in sequence-to-sequence transformers, enabling the decoder to gather information from the encoded source sequence while generating output tokens. This single mechanism solves the fundamental problem of sequence-to-sequence modeling: how does the decoder know what to say without being able to read the source?

Key takeaways from this chapter:

• Q from decoder, K and V from encoder: This asymmetry defines cross-attention. Queries represent what the decoder is looking for; keys and values represent what the encoder offers. The separation between keys (used for matching) and values (used for content transmission) gives the model flexibility to learn different representations for retrieval and content transfer.

• Rectangular attention matrix: Unlike self-attention's $(n \times n)$ square matrix, cross-attention produces a $(n_{\text{dec}} \times n_{\text{enc}})$ rectangular weight matrix, where $n_{\text{dec}}$ is the target sequence length and $n_{\text{enc}}$ is the source sequence length. Each row represents how one decoder position distributes attention across all encoder positions, forming a probability distribution that sums to 1.

• Padding masks, not causal masks: Cross-attention masks padding tokens in the encoder but doesn't need causal masking. The encoder sequence is fully available because it is a fixed input processed entirely before decoding begins. Padding masks are essential for batched training with variable-length source sequences.

• Fixed encoder output and KV caching: The encoder is computed once, and its keys and values can be cached for efficient inference across all decoder steps and all decoder layers. This makes cross-attention significantly more efficient than decoder self-attention during autoregressive generation.

• Placement in decoder layers: Cross-attention appears between masked self-attention and the feed-forward network in each decoder layer. This ordering ensures the decoder first processes its own context (via self-attention) before querying the encoder (via cross-attention), allowing the queries to be informed by the decoder's generated context.

• Multi-head diversity: Like self-attention, cross-attention benefits from multiple heads that can specialize in different alignment patterns, simultaneously capturing semantic equivalences, syntactic correspondences, and positional relationships between source and target sequences.

• Broad applicability: Cross-attention extends well beyond translation to summarization, question answering, speech recognition, code generation, and any task where you need to generate a sequence conditioned on a different input sequence. The same mathematical machinery applies in all these settings.

The next chapter explores weight tying, a technique for sharing parameters between embedding layers and output projections, reducing model size while maintaining performance.