Cross-Attention › Cross-Attention: Connecting Encoder and Decoder in Transformers

In the previous chapters, we explored encoder-only and decoder-only transformer architectures, each using self-attention to let tokens within a sequence attend to each other. These architectures are powerful for their respective tasks: encoders excel at building rich contextual representations of an input sequence, while decoders excel at generating fluent text one token at a time. But many of the most important tasks in language AI require something different: they require reading one sequence and writing another. Translating English to French, summarizing a long document into a short paragraph, answering a question given a passage of text, converting speech to transcription, or transforming a piece of code into a docstring. All of these tasks share a fundamental structure. You have a source sequence you must understand, and a target sequence you must produce. Cross-attention is the mechanism that makes this possible.

Think of cross-attention as a bridge. On one side stands the encoder, which has carefully read and processed the entire source sequence. On the other side stands the decoder, which is in the middle of generating the target sequence one token at a time. At each step of generation, the decoder needs to look back across the bridge and ask: "Which parts of the source should I pay attention to right now?" Cross-attention is the formal mechanism for answering that question. The decoder formulates a query expressing what it currently needs, the encoder's representations provide the keys and values that answer that query, and the result is a rich, dynamically computed fusion of source and target information.

Understanding cross-attention requires seeing it clearly in contrast to self-attention. In self-attention, a sequence talks to itself: each token in the sequence asks "which other tokens in this same sequence are relevant to me?" The attention matrix is square, and both the questions and the answers come from the same pool of tokens. Cross-attention breaks this symmetry deliberately. Questions still come from the decoder, but the answers come from an entirely different sequence processed by the encoder. This asymmetry is not a complication; it is the core design feature. It is what allows the decoder to incorporate external information while generating text, rather than being confined to what it has already produced.

The mathematical machinery of cross-attention is identical to scaled dot-product attention: QKV representations combined via a softmax-normalized dot product. What differs is only the origin of those three components. This elegance is part of what makes the transformer architecture so modular and extensible. The same attention primitive, applied in different configurations, enables intra-sequence modeling (self-attention), inter-sequence modeling (cross-attention), and constrained generation (masked self-attention). In this chapter, we will understand each aspect of cross-attention in depth: why it is designed the way it is, how the mathematics work, where it lives inside the decoder, how padding and caching affect it, and what its real-world limitations look like.

By the end of this chapter, you will have a complete understanding of cross-attention as both a mathematical object and a practical engineering component. You will be able to implement it from scratch, reason about its behavior during training and inference, and understand why encoder-decoder models like T5, BART, and mBART rely on it as their central information-routing mechanism.