Cross-Attention › Why Queries from Decoder, Keys and Values from Encoder?

The choice of where $Q$, $K$, and $V$ come from is not arbitrary. It reflects the fundamental asymmetry of sequence-to-sequence tasks and a carefully designed information flow. Understanding this design choice deeply will help you appreciate why cross-attention is structured the way it is, and why alternatives do not work as well.

Queries represent what you're looking for. At each decoder position, the model is trying to generate the next token. The query encodes "what information do I need from the source to make this prediction?" The decoder has access to its own context (previous tokens, positional information) and uses this to formulate questions. Think of queries as the decoder's "search intent": they are derived from the decoder's current state, which encodes everything the decoder knows so far about what it has generated. A decoder that has generated "The cat" in French might formulate a query that implicitly asks "what verb follows the cat in the English sentence?"

Keys represent what's available. The encoder has processed the entire source sequence and built representations that capture its meaning. Keys advertise "here's what I know about this position in the source." The encoder output is fixed once computed, so keys remain constant throughout decoding. Each key is a compact descriptor of an encoder position's content, learned to be easily comparable to decoder queries in the shared $d_k$-dimensional space. Because queries and keys live in the same space (they are both projected to $\mathbb{R}^{d_k}$), the dot product $Q_i \cdot K_j$ measures how well the decoder's current need matches the encoder's offering at position $j$.

Values represent what gets transmitted. When attention flows from decoder to encoder, the values determine what information transfers. If the decoder strongly attends to a particular encoder position, that position's value vector contributes heavily to the decoder's output. The value representation can be different from the key representation (they use different projection matrices $W^V$ vs $W^K$), allowing the model to learn that "what identifies a position" and "what information that position transmits" are distinct concerns.

Consider translation from English to French. When generating the French word for "cat," the decoder's query might encode "I need information about an animal noun." The encoder's keys for the position containing "cat" would encode "animal, noun, subject." The high dot product between these creates a strong attention weight, and the encoder's value for "cat" (containing semantic features about cats) flows into the decoder's representation. The separation of key and value means the model can learn to index positions (via keys) and transmit content (via values) in ways that are not necessarily the same representation. This flexibility has proven beneficial in practice.

The alternative design, where the decoder queries its own representations rather than the encoder's, would not accomplish the task. A decoder attending only to itself can only look backward at what it has already generated. It cannot read the source sentence. The key innovation of cross-attention is precisely that it routes a query from one sequence to an answer from a completely different sequence, enabling the integration of source and target information in a fully differentiable way that can be optimized end-to-end.

The visualization shows "chat" (French for "cat") attending strongly to "cat" in the encoder. This alignment emerges naturally from the learned query and key projections, which encode semantic relationships between source and target tokens. In a trained model, these patterns can become surprisingly interpretable: different heads learn to track word alignment, positional correspondence, syntactic relationships, and semantic similarity simultaneously.