Cross-Attention › Self-Attention vs. Cross-Attention

Self-attention and cross-attention share the same mathematical formulation: QKV representations combined with scaled dot-product attention. The critical difference lies in where these components come from. To understand this distinction precisely, we need to see both forms side by side and appreciate what changes when you move from one to the other.

In self-attention, all three projections derive from the same sequence. The intuition is that every token in the sequence can act as a questioner (query), an advertiser (key), and a content provider (value) simultaneously. A token might ask "what context surrounds me?" while also answering the questions of other tokens by advertising its own content. This simultaneous questioning and answering within a single sequence is what lets self-attention capture rich intra-sequence dependencies: subject-verb agreement, pronoun resolution, long-range semantic coherence.

Given an input matrix $X$ containing token representations, self-attention computes QKV representations as:

where:

• $X \in \mathbb{R}^{n \times d}$: the input sequence matrix, with $n$ tokens each represented as a $d$-dimensional vector
• $W^Q \in \mathbb{R}^{d \times d_k}$: the learned query projection matrix
• $W^K \in \mathbb{R}^{d \times d_k}$: the learned key projection matrix
• $W^V \in \mathbb{R}^{d \times d_v}$: the learned value projection matrix
• $Q, K \in \mathbb{R}^{n \times d_k}$: the resulting query and key matrices
• $V \in \mathbb{R}^{n \times d_v}$: the resulting value matrix

Every token attends to every other token (and itself) within the same sequence. The attention matrix is square with shape $(n \times n)$, and each entry $(i, j)$ describes how much token $i$ attends to token $j$.

In cross-attention, the queries come from one sequence while keys and values come from a different sequence. This separation is the key distinction: the decoder generates queries to ask "what do I need?" while the encoder provides keys and values to answer "here's what I have." The decoder cannot answer its own questions by consulting only its own partial output. It needs access to the encoded representation of the source sequence. Cross-attention provides exactly that access, in a learnable and differentiable way.

where:

• $X_{\text{dec}} \in \mathbb{R}^{n_{\text{dec}} \times d}$: the decoder's current representations (what's been generated so far)
• $X_{\text{enc}} \in \mathbb{R}^{n_{\text{enc}} \times d}$: the encoder's output representations (the processed source sequence)
• $W^Q, W^K \in \mathbb{R}^{d \times d_k}$: the learned projection matrices for queries and keys
• $W^V \in \mathbb{R}^{d \times d_v}$: the learned projection matrix for values
• $n_{\text{dec}}$: the number of tokens in the decoder sequence (target length so far)
• $n_{\text{enc}}$: the number of tokens in the encoder sequence (source length)
• $d$: the model dimension (size of each token's representation)

The key insight is that the encoder output appears twice: once for the keys and once for the values. The keys tell the decoder "this is what I offer," and the values tell the decoder "this is what I'll send you if you attend to me." These two roles are separated because the model benefits from being able to project the encoder's information into two different spaces: one for matching (keys against queries) and one for content transmission (values).

This asymmetry is what makes cross-attention powerful for tasks like translation. The decoder asks questions (queries) based on what it has generated so far, and the encoder's representations provide the answers (keys and values). The learned projection matrices $W^Q$, $W^K$, and $W^V$ are trained end-to-end by gradient descent, so the model learns to ask good questions and provide informative answers entirely from the supervision signal (for translation, the signal is simply "produce the correct target sentence given the source").