Multimodal Foundations › Mathematical Foundations of Cross-Modal Attention

The most powerful form of cross-modal interaction is cross-modal attention, a direct extension of the self-attention mechanism you studied in the attention chapters of this handbook. Understanding it precisely will help you read papers on multimodal transformers.

Recall from our earlier discussion of self-attention that it computes a weighted combination of value vectors, where the weights are determined by the similarity between query vectors and key vectors:

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$$

In self-attention, all three matrices $Q$, $K$, $V$ are derived from the same input sequence by linear projections. For a sequence $X \in \mathbb{R}^{N \times d_\text{model}}$:

$$Q = X W_Q, \quad K = X W_K, \quad V = X W_V$$

Each token queries and answers questions about other tokens in the same sequence. This is appropriate for processing a single modality but does not allow tokens from one modality to attend to tokens from another.

Cross-modal attention modifies this by letting queries come from one modality and keys and values come from the other. Specifically, to attend from text to visual features, we use:

• Queries $Q$ from the text representation: $Q = X_T W_Q$ where $X_T$ is the text token sequence
• Keys $K$ and Values $V$ from the visual representation: $K = X_V W_K$ and $V = X_V W_V$ where $X_V$ is the visual token sequence

The attention output is then:

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$$

where:

• $Q \in \mathbb{R}^{N_T \times d_k}$ is the query matrix from $N_T$ text tokens with query dimension $d_k$
• $K \in \mathbb{R}^{N_V \times d_k}$ is the key matrix from $N_V$ visual tokens with key dimension $d_k$
• $V \in \mathbb{R}^{N_V \times d_v}$ is the value matrix from $N_V$ visual tokens with value dimension $d_v$
• The output has shape $\mathbb{R}^{N_T \times d_v}$, one attended feature per text token

Each text token attends to all visual tokens, creating a weighted combination of visual values that is most relevant to that text token's query. A text token representing "cat" will attend strongly to image patches containing cat-like features and weakly to patches containing irrelevant background content. This soft, differentiable attention mechanism is precisely how cross-modal alignment is computed through the transformer architecture.

The scaling factor $\frac{1}{\sqrt{d_k}}$ prevents the dot products from growing large as $d_k$ increases, which would otherwise cause softmax to saturate in a region of very small gradients. Recall from the attention chapters that this scaling is necessary for stable training, especially with larger key dimensions.

Many architectures apply cross-attention in both directions: text-to-image and image-to-text. This allows each modality to gather information from the other, creating representations where every token is cross-modally informed.

For text-to-image attention, queries come from the text representation and keys and values come from the image representation. Each text token learns which image regions are most relevant to its meaning. The output is a set of image-conditioned text representations where each text token has absorbed relevant visual context.

For image-to-text attention, queries come from the image representation and keys and values come from the text representation. Each image patch learns which text tokens are most relevant to its visual content. The output is a set of text-conditioned image representations where each patch has absorbed relevant linguistic context.

When combined with self-attention within each modality, this bidirectional cross-attention produces richly cross-informed representations. Each token has access to relevant content from both its own modality, through self-attention, and from the other modality, through cross-attention. The interplay between self-attention and cross-attention is what gives cross-encoder architectures their expressive power.