Multimodal Foundations › Multimodal Architectures

Given modality-specific encoders and the alignment objective, there are several architectural patterns for building multimodal models. These differ in when and how the modalities interact and in what tradeoffs they make between efficiency and representational power.

The simplest architecture processes each modality independently with separate encoders and combines them only at the final embedding level via a similarity computation. This is the architecture used by CLIP.

The visual encoder processes the image through a series of transformer or convolutional layers and produces a single image embedding vector. The text encoder processes the text through a series of transformer layers and produces a single text embedding vector. The two embeddings are compared using cosine similarity. The encoders never attend to each other; they interact only through the scalar similarity score at the very end.

The advantage of this architecture is computational efficiency. Because the encoders never attend to each other, embeddings can be computed independently for any image and any text. This means you can precompute and cache embeddings for an entire database of images or documents offline. At query time, you compute only the query embedding and perform a single nearest-neighbor lookup in the precomputed database. For a database of one million images, this takes milliseconds rather than the minutes it would take if you had to run the full model over every image at query time.

The disadvantage of the dual-encoder is limited cross-modal reasoning. When each modality is processed in isolation, the model cannot use textual context to guide visual attention, or visual context to disambiguate textual meaning. The only interaction is the global similarity score, which is a single number summarizing the relationship between two entire inputs. This global summary is insufficient for tasks that require fine-grained correspondence: the model cannot answer "is the person in this image the one mentioned in this text?" because it only knows the overall similarity, not which visual regions correspond to which textual descriptions.

Cross-encoders process both modalities jointly using cross-attention. In the most common formulation, the image is divided into patches using a ViT-style tokenization, creating a sequence of visual tokens. These visual tokens are concatenated with the text tokens, and a transformer processes the combined sequence with full self-attention across both modalities simultaneously.

This enables rich interaction: text tokens can attend to image patches and image patches can attend to text tokens. A text token representing "cat" can attend strongly to the image patch depicting the cat and weakly to background patches. An image patch depicting a face can attend strongly to any text token describing facial features. This fine-grained cross-modal attention produces much richer representations than the dual-encoder and typically outperforms it on tasks requiring detailed visual understanding or grounding.

The cost of cross-encoders is computational. The representation of each modality depends on the other, so embeddings cannot be precomputed independently. For retrieval over large databases, this means running the cross-encoder over every candidate image-text pair at query time, which is prohibitively expensive at scale. A database of one million images would require one million cross-encoder forward passes per query, taking minutes rather than milliseconds.

Many modern systems therefore use a two-stage pipeline: a dual-encoder for fast coarse retrieval that narrows a database of millions to a shortlist of tens or hundreds of candidates, followed by a cross-encoder reranker that performs detailed reasoning over the shortlist. This combines the scalability of dual-encoders with the accuracy of cross-encoders.

Between the extremes of late fusion (dual-encoder, modalities interact only at the similarity computation) and early fusion (cross-encoder, modalities interact at every layer), several intermediate architectures offer different tradeoffs.

Mid-fusion architectures process each modality separately for several early layers, then concatenate the intermediate representations and process them jointly through later layers. The early layers learn modality-specific features, and the later shared layers learn cross-modal interactions. This balances the speed benefits of independent processing with the expressive power of cross-modal attention.

Perceiver-style cross-attention introduces a fixed set of latent query vectors that attend to features from both modalities via cross-attention. The latent vectors serve as a bottleneck that forces the model to compress cross-modal information into a compact representation. The number of latent vectors is fixed and independent of the input lengths, making the computational cost predictable even for very long inputs. This architecture was popularized by the Perceiver and Flamingo models.

Prefix-based fusion prepends visual embeddings as prefix tokens to a language model, letting the language model to generate text conditioned on visual content. The visual encoder produces a sequence of patch embeddings, a linear projection maps these into the language model's token embedding space, and these projected embeddings are prepended to the text token sequence. The language model can then attend to visual tokens using its standard self-attention. This is the approach used by LLaVA and InstructBLIP, which we examine in later chapters of this part.