Multimodal Foundations › Multimodal Foundations: Cross-Modal Alignment and Joint Embeddings

Language models, as covered throughout this handbook, are remarkable at reasoning over text. They predict the next token, encode semantic relationships, and generate fluent prose. But the world is not text-only. A doctor reading an X-ray does not translate the image into words before diagnosing. A shopper evaluating a product does not transcribe a written description before deciding. Intelligence in the real world integrates signals from multiple sources simultaneously: vision, language, audio, structured data, and more.

Multimodal AI is the field that asks: how do we build systems that reason over multiple modalities at once? The question sounds natural, but the engineering and mathematical challenges run surprisingly deep. The very things that make each modality powerful in isolation, the spatial structure of images, the temporal dynamics of audio, the discrete compositional structure of text, also make them fundamentally incompatible at the raw representation level. No single architecture trivially handles all of them.

This chapter lays the conceptual and technical foundations you need to understand modern multimodal models. We start from first principles about what modalities are and why combining them is difficult, work through the mathematics of joint embedding spaces and cross-modal alignment, examine the dominant training objectives and architectural patterns, and close with a careful look at the limitations that still constrain current systems.

By the end of this chapter, you will understand why multimodality is difficult and what mathematical machinery supports it. You will also see how familiar ideas from attention and transformer-based representation learning extend into the multimodal setting. The following chapters in this part of the handbook examine CLIP, visual instruction tuning, and full vision-language architectures in detail; everything here prepares you to read them clearly.