Multimodal Foundations › Summary

This chapter laid the groundwork for understanding multimodal AI systems, from the nature of modalities through the mathematics of joint embeddings to the architectural patterns and limitations of current systems. The key ideas to carry forward are:

• Modalities are distinct channels of information (images, text, audio, video) with their own structure and native representations. Each modality requires modality-specific encoders before cross-modal alignment can occur.
• The core challenges of multimodal learning are the representation gap (different native spaces), the alignment problem (correspondence between modalities is fuzzy), the semantic gap (raw signals carry meaning at different levels of abstraction), and data heterogeneity and noise.
• Joint embedding spaces are shared $d$-dimensional vector spaces where semantically similar content from different modalities is placed close together. Cosine similarity measures closeness, and the joint space enables zero-shot retrieval, classification, and cross-modal search without task-specific fine-tuning.
• The modality gap is an emergent property of joint embedding spaces where image and text embeddings occupy different regions of the hypersphere even after successful training. It arises because the contrastive objective enforces relative ordering rather than exact positional overlap.
• Contrastive learning with the symmetric InfoNCE loss is the dominant training objective. It rewards matched pairs for high similarity and penalizes mismatched pairs, using all in-batch pairs as efficient negatives and generating $O(N^2)$ training signal from $N$ pairs. The temperature parameter controls softmax sharpness and is learned jointly with the encoders.
• Dual-encoder architectures process modalities independently and are highly efficient for large-scale retrieval through precomputed embedding indices. Cross-encoders process modalities jointly with attention and enable richer reasoning but cannot precompute embeddings independently, limiting their scalability.
• Cross-modal attention extends self-attention by using queries from one modality and keys and values from the other, letting each token to gather information from the other modality. Bidirectional cross-modal attention, combined with within-modality self-attention, produces fully cross-informed representations.
• Hard negatives are necessary for learning fine-grained semantics. Large batch sizes increase hard negative density naturally; explicit hard negative mining and curriculum learning can improve efficiency further.
• Limitations include the modality gap, difficulty with fine-grained compositional reasoning, inherited data bias, weak spatial grounding, and challenges with temporal reasoning in video. These are active areas of research.

With these foundations in place, you are ready to understand CLIP in the next chapter, which instantiates these ideas at scale with architectural and training decisions that became the blueprint for the field. The chapters that follow examine visual instruction tuning and full vision-language architectures that build on the joint embedding foundation. These models support open-ended generation and visual grounding within multimodal dialogue.