Multimodal Foundations › What Is a Modality?
Language AI HandbookAll chapters Multimodal Foundations › What Is a Modality?One section · Sections in this chapter2 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
A modality is a distinct channel of information with its own structure and statistical properties, represented in a form suited to that channel. The most common modalities in AI research today are:
- Language: Sequences of discrete tokens drawn from a finite vocabulary. You have studied language models extensively throughout this handbook. The discrete token structure, the long-range compositional dependencies, and the absence of spatial layout distinguish text from all other modalities.
- Images: 2D grids of pixel intensities, typically stored as RGB triples. Images have rich spatial structure: nearby pixels tend to be correlated, objects occupy contiguous regions, and scale-invariant features recur across spatial positions. This spatial regularity is why convolutional networks and vision transformers, which we discuss below, were developed specifically for image processing.
- Audio: 1D waveforms sampled at high frequency (commonly 16,000 or 44,100 samples per second), or equivalently 2D spectrograms representing energy across frequency and time. Audio has strong short-range temporal correlations and multi-scale structure from phonemes up through prosody and melody.
- Video: Sequences of image frames at a fixed frame rate, combining spatial structure within each frame with temporal dynamics across frames. Video is the most demanding modality in terms of data volume and alignment complexity.
- Structured data: Tabular records, knowledge graphs, or relational databases with explicit structure and typed attributes. Unlike the other modalities, structured data has no natural sequential or spatial organization.
Each modality has its own natural representation. Images have pixels arranged in a spatial grid where nearby pixels tend to be correlated. Audio has temporal correlations across short windows. Text has long-range dependencies captured by attention. These structural differences mean there is no single obviously correct way to encode all modalities into a common format, and no single neural architecture that is optimal for all of them.
A modality is the source channel (image, audio, text). A representation is the learned numerical encoding of content from that channel. Two inputs from the same modality can have very different representations; two inputs from different modalities can, in principle, share a common representation if they convey the same information. The goal of multimodal learning is precisely to build representations where the second situation occurs as often as possible.
Why Modality Boundaries Matter
The importance of distinguishing modalities is not just taxonomic. Different modalities require different preprocessing pipelines, different encoder architectures, and different inductive biases. An architecture that works well for images, such as a ViT that treats the image as a grid of patches, would be a poor fit for audio without significant modifications. Audio requires careful handling of temporal resolution and frequency, and spectrograms have very different statistics from natural image patches.
Beyond engineering differences, different modalities also carry information at different semantic granularities. A pixel is an extremely low-level unit: its value alone carries almost no meaning without context from surrounding pixels. A word token, by contrast, already carries rich semantic information: the word "dog" activates a dense set of associations before any surrounding context is processed. This disparity in semantic granularity creates basic challenges when trying to learn correspondences between modalities, and we return to it throughout the chapter.