Capability Frontiers › Multimodal Reasoning

Language models began as purely text-based systems, but modern architectures increasingly process images, audio, and video alongside text. This multimodal capability opens new possibilities but also introduces new failure modes.

The integration of vision and language in large models follows a pattern similar to language-only scaling: capable representations emerge from training on large datasets of image-text pairs, with quality improving as model size and data scale increase. The architecture typically involves encoding images into a sequence of visual tokens using a vision encoder (often a vision transformer or CNN), projecting those tokens into the language model's embedding space, and treating the combined sequence of visual and text tokens as input to a single transformer.

This design choice, treating visual tokens as "visual words" that the language model processes alongside text, has proved surprisingly effective. The language model applies the same attention mechanism to visual tokens as to text tokens, allowing visual and linguistic information to be integrated at every layer of processing rather than just at the final output stage.

Vision-language models can describe images, answer questions about visual content, read text in images (optical character recognition), and reason about spatial relationships between objects. At their best, they can interpret complex charts and diagrams, understand visual metaphors and idioms, and integrate visual and textual information to answer questions that neither modality alone could answer. A question like "Based on this photograph, what does the building's architectural style suggest about the period of construction?" requires both perceptual capabilities (recognizing architectural features) and linguistic/cultural knowledge (associating those features with historical periods).

The integration is not perfect, however. Models frequently struggle with fine-grained visual details: counting objects accurately in cluttered images, identifying subtle differences between similar objects (two slightly different dog breeds, two similar paint colors), or tracking the positions of multiple objects simultaneously. They can also hallucinate visual details with the same confident fluency that language-only models use when confabulating facts. A model asked about the number of people in a crowd may give a plausible-sounding but incorrect count, reported with the same confidence as accurate answers.

A recurring challenge is that vision-language models tend to rely heavily on language priors when visual information conflicts with expectations. If an image shows an unusual arrangement of familiar objects, the model may describe the expected arrangement rather than the actual one. A banana resting on a table that is itself balanced on the floor of a swimming pool might be described as "a banana on a table" without acknowledging the unusual context, because that is what the language prior expects. This reflects a fundamental tension in multimodal models: the language component has learned strong priors about how the world typically looks, and these priors can override direct visual evidence.

Spatial reasoning is a particular frontier for multimodal models. Humans effortlessly reason about 3D structure from 2D images, track object positions as viewpoints change, and perform mental rotations to understand how objects relate in space. These capabilities appear to be deeply connected to physical intuition developed through embodied experience of moving through and manipulating a three-dimensional world.

Current vision-language models handle simple spatial reasoning tasks reasonably well. Asking which of two objects is to the left, which is above another, or whether an object is inside or outside a container yields accurate answers in many standard cases. Performance degrades significantly on tasks requiring 3D understanding from 2D images, viewpoint invariance (recognizing the same scene from a different angle), or mental simulation of physical processes. Asking whether a certain object would be visible after rotating a scene, or which of two objects would fall first if a table were tilted, produces much less reliable answers than asking about color or shape.

This gap reflects a deeper challenge: spatial and geometric intuition may require a kind of grounded experience that text and image data cannot fully provide. Knowing what it feels like to be inside a room, to pick up an object, or to watch something fall gives humans intuitions about 3D space that are not easily extracted from static images. Some researchers argue that progress here will require models trained on simulated physical environments, allowing them to develop representations tied to the consequences of actions, not just to the appearance of scenes.

This has motivated a research direction called embodied AI, which trains agents in simulated environments where they can navigate, manipulate objects, and observe the consequences of their actions. The hypothesis is that exposure to interactive experience creates richer spatial representations than passive observation. Early results are promising, with embodied agents showing better performance on spatial reasoning benchmarks than equivalent models trained only on static data.

An important recent finding is that chain-of-thought reasoning extends to multimodal settings. When models are prompted to reason step by step before answering visual questions, accuracy improves substantially, with the improvement following a similar threshold pattern to language-only chain-of-thought. A model asked to interpret a complex chart performs better if it first describes what it sees ("There are three lines representing different model sizes; the y-axis shows accuracy and the x-axis shows training compute") and then answers the question, compared to answering directly.

This suggests that the underlying reasoning mechanism is not purely visual or purely linguistic but operates at a higher level of abstraction. The model can use language to scaffold visual reasoning, breaking a complex perceptual judgment into describable intermediate steps that it can work through sequentially. Language provides a medium for representing visual concepts in a form that the model's reasoning mechanisms can operate on more reliably.

Multimodal chain-of-thought is particularly valuable for tasks that mix visual and quantitative reasoning, such as reading a chart and then computing a derived quantity, or examining a diagram and then answering questions that require understanding spatial relationships. In these hybrid tasks, externalizing the reasoning process appears to reduce compounding errors significantly. The model is forced to make its perceptual judgments explicit before attempting the quantitative reasoning, creating checkpoints where errors in perception can be caught before they propagate into the reasoning phase.