Multimodal Applications › Multimodal Reasoning

The hardest class of multimodal tasks requires real reasoning: integrating information across modalities, following chains of inference, and creating answers that cannot be obtained by simple pattern matching on any single modality. Multimodal reasoning tasks test whether models truly understand the relationship between what they see and what they read, or whether they are using superficial statistical correlations that happen to generalize to the evaluation set.

This distinction between real reasoning and pattern matching is philosophically slippery but practically important. A model that reaches 80% on a geometry benchmark by recognizing that certain visual configurations tend to co-occur with certain answer patterns in the training data may look impressive on aggregate metrics while completely failing on novel configurations it has not seen. Evaluating reasoning requires carefully designed tests that control for these shortcuts and measure real compositional inference.

Scientific figures are among the richest and most challenging multimodal documents. A figure from a machine learning paper might show a training curve with annotations, a bar chart comparing methods, an architecture diagram, or a visualization of learned representations. Understanding the figure requires knowing what each element is, reading the axis labels and tick marks, parsing the legend, and relating the figure's content to the claims in the paper's text.

SciGraphQA and similar benchmarks evaluate whether models can answer questions about figures drawn from scientific papers, including questions that require combining information from the figure and its caption, or reasoning about the experimental design depicted. A question like "Does the proposed method outperform the baseline at all dataset sizes?" requires reading the x-axis (dataset sizes), reading two separate data series (proposed method and baseline), comparing them at each x value, and synthesizing a boolean answer. Models that simply match visual patterns without extracting the quantitative content of plots fail these tasks systematically.

A particularly illuminating failure mode is the qualitative-quantitative gap. A model may correctly identify that "Bar A is taller than Bar B" (qualitative comparison) but fail to answer "By what percentage does Method A outperform the baseline?" (quantitative extraction requiring axis reading and arithmetic). This distinction reveals that visual language models often learn ordinal comparisons without parsing the metric values encoded by visual elements. They understand "more" and "less" but not "how much more" and "how much less." Addressing this gap requires either specialized training on quantitative figure understanding or chain-of-thought prompting that forces the model to explicitly extract numerical values before reasoning about them.

Mathematical reasoning in multimodal settings requires reading equations and diagrams alongside geometric figures, then performing computation or proof-writing based on what is seen. MathVista is a complete benchmark covering geometry problems with diagrams, function plotting (understanding what a graph of a function looks like), and word problems accompanied by relevant images.

The challenge for geometry problems is that the image contains constraints such as angle labels, parallel line markers, labeled lengths, and geometric annotations that must be extracted and combined with the question to set up the correct equations. A model that reads "Find the length of AB" while looking at a triangle labeled with angle measures and one side length must understand the diagram's geometric structure, recognize which theorem applies (sine rule, Pythagorean theorem, similar triangles), and execute the calculation correctly.

This task decomposes into several distinct sub-capabilities: visual parsing (identifying what geometric elements are present and how they are labeled), theorem recognition (mapping the visual configuration to the relevant mathematical fact), and symbolic computation (executing the calculation correctly). Current models tend to be stronger at the first step and weaker at the second and third. They can identify that a triangle is present and read the labels, but they struggle to reliably map visual configurations to the correct theorem, especially when the configuration is non-standard or when multiple theorems could apply.

Current models vary substantially in their geometry problem-solving ability. GPT-4V reaches around 50% on MathVista geometry problems, compared to roughly 60% for humans, suggesting real but limited geometric reasoning. Models specifically trained on mathematical figure data perform better. This shows the importance of domain-specific training data. The performance gap between general models and domain-specific models is larger here than in most other multimodal tasks, which points to a real reasoning deficit rather than merely a data quality issue.

Chain-of-thought prompting sharply improved language model performance on complex reasoning tasks by eliciting intermediate reasoning steps before the final answer. As we discussed in earlier chapters, chain-of-thought works by creating training examples where the reasoning process is made explicit, teaching the model to decompose complex problems into tractable sub-steps rather than attempting to map directly from input to output.

The same idea applied to multimodal settings produces multimodal chain-of-thought, where the model interleaves visual analysis with verbal reasoning steps. For a geometry problem, a multimodal chain-of-thought might proceed as follows. First, the model describes what it sees in the diagram: "The diagram shows triangle ABC with angle A labeled as 30 degrees and side BC labeled as 10." Second, it identifies what is needed: "The question asks for side AB." Third, it identifies the relevant theorem: "I can use the sine rule, which states that AB divided by sin(C) equals BC divided by sin(A)." Fourth, it extracts a missing value from the diagram: "I need angle C. The diagram labels angle B as 90 degrees, and angles in a triangle sum to 180, so angle C equals 180 minus 90 minus 30, which equals 60 degrees." Fifth, it executes the calculation: "AB equals BC times sin(C) divided by sin(A), which equals 10 times sin(60) divided by sin(30), which equals 10 times 0.866 divided by 0.5, which equals 17.32."

Research on multimodal chain-of-thought reveals a disconcerting pattern. Models often produce correct final answers through a combination of pattern matching and arithmetic, without the intermediate reasoning chain being the actual mechanism that produced the correct answer. When researchers modify the chain-of-thought to contain a deliberate error in the intermediate steps, models sometimes still arrive at the correct final answer, which is impossible if they were following the chain. This suggests that the chain-of-thought is, in some cases, a rationalization generated after the fact rather than the actual reasoning process. Evaluating intermediate steps rather than only final answers is therefore an important direction for more reliable assessment of reasoning quality.

Infographics occupy a particularly interesting point in the spectrum of multimodal difficulty. Unlike scientific figures, which follow relatively standardized conventions for axes and series, infographics use varied and creative visual encodings to convey information. A single infographic might combine proportional areas, color gradients, spatial metaphors, and pictograms in a layout designed for visual appeal rather than data accuracy.

Understanding infographics requires the model to first identify the visual encoding scheme being used (is this a bar chart, a map, a timeline, a proportional circle diagram?), then decode the mapping from visual properties to data values, then integrate that data with any accompanying text to answer questions. The InfographicVQA benchmark captures this challenge and reveals that current models lag substantially behind human performance, particularly on questions that require understanding non-standard visual encodings.

The gap is largest for questions that require quantitative extraction from unusual encodings. A model that has seen many bar charts can develop heuristics for reading bar heights. A model that has never encountered a particular type of proportional diagram has no such heuristic and must reason from first principles about what the visual encoding means. This requirement for visual compositional understanding, where understanding new visual encodings from context rather than from memorized patterns, is a significant frontier in multimodal reasoning research.

The final frontier of multimodal applications is multimodal agents: systems that answer questions and take actions in environments that contain visual information. A web agent receives screenshots of a browser and interacts with it by clicking buttons, filling forms, and moving through pages. A computer-use agent receives a screenshot of a desktop and controls the mouse and keyboard to accomplish tasks. A robotics policy model receives camera feeds and proprioceptive sensor readings and outputs motor commands.

The key capability that separates agents from passive question-answering models is grounding: the model must understand that "click the Submit button" requires identifying the physical location of the Submit button in the current screenshot and generating a precise $(x, y)$ coordinate click action rather than a text response. This spatial grounding requirement connects back to the bounding box prediction and region understanding capabilities we discussed in the image understanding chapter. A model that can only answer questions about images but cannot produce spatial coordinates cannot function as a browser agent, regardless of how well it understands what it sees.

Multimodal agents face challenges that are qualitatively different from those in single-turn QA settings. The environment is dynamic: actions change the state of the world, so the model must reason about what the current state is after previous actions rather than treating each observation independently. Errors compound across steps: a wrong click might open the wrong page, making all subsequent actions more likely to fail. The action space is large: a web browser might have hundreds of interactive elements at any given moment, and the model must reliably identify the correct one for the current sub-task. And the episode can be long: completing a multi-step task like "book a flight from New York to London for next Tuesday and pay with the saved credit card" requires maintaining task state across tens of actions.

ScreenAgent and similar systems address these challenges by combining visual grounding (localizing elements in the screenshot using bounding box prediction), planning (decomposing a high-level task into a sequence of sub-steps using chain-of-thought), and self-correction (detecting when an action did not produce the expected result and generating a recovery plan). These systems stand for the current frontier of multimodal AI capability, combining the visual understanding, language reasoning, and action execution capabilities developed across the prior chapters of this part.

The performance gap between current multimodal agents and human-level performance remains large on realistic benchmarks like WebArena and OSWorld. Human annotators complete multi-step web tasks with success rates above 90%. Current multimodal agents reach 20 to 40% on similar tasks. The primary failure modes are grounding errors (clicking the wrong element), planning failures (misunderstanding the task requirements), and recovery failures (getting stuck after an error without a viable replanning strategy). Closing this gap is one of the most active areas in multimodal AI research.