Multimodal Applications: Documents, Medical, Video

The true measure of any technology is what it lets in the real world. Throughout Part L, we have built up the machinery of multimodal models layer by layer: vision encoders that compress images into tokens, projection layers that bridge visual and language spaces, architectures like LLaVA and Flamingo that fuse vision and language, and training strategies that teach models to reason jointly across modalities. Now we arrive at the chapter that ties everything together by asking what all of this is for.

Multimodal applications span a remarkable range of human problems. A radiology model reads chest X-rays and generates structured diagnostic reports. A document understanding system extracts tables and figures alongside text from a scanned PDF and answers questions about the contract buried within it. A video surveillance model watches a construction site and flags safety violations in real time. A multimodal agent receives a screenshot of a broken UI and writes the code to fix it. These are not toy demonstrations. They are deployed systems running today in hospitals and law firms as well as factories and software companies, and their practical consequences are significant.

What makes these applications interesting from a technical standpoint is not just that they use vision and language together. The interesting part is the specific ways they combine modalities to solve problems that neither modality could solve alone. A radiologist reading an X-ray needs both the pixel-level detail of the image and the structured clinical vocabulary of medical literature. A document parser needs to understand that a table's header row defines what the cells below it mean. A video understanding model needs to recognize that an action at time $t$ has consequences that only become visible at time $t + 30$ seconds. Each domain introduces its own constraints, its own characteristic failure modes, and its own specialized design choices that have emerged from years of research and deployment experience.

Understanding these domains deeply is useful for a practical reason: when you build multimodal systems, you will almost certainly be working within one of them. The general-purpose architectures we have studied in previous chapters give the foundation, but successful deployment requires understanding the domain-specific challenges that make each application difficult. The gap between a model that gets 80% on a benchmark and one that is trusted in a hospital or a courtroom is filled with understanding those domain-specific difficulties and addressing them systematically.

This chapter examines four major application domains in depth: document understanding, medical imaging, video understanding, and multimodal reasoning. For each domain, we will cover the unique challenges that domain presents, the specialized architectures and training strategies developed to address those challenges, and the key failure modes and limitations that practitioners must understand before deployment. We close with a code implementation section that builds a practical document understanding pipeline and shows the key concepts in video frame analysis.

Documents present a deceptively difficult multimodal challenge. A PDF or scanned page looks like an image, but understanding it requires integrating at least three types of information: visual layout (where things are on the page), textual content (what the words say), and structural semantics (how the layout signals meaning). A table header is not merely text in a different font. It defines the schema for the data below it. A figure caption is not merely nearby text. It is semantically linked to the image it describes. A section heading signals a topical boundary that affects how all subsequent text should be interpreted.

This document-as-multimodal-object insight distinguishes modern document AI from earlier text extraction pipelines. Traditional approaches treated a document purely as a container for text: use OCR to get the characters out, then feed them to a language model as a flat string. That approach works adequately when the semantic content resides entirely in the words themselves. But in the kinds of documents that matter most in business and legal contexts, layout is not decoration. It is meaning. The difference between "Gross Income" appearing as a form field label versus as a column header versus as a running header determines how the associated number should be interpreted, and that difference is encoded entirely in spatial position and typographic treatment.

The basic insight behind layout-aware document models is that the bounding box of a text token is as semantically informative as the text itself. LayoutLM, introduced by Xu et al. at Microsoft Research in 2020, was the first model to systematically exploit this insight at scale. It extends BERT with two additional embedding types: a 2D position embedding that encodes the $(x_1, y_1, x_2, y_2)$ bounding box coordinates of each token, and an image embedding extracted from the document region corresponding to each token.

The 2D position embedding for a token at bounding box $(x_1, y_1, x_2, y_2)$ is computed as:

where:

• $\text{Emb}_{x_1}, \text{Emb}_{y_1}, \text{Emb}_{x_2}, \text{Emb}_{y_2}$: learned embedding tables for the four bounding box coordinates, each quantized to a fixed range of 0 to 1000 to match a normalized page coordinate system
• $\text{Emb}_{w}, \text{Emb}_{h}$: learned embedding tables for token width and height, which encode information about the visual size of the region

This embedding design has a pleasing property: two tokens with identical text content but different page positions will have different representations, letting the model to distinguish "Total" appearing as a row header from "Total" appearing as a column footer. The model learns from pre-training data that "Total" in the bottom-right of a document tends to co-occur with numerical summaries, while "Total" in a column header tends to co-occur with column-specific computations. The spatial signal bootstraps the semantic signal.

The coordinate quantization step deserves attention. Real document bounding boxes come in arbitrary pixel coordinates that depend on the document's resolution and physical size. By normalizing all coordinates to a fixed 0-1000 range (regardless of whether the original PDF is rendered at 72 dpi or 300 dpi), the model gains resolution invariance: the same document scanned at two different resolutions will produce the same normalized tokens. The 1000-unit range is fine enough to stand for spatial distinctions that matter (a label at $x=30$ versus a value at $x=180$) without requiring a prohibitively large embedding table.

LayoutLMv2 extended the original design by incorporating patch-level visual features from a convolutional neural network directly into the transformer's attention mechanism. Rather than treating text and image as separate streams that are concatenated at the beginning, LayoutLMv2 allows text tokens to attend to image patches at every transformer layer, letting richer cross-modal fusion. This matters for cases where the visual appearance of a field (font weight, background shading, border style) disambiguates its semantic role in ways that text alone cannot.

LayoutLMv3, the current state of the art in the series, went further by treating masked image patches and masked text tokens symmetrically during pre-training. Instead of separate objectives for text and vision, LayoutLMv3 uses a single masked multimodal modeling objective that randomly masks both text tokens and image patches and asks the model to reconstruct them. This forces the model to develop representations that integrate both modalities rather than solving the text and image tasks with separate pathways.

Document question answering (DocQA) is the task of answering natural language questions about a document image. A user uploads a scanned invoice and asks "What is the total amount due?" The model must locate the relevant region of the document, parse the number, and return it as a structured answer.

The challenge is more subtle than it first appears. The document may be a scanned photograph rather than a digital PDF, introducing OCR errors and skewed text. It may have multiple pages with cross-page references (a summary that reads "see Schedule A on page 7"). The answer may be in a table cell whose meaning depends on headers that appear several rows above, requiring the model to understand column and row scope. The question may require arithmetic ("What is the sum of the line items?") rather than simple extraction, adding a reasoning layer on top of the understanding layer.

Modern DocQA systems typically combine OCR (to extract token positions and text), a layout-aware encoder (to stand for the combined text and layout and image), and a generation or span-prediction decoder. For extractive tasks, the decoder predicts start and end positions within the token sequence. For abstractive tasks, the decoder generates free text conditioned on the encoded document representation.

The DocVQA benchmark, which tests question answering on a diverse collection of business documents including forms and invoices as well as tables, has seen large progress since 2020. Early layout-aware models achieved around 70% character-level F1 on DocVQA. Current state-of-the-art systems exceed 95%, which is approaching human annotator agreement on this benchmark. However, this progress should be interpreted carefully: DocVQA favors extractive questions with unambiguous answers, and performance on more open-ended document tasks remains substantially lower.

Tables are ubiquitous in documents and stand for one of the hardest subproblems in document understanding. A table has structure at multiple levels: the cell level (a single value), the row level (a record), the column level (an attribute), and the table level (a relation between entities). Understanding a cell requires knowing what its column header says, what its row header says, and how those headers were merged or spanned across adjacent cells.

Consider a financial table where "Q1", "Q2", "Q3", "Q4" are column headers and "Revenue", "Operating Income", "Net Income" are row headers. The cell where the Q3 column meets the Operating Income row has meaning only in the context of both headers simultaneously. A model that reads this table as a flat token sequence, ignoring the two-dimensional grid structure, cannot reliably extract "operating income for Q3" because the text "43.2" by itself is meaningless.

Table detection models first locate the bounding box of the table within the document image using object detection approaches. Table structure recognition models then parse the internal grid, identifying rows and columns as well as spanning cells. This is harder than it appears because real-world tables use visual cues such as borders and shading alongside whitespace to signal structure, and these visual conventions vary widely across document types, time periods, and publishers. A table in a government form from 1985 uses visual conventions that differ systematically from a table in a modern SaaS dashboard export.

Once structure is recognized, table question answering requires compositional reasoning. Questions like "Which product had the highest revenue in Q3?" require identifying the Revenue column, scanning its Q3 values, finding the maximum, and looking up the corresponding product name in another column. This kind of multi-step compositional reasoning extends beyond what a language model trained on text alone handles well. The precise spatial grounding that layout-aware encoders give is necessary but not sufficient: the model also needs the reasoning capability to chain together the column lookup, the comparison, and the row retrieval in the correct order. This is one reason hybrid symbolic-neural approaches that combine learned neural representations with structured table operations remain active research.

Forms are a specialized but practically important document type. A structured form is designed around a specific information schema: a tax form has fields for income and deductions plus credits; a medical intake form covers symptoms and medications plus allergies; an employment application covers work history and education plus references.

Extracting information from forms at scale requires understanding the relationship between field labels and field values, handling handwritten and typed content mixed on the same form, and dealing with forms that deviate from their template (crossed-out values, corrections, attachments). Models trained on form understanding, such as the FUNSD (Form Understanding in Noisy Scanned Documents) benchmark, learn to classify tokens into question and answer categories alongside header and other categories, then link questions to their corresponding answers.

The practical value of form understanding is enormous. Insurance companies process millions of claims forms annually. Healthcare systems process patient intake and consent forms. Government agencies process tax returns and benefit applications. Even modest improvements in automated understanding accuracy translate to significant labor savings and processing speed increases in these contexts.

Medical imaging is both the highest-stakes and the most technically demanding application domain for multimodal AI. The stakes are high because errors can directly cause patient harm, whether through missed diagnoses that delay treatment or false positive findings that lead to unnecessary procedures. The technical demands are high because medical images have unusual properties: they are often grayscale or pseudocolored rather than RGB, they encode three-dimensional anatomy in two-dimensional projections, and the diagnostically relevant features may be subtle variations in pixel intensity that a standard ImageNet-pretrained model would never have learned to detect.

The mismatch between natural image statistics and medical image statistics is not a minor technical detail. It is a basic challenge that shapes how medical AI systems must be built and validated. A vision encoder pre-trained on ImageNet has learned to detect edges and textures alongside object boundaries at the scale of everyday objects: a dog, a chair, a car. A chest X-ray contains none of these. What it contains are subtle variations in radiodensity that correspond to lung tissue, vascular structures, and possible pathologies, all projected onto a flat 2D plane from a three-dimensional anatomy. Learning to read these signals requires either extensive domain-specific pre-training or careful fine-tuning with curated medical data.

Radiology report generation is the task of automatically creating a structured medical report from a chest X-ray or CT scan. A radiologist writes such reports by systematically examining the image for findings (observable abnormalities), then formulating an impression (clinical interpretation) that synthesizes the findings into a diagnosis and recommendation.

The model must solve several sub-problems simultaneously:

• Finding detection: Which regions of the image contain abnormalities?
• Finding description: How should each abnormality be characterized in clinical language?
• Severity assessment: Is this finding incidental or clinically significant?
• Report structure: How should findings be organized into sections (Findings, Impression, Recommendation)?

Early approaches treated this as image captioning and applied standard CNN-LSTM architectures. These models generated fluent text but often hallucinated findings not present in the image or missed real findings that were subtle. Hallucination is especially dangerous in medicine: a model that generates "No evidence of pneumothorax" when one is visible creates real patient risk. This is qualitatively different from hallucination in a general-purpose chatbot, where the consequences of a confident but wrong statement are usually minor. In radiology, the consequences can be life-threatening.

Modern approaches address hallucination through several strategies. Grounded report generation requires the model to attend to specific image regions for each finding it reports, which makes it harder to confabulate content that is not visually supported. The model must, in effect, show its work: for each clinical term it generates, there must be an image region it can point to. Factual consistency verification adds a secondary model that reads the generated report and the image and flags claims that cannot be grounded in the visual evidence. Retrieval-augmented approaches retrieve similar cases from a training database and use their associated reports as conditioning context, encouraging the model to stay close to verified clinical language rather than generating novel combinations that may be clinically incoherent.

The mathematical formulation of grounded generation introduces an attention mask conditioned on image regions. For a token $y_t$ in the generated report, the attention weight over image patch $i$ is:

where $e_{t,i} = \mathbf{q}_t^\top \mathbf{k}_i$ is the dot product between the query vector derived from the current decoding state and the key vector for patch $i$.

where:

• $\mathbf{q}_t \in \mathbb{R}^d$: query vector computed from the decoder's hidden state at step $t$
• $\mathbf{k}_i \in \mathbb{R}^d$: key vector for image patch $i$, encoded by the vision encoder
• $\alpha_{t,i}$: the attention weight, representing how much the model "looks at" patch $i$ when generating token $t$

This mechanism produces an implicit saliency map for each generated token: we can visualize which image regions the model attended to when writing "pneumothorax" or "consolidation," and verify that these align with radiologically plausible locations. When they do not, that misalignment is evidence of hallucination, and automated grounding verification can use this signal to flag suspect claims before they reach a clinician.

A key advance in medical vision-language models has been the development of radiology-specific pre-training strategies that use the enormous amount of paired image-report data available in hospital systems. CheXpert, MIMIC-CXR, and similar datasets contain hundreds of thousands of chest X-rays with associated radiology reports, covering a wide range of pathologies and patient populations. These datasets, when combined with contrastive pre-training objectives similar to CLIP but adapted for the medical domain, produce vision encoders that are far more sensitive to clinically relevant image features than general-purpose encoders.

The BioViL-T model, for example, applies a temporal contrastive objective that trains the model to align changes in a patient's X-rays over time with changes in the associated radiology reports. This temporal grounding teaches the model concepts like "worsening infiltrate" or "resolving effusion" that require comparing two images, not just reading one in isolation. Such temporal reasoning is needed for real clinical workflows, where radiologists routinely compare the current study to prior studies to detect interval changes.

Pathology slides present a fundamentally different challenge from radiology. A whole-slide image (WSI) of a tissue biopsy may be 100,000 by 100,000 pixels at full resolution, far too large to process as a single input to a transformer. Pathologists examine slides at multiple magnifications: low magnification to identify tissue architecture and high magnification to examine individual cells.

Multiple-instance learning (MIL) is the dominant paradigm for WSI analysis. The slide is divided into thousands of small patches (typically 256x256 pixels at a fixed magnification). Each patch is independently encoded by a vision encoder pre-trained on pathology data, such as the CONCH or UNI models trained on millions of pathology images from diverse tissue types and staining protocols. The patch embeddings are then aggregated into a slide-level representation using attention pooling:

where:

• $N$: total number of patches from the slide
• $\mathbf{h}_i \in \mathbb{R}^d$: the encoded representation of patch $i$
• $a_i$: the attention weight for patch $i$, computed by a small neural network that assigns higher weight to diagnostically relevant patches
• $\mathbf{z}_{\text{slide}} \in \mathbb{R}^d$: the aggregated slide-level representation used for downstream classification or generation

The attention weights $a_i$ give interpretability, which is especially important in pathology where clinicians need to understand which tissue regions informed a model's conclusion. The patches with the highest weights are the ones the model found most informative, and visualizing these regions allows pathologists to verify whether the model's reasoning aligns with known pathological features such as tumor cells, mitotic figures, stromal invasion, or necrosis.

A necessary challenge in pathology is the variability introduced by staining. Hematoxylin and eosin (H&E) staining, which is the standard in most pathology labs, produces images where cell nuclei appear purple and cytoplasm appears pink. But the exact shade varies with staining protocol, tissue thickness, and scanner settings, and these variations can shift feature distributions in ways that cause models trained at one institution to fail at another. Stain normalization preprocessing and stain-augmented training are common mitigations, but achieving reliable cross-institution generalization remains an open problem.

A key advantage of multimodal AI in medicine is the ability to integrate imaging with structured clinical data. A patient's chest X-ray alone is not the whole picture: their age, smoking history, prior diagnoses, and current lab values all modulate how imaging findings should be interpreted. A new opacity in a patient with a history of lung cancer has a very different clinical significance than the same opacity in a healthy 25-year-old. A consolidation in a patient with fever and a high white blood cell count points toward pneumonia; the same consolidation in an afebrile patient with weight loss points toward malignancy.

Models that fuse imaging with electronic health record (EHR) data encode the clinical notes using a clinical language model such as BioBERT or ClinicalBERT, which have been pre-trained on large corpora of medical text to understand clinical abbreviations, drug names, and diagnostic terminology. The imaging modality is encoded using a radiology-specific vision encoder. The two representations are then combined using cross-attention before the final classification or generation head.

The challenge is handling the heterogeneity of clinical data. Notes may be long, reaching thousands of tokens for complex patients with multi-year histories. They contain abbreviations that vary by institution and clinician (SOB for shortness of breath, CP for chest pain, RLL for right lower lobe). They reference prior visits not in the current context window, requiring either longer context handling or efficient retrieval of relevant historical content. They mix structured and unstructured content, alternating between narrative prose and structured lists of medications and allergies alongside problem lists.

Despite these challenges, fusion models consistently outperform image-only models on downstream clinical tasks. This shows that clinical context gives signal that cannot be inferred from images alone. The performance gain is especially pronounced for tasks where clinical context is directly diagnostic, such as distinguishing malignant from benign pulmonary nodules in patients with known cancer history.

Video introduces the temporal dimension that static images lack. A single frame cannot tell you that a person is running (versus standing with one leg raised), that a car is decelerating (versus at rest), or that a conversation is becoming heated (versus a momentary pause). Understanding video requires modeling change over time, and the computational cost of doing so at scale is substantial.

The temporal dimension does not simply add one more feature to process. It introduces a qualitatively new type of information: causality. Two frames that look similar can be part of very different events depending on what happened in the frames between them. A person with arms raised might be celebrating a goal, surrendering, or reaching for something on a high shelf. Disambiguating these requires understanding the sequence of events leading up to the current frame alongside the current frame itself. This dependency on temporal context fundamentally shapes how video understanding systems must be designed and evaluated.

Early video understanding models extended 2D convolutional networks to 3D by adding a temporal convolution dimension, creating 3D ConvNets (C3D, I3D) that could detect spatiotemporal features like motion edges and action-specific appearance changes. These architectures worked for action recognition tasks on short clips but were computationally expensive and required large labeled video datasets to train from scratch. Their parameter efficiency was poor: a 3D convolution kernel has one more dimension than its 2D counterpart, multiplying parameter count proportionally with kernel temporal size.

The transformer era brought video transformers (ViViT, TimeSformer) that tokenize video frames into patches and apply attention across both spatial and temporal dimensions. This is conceptually straightforward but computationally challenging. Full spatiotemporal attention computes attention between every patch in every frame and every other patch in every frame, which has complexity $O((N \cdot T)^2)$ where $N$ is the number of patches per frame and $T$ is the number of frames. For a 10-second video at 30 frames per second with 14x14 patches per frame, this produces roughly $176$ billion attention pairs, which is completely intractable on current hardware.

Factored attention addresses this by separating spatial and temporal attention into alternating layers. A spatial attention layer computes attention only among patches within the same frame, independently for each frame. A temporal attention layer computes attention only among the same spatial position across different frames, independently for each spatial location. This reduces complexity to $O(N^2 T + N T^2)$, which is manageable for videos of moderate length.

The factored attention formula for a patch at spatial position $(i, j)$ in frame $t$ is computed in two stages:

where:

• $\mathbf{q}_{i,j,t}$: the query vector for the patch at spatial position $(i,j)$ in frame $t$
• The spatial attention equation: keys and values come from all positions $(i', j')$ within the same frame $t$, letting the model to integrate context within a single moment
• The temporal attention equation: keys and values come from the same spatial position $(i,j)$ across all frames $t'$, letting the model to track how a specific location changes over time

The factored approach makes an implicit assumption that has both a cost and a benefit. The cost is that spatial context and temporal context cannot be jointly considered in a single attention operation: a patch cannot attend simultaneously to a different spatial location and a different time. The benefit is that this factored structure matches certain natural properties of video. Spatial relationships (object layout, scene structure) change relatively slowly, while temporal relationships (motion, causality) can be tracked by following specific locations through time. By separating these two concerns, factored attention allocates its computational budget efficiently.

Before a video model can process any frames, it must decide which frames to process. The naive answer is to sample uniformly, but uniform sampling has a significant weakness: most of the information in many videos is concentrated in short temporal windows surrounding key events, with long stretches of relatively static content in between. A 30-second cooking video might have 20 seconds of chopping (repetitive, uniform content) and 10 seconds of necessary technique (varied, informative content). Uniform sampling allocates frames equally across both, under-representing the necessary moments.

Content-adaptive sampling strategies try to allocate frame budget to moments of high information content. The simplest version computes the L2 distance between consecutive frame embeddings and samples more densely near local maxima of this distance signal. A scene cut produces a large spike in frame-to-frame distance; a period of slow camera pan produces low distance values. Sampling adaptively near these spikes captures scene transitions, action onsets, and key events more reliably than uniform sampling.

More advanced approaches use learned keyframe extractors that are trained on downstream task supervision. For a video question answering system, the keyframe extractor learns to select frames that are most informative for answering the types of questions in the training set. This is a form of differentiable frame selection that jointly optimizes keyframe identification and question answering, though training such systems end-to-end requires careful design to keep gradients flowing through the discrete selection operation.

Video question answering (VideoQA) extends image-based VQA to the temporal domain. A model watches a video clip and answers questions that may require understanding temporal sequences, counting events, or reasoning about causality. "What happened immediately before the person fell?" requires identifying the temporal event boundaries and reasoning about the causal chain. "How many times did the player touch the ball?" requires counting events across the full video duration, not just identifying that an event happened at all.

Benchmarks like NExT-QA and STAR specifically target causal and temporal reasoning in video, in contrast to earlier benchmarks like MSVD-QA and MSRVTT-QA that could often be answered by looking at a single representative frame. The causal and temporal questions in NExT-QA require models to understand that some questions can only be answered by examining how the video changes over time, not just what it looks like at any moment.

Modern VideoQA models typically sample a fixed number of frames from the video (often 8 to 32, depending on video length and computational budget), encode each frame independently using a vision encoder, and then use cross-frame attention to aggregate temporal context before generating the answer. The cross-frame attention step is where temporal reasoning happens: the model learns to attend to the frames most relevant to the question and to reason about their temporal ordering.

A revealing diagnostic for VideoQA systems is comparing performance on questions answerable from a single frame versus questions requiring temporal integration. Many early VideoQA models showed negligible improvement from temporal modeling, suggesting they were largely solving the task from static frame content. Models specifically designed for temporal reasoning, with attention mechanisms that operate across frames and training objectives that require temporal understanding, show larger improvements on the temporally-demanding subsets of these benchmarks.

Most video understanding benchmarks focus on short clips of 10 to 30 seconds. Real-world applications often require understanding videos that are minutes or hours long: analyzing a surgical procedure, summarizing a lecture, understanding a documentary, reviewing surveillance footage. At these scales, even sampling 32 frames per video misses the majority of content, and the important moments may be scattered throughout the full duration with no obvious prior way to identify their location.

Long-form video understanding is an active research area with several competing approaches. Memory-augmented models maintain a compact summary of previously seen frames that is updated as new frames arrive. Each new frame is processed in the context of this running summary, which allows the model to integrate information from across the full video duration without storing all frames in memory. The challenge is that the memory must be compact enough to fit in the context window but rich enough to preserve information that will be relevant for future reasoning.

Hierarchical models first perform coarse-grained segment-level analysis to identify relevant portions of the video, then apply fine-grained analysis only to those portions. For a question about a specific event in a long video, the coarse-grained pass might identify the 2-minute segment most likely to contain the relevant content, and the fine-grained pass then processes only those 2 minutes at full resolution. This hierarchical strategy can reach near-full-resolution understanding of specific moments with a small fraction of the computational cost of processing the entire video.

Retrieval-based approaches embed all frames in a shared embedding space and use approximate nearest-neighbor search to retrieve the frames most semantically similar to a given query before running the full model. For a query about a specific event, retrieval narrows the processing to a small subset of relevant frames, sharply reducing the effective computational cost. The Goldfish model and similar work on Needle-in-a-Haystack video benchmarks show that retrieval-based approaches reach strong performance on queries requiring specific information in hours-long videos, while reducing computation by orders of magnitude compared to processing all frames uniformly.

The hardest class of multimodal tasks requires real reasoning: integrating information across modalities, following chains of inference, and creating answers that cannot be obtained by simple pattern matching on any single modality. Multimodal reasoning tasks test whether models truly understand the relationship between what they see and what they read, or whether they are using superficial statistical correlations that happen to generalize to the evaluation set.

This distinction between real reasoning and pattern matching is philosophically slippery but practically important. A model that reaches 80% on a geometry benchmark by recognizing that certain visual configurations tend to co-occur with certain answer patterns in the training data may look impressive on aggregate metrics while completely failing on novel configurations it has not seen. Evaluating reasoning requires carefully designed tests that control for these shortcuts and measure real compositional inference.

Scientific figures are among the richest and most challenging multimodal documents. A figure from a machine learning paper might show a training curve with annotations, a bar chart comparing methods, an architecture diagram, or a visualization of learned representations. Understanding the figure requires knowing what each element is, reading the axis labels and tick marks, parsing the legend, and relating the figure's content to the claims in the paper's text.

SciGraphQA and similar benchmarks evaluate whether models can answer questions about figures drawn from scientific papers, including questions that require combining information from the figure and its caption, or reasoning about the experimental design depicted. A question like "Does the proposed method outperform the baseline at all dataset sizes?" requires reading the x-axis (dataset sizes), reading two separate data series (proposed method and baseline), comparing them at each x value, and synthesizing a boolean answer. Models that simply match visual patterns without extracting the quantitative content of plots fail these tasks systematically.

A particularly illuminating failure mode is the qualitative-quantitative gap. A model may correctly identify that "Bar A is taller than Bar B" (qualitative comparison) but fail to answer "By what percentage does Method A outperform the baseline?" (quantitative extraction requiring axis reading and arithmetic). This distinction reveals that visual language models often learn ordinal comparisons without parsing the metric values encoded by visual elements. They understand "more" and "less" but not "how much more" and "how much less." Addressing this gap requires either specialized training on quantitative figure understanding or chain-of-thought prompting that forces the model to explicitly extract numerical values before reasoning about them.

Mathematical reasoning in multimodal settings requires reading equations and diagrams alongside geometric figures, then performing computation or proof-writing based on what is seen. MathVista is a complete benchmark covering geometry problems with diagrams, function plotting (understanding what a graph of a function looks like), and word problems accompanied by relevant images.

The challenge for geometry problems is that the image contains constraints such as angle labels, parallel line markers, labeled lengths, and geometric annotations that must be extracted and combined with the question to set up the correct equations. A model that reads "Find the length of AB" while looking at a triangle labeled with angle measures and one side length must understand the diagram's geometric structure, recognize which theorem applies (sine rule, Pythagorean theorem, similar triangles), and execute the calculation correctly.

This task decomposes into several distinct sub-capabilities: visual parsing (identifying what geometric elements are present and how they are labeled), theorem recognition (mapping the visual configuration to the relevant mathematical fact), and symbolic computation (executing the calculation correctly). Current models tend to be stronger at the first step and weaker at the second and third. They can identify that a triangle is present and read the labels, but they struggle to reliably map visual configurations to the correct theorem, especially when the configuration is non-standard or when multiple theorems could apply.

Current models vary substantially in their geometry problem-solving ability. GPT-4V reaches around 50% on MathVista geometry problems, compared to roughly 60% for humans, suggesting real but limited geometric reasoning. Models specifically trained on mathematical figure data perform better. This shows the importance of domain-specific training data. The performance gap between general models and domain-specific models is larger here than in most other multimodal tasks, which points to a real reasoning deficit rather than merely a data quality issue.

Chain-of-thought prompting sharply improved language model performance on complex reasoning tasks by eliciting intermediate reasoning steps before the final answer. As we discussed in earlier chapters, chain-of-thought works by creating training examples where the reasoning process is made explicit, teaching the model to decompose complex problems into tractable sub-steps rather than attempting to map directly from input to output.

The same idea applied to multimodal settings produces multimodal chain-of-thought, where the model interleaves visual analysis with verbal reasoning steps. For a geometry problem, a multimodal chain-of-thought might proceed as follows. First, the model describes what it sees in the diagram: "The diagram shows triangle ABC with angle A labeled as 30 degrees and side BC labeled as 10." Second, it identifies what is needed: "The question asks for side AB." Third, it identifies the relevant theorem: "I can use the sine rule, which states that AB divided by sin(C) equals BC divided by sin(A)." Fourth, it extracts a missing value from the diagram: "I need angle C. The diagram labels angle B as 90 degrees, and angles in a triangle sum to 180, so angle C equals 180 minus 90 minus 30, which equals 60 degrees." Fifth, it executes the calculation: "AB equals BC times sin(C) divided by sin(A), which equals 10 times sin(60) divided by sin(30), which equals 10 times 0.866 divided by 0.5, which equals 17.32."

Research on multimodal chain-of-thought reveals a disconcerting pattern. Models often produce correct final answers through a combination of pattern matching and arithmetic, without the intermediate reasoning chain being the actual mechanism that produced the correct answer. When researchers modify the chain-of-thought to contain a deliberate error in the intermediate steps, models sometimes still arrive at the correct final answer, which is impossible if they were following the chain. This suggests that the chain-of-thought is, in some cases, a rationalization generated after the fact rather than the actual reasoning process. Evaluating intermediate steps rather than only final answers is therefore an important direction for more reliable assessment of reasoning quality.

Infographics occupy a particularly interesting point in the spectrum of multimodal difficulty. Unlike scientific figures, which follow relatively standardized conventions for axes and series, infographics use varied and creative visual encodings to convey information. A single infographic might combine proportional areas, color gradients, spatial metaphors, and pictograms in a layout designed for visual appeal rather than data accuracy.

Understanding infographics requires the model to first identify the visual encoding scheme being used (is this a bar chart, a map, a timeline, a proportional circle diagram?), then decode the mapping from visual properties to data values, then integrate that data with any accompanying text to answer questions. The InfographicVQA benchmark captures this challenge and reveals that current models lag substantially behind human performance, particularly on questions that require understanding non-standard visual encodings.

The gap is largest for questions that require quantitative extraction from unusual encodings. A model that has seen many bar charts can develop heuristics for reading bar heights. A model that has never encountered a particular type of proportional diagram has no such heuristic and must reason from first principles about what the visual encoding means. This requirement for visual compositional understanding, where understanding new visual encodings from context rather than from memorized patterns, is a significant frontier in multimodal reasoning research.

The final frontier of multimodal applications is multimodal agents: systems that answer questions and take actions in environments that contain visual information. A web agent receives screenshots of a browser and interacts with it by clicking buttons, filling forms, and moving through pages. A computer-use agent receives a screenshot of a desktop and controls the mouse and keyboard to accomplish tasks. A robotics policy model receives camera feeds and proprioceptive sensor readings and outputs motor commands.

The key capability that separates agents from passive question-answering models is grounding: the model must understand that "click the Submit button" requires identifying the physical location of the Submit button in the current screenshot and generating a precise $(x, y)$ coordinate click action rather than a text response. This spatial grounding requirement connects back to the bounding box prediction and region understanding capabilities we discussed in the image understanding chapter. A model that can only answer questions about images but cannot produce spatial coordinates cannot function as a browser agent, regardless of how well it understands what it sees.

Multimodal agents face challenges that are qualitatively different from those in single-turn QA settings. The environment is dynamic: actions change the state of the world, so the model must reason about what the current state is after previous actions rather than treating each observation independently. Errors compound across steps: a wrong click might open the wrong page, making all subsequent actions more likely to fail. The action space is large: a web browser might have hundreds of interactive elements at any given moment, and the model must reliably identify the correct one for the current sub-task. And the episode can be long: completing a multi-step task like "book a flight from New York to London for next Tuesday and pay with the saved credit card" requires maintaining task state across tens of actions.

ScreenAgent and similar systems address these challenges by combining visual grounding (localizing elements in the screenshot using bounding box prediction), planning (decomposing a high-level task into a sequence of sub-steps using chain-of-thought), and self-correction (detecting when an action did not produce the expected result and generating a recovery plan). These systems stand for the current frontier of multimodal AI capability, combining the visual understanding, language reasoning, and action execution capabilities developed across the prior chapters of this part.

The performance gap between current multimodal agents and human-level performance remains large on realistic benchmarks like WebArena and OSWorld. Human annotators complete multi-step web tasks with success rates above 90%. Current multimodal agents reach 20 to 40% on similar tasks. The primary failure modes are grounding errors (clicking the wrong element), planning failures (misunderstanding the task requirements), and recovery failures (getting stuck after an error without a viable replanning strategy). Closing this gap is one of the most active areas in multimodal AI research.

This section builds a practical document understanding pipeline: we will load a document image, run layout-aware OCR to extract text with bounding boxes, and implement a simple document question answering system that uses spatial context to answer queries about document structure. We will also implement the video frame sampling comparison that shows the key tradeoffs in temporal video analysis.

We start by installing the required packages and setting up the processing environment.

These visualizations confirm the core intuition behind layout-aware document understanding: spatial position and size carry semantic meaning. The header occupies the full page width (high mean width, high variance from short to long text elements), field labels are narrow and consistently positioned, and total section tokens cluster in the bottom-right quadrant of the document with characteristic width patterns that distinguish labels from values.

This section shows the key computational challenge in video understanding: selecting informative frames from a video sequence and analyzing how content changes over time. We simulate a video as a sequence of frames with controlled temporal dynamics, letting us to measure how well each sampling strategy covers key events.

The code models a video as a sequence of feature vectors rather than actual pixels. In a real system, each feature vector would be the output of a vision encoder applied to a video frame. The temporal dynamics, including scene transitions and periods of slow change, are captured by controlling how the simulated feature distributions change across time.

The plot confirms the intuition behind adaptive sampling: scene transitions create spikes in the frame-to-frame distance signal, and a sampling strategy that selects frames near those spikes captures the most semantically important moments. For tasks that require understanding what happens at a transition (an action starting, a new speaker appearing, a scene cut), this translates directly to better model performance. Uniform sampling is simpler and requires no feature computation, but it systematically under-is the high-information transition moments that contain most of the answers to temporal reasoning questions.

Multimodal applications have made extraordinary progress over the past five years, but several important limitations should temper enthusiasm for real-world deployment. Understanding these limitations is not pessimism. It is the prerequisite for building trustworthy systems.

Hallucination is perhaps the most significant challenge for multimodal systems in high-stakes applications. Models trained to generate fluent text can produce confident, well-formed descriptions of image content that is absent. In document understanding, this means a model might report an invoice total that does not appear in the document. In medical imaging, it might report a finding that is not visible in the scan. The fluency of the output makes hallucination hard to detect without manual verification, because a hallucinated sentence reads exactly like a factual one.

Several evaluation benchmarks specifically target hallucination in multimodal models. POPE (Polling-based Object Probing Evaluation) asks yes/no questions about object presence in images, where some objects are absent. Hallucination rates on POPE vary widely across models, from around 20% for weaker models to under 5% for the strongest current systems. However, even 5% error rates may be unacceptable in medical or legal contexts where errors have real consequences.

The basic source of hallucination is that language priors are stronger than visual signals for rare or ambiguous content. If an image shows an unusual pattern that superficially resembles a common one, the model's language prior pulls it toward describing the common pattern, overriding the actual visual evidence. A radiological shadow in an unusual position might be described using the vocabulary for a common finding it resembles, even though the clinical interpretation is completely different. Grounded generation, contrastive training, and post-hoc verification are partial mitigations, but no complete solution exists. The basic tension between fluency (which benefits from strong language priors) and factual accuracy (which requires deferring to visual evidence over prior expectations) has no easy resolution.

Models trained on natural images often fail to generalize to specialized domains like medical imaging, satellite imagery, or scientific figures. A vision encoder pre-trained on ImageNet has learned features useful for recognizing objects in everyday photographs, but medical images have different statistics (often grayscale, containing subtle intensity gradients), different scales of interest (a tumor may be a few millimeters in a 512x512 CT slice), and different semantic structure (the background is tissue, not sky or grass).

Fine-tuning on domain-specific data is needed but requires careful data collection. Medical imaging data is subject to privacy regulations and institutional review board requirements that limit data sharing across institutions. Scientific figure data requires licensing agreements with publishers. The result is that domain-specific datasets are smaller and harder to access than general web data, and models trained on them may overfit to specific formats and institutions as well as patient populations represented in the training set. A model that reaches high performance on data from one hospital system may perform substantially worse on data from a different system with different imaging protocols and patient demographics.

Multimodal models are substantially more expensive to run than text-only models. A vision encoder processing a 224x224 image adds hundreds of millions of multiply-accumulate operations on top of the base language model computation. Video models compound this further: processing 32 frames means 32 separate vision encoder passes plus the cross-frame attention overhead.

For real-time applications such as live video analysis and interactive document exploration, inference latency becomes a hard constraint. The techniques developed to reduce language model inference cost, including quantization, speculative decoding, and KV caching, extend to multimodal models but with additional complications. Vision encoder outputs are high-dimensional and cannot always be cached across requests. The visual token sequence added to the context can be much longer than typical text prompts, increasing memory bandwidth requirements. And the multi-modal context means that both the visual and textual components must be resident in GPU memory simultaneously.

Efficient vision encoding is an active area of research. Token compression techniques that reduce the number of visual tokens passed to the language model (through pooling, pruning, or learned merging) can substantially reduce latency without proportional quality loss. The tradeoff between visual resolution (more tokens, more detail) and computational efficiency (fewer tokens, faster inference) is a key design choice for production systems.

Current multimodal benchmarks have significant gaps relative to real-world task requirements. DocVQA evaluates single-page document understanding; real enterprise document workflows involve multi-page documents with cross-page references, version histories, and accompanying correspondence. MedQA-VQA evaluates structured question answering on predefined questions; real radiology workflows involve open-ended reporting where the clinician must identify what is relevant without a prompt. NExT-QA evaluates short-clip video QA; real video applications involve hours of footage where the key content may be sparse and unpredictable.

This benchmark gap means that impressive benchmark scores do not straightforwardly translate to deployment success. A model that reaches 90% on DocVQA may still fail regularly on the specific document formats used in a particular organization. A model that reaches 85% on POPE may still produce clinically dangerous hallucinations on unusual pathology cases that are rare in benchmarks but common in practice. Careful task-specific evaluation on real data distributions, with appropriate error analysis and failure mode characterization, remains needed before deploying multimodal systems in any high-stakes production context.

The benchmark comparison reveals a consistent pattern: the gap between small and large models is larger on reasoning-intensive tasks than on structured extraction tasks. DocVQA, which requires mostly finding and extracting pre-existing content, shows a 16-point gap between the 7B and 70B model. MathVista, which requires multi-step visual reasoning and computation, shows a 20-point gap. This suggests that scale helps reasoning more than it helps extraction, which aligns with what we know about language model scaling from text-only settings.

This chapter examined four major application domains where multimodal AI creates practical value beyond what either images or text alone can give, and drew out the technical principles that make each domain's challenges tractable.

Document understanding benefits from layout-aware representations that encode the spatial position of text alongside its content. LayoutLM-style models stand for each token with a 2D bounding box embedding, letting the model to use spatial proximity as a signal for semantic relationships between labels and values. The coordinate normalization to a 0-1000 range gives resolution invariance, while the separate width and height embeddings capture document typography as a semantic signal. Document question answering and table understanding build on these representations to let structured information extraction from PDFs, scanned forms, and complex formatted documents at scale. The key insight is that layout is not decoration: in structured documents, where something appears on the page is as informative as what it says.

Medical imaging presents the highest-stakes multimodal application, requiring specialized architectures for each imaging modality and careful attention to distribution shift and evaluation rigor. Radiology report generation uses attention-based grounding to connect generated text to specific image regions, reducing hallucination and giving interpretability that clinicians can audit. Whole-slide pathology analysis uses multiple-instance learning with attention pooling to handle gigapixel images too large for standard transformers, and the per-patch attention weights give spatial interpretability for understanding which tissue regions drove the model's conclusion. Clinical note integration combines imaging with EHR data to produce contextually aware diagnostic assistance that outperforms image-only models on tasks where clinical history is directly informative.

Video understanding introduces the temporal dimension and the computational challenge of processing dozens to hundreds of frames efficiently. Factored spatial-temporal attention reduces the quadratic complexity of full spatiotemporal attention by decomposing it into separate spatial and temporal passes, with complexity $O(N^2 T + N T^2)$ instead of $O((NT)^2)$. Adaptive frame sampling concentrates the frame budget on moments of high temporal change, capturing scene transitions and key events more reliably than uniform sampling. Long-form video understanding requires memory, hierarchy, or retrieval to handle videos too long for direct processing, with retrieval-based approaches showing strong practical performance on Needle-in-a-Haystack-style queries.

Multimodal reasoning is the frontier of the field, requiring models to integrate visual and textual information for compositional tasks. Scientific figure understanding requires quantitative extraction, not just qualitative comparison. Geometric problem solving requires visual parsing, theorem recognition, and symbolic computation in sequence. Multimodal chain-of-thought improves performance by eliciting intermediate steps, though the intermediate chains are not always reliable indicators of the actual computation performed. Multimodal agents push further to action-taking systems that combine visual grounding, task planning, and self-correction to interact with digital environments.

The limitations section highlighted four key challenges: hallucination driven by language priors overriding visual evidence, domain shift requiring careful domain-specific fine-tuning and evaluation, computational cost that constrains real-time deployment, and benchmarking gaps that make benchmark scores imperfect proxies for deployment success. Addressing these limitations requires both continued architectural and training research and careful engineering discipline from multimodal system design through evaluation into deployment.

With these applications understood, we have completed the core arc of Part L. The chapters ahead will extend these ideas to additional modalities and application contexts, including audio-language models and the challenges of aligning multimodal systems with human preferences.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about multimodal applications.