Multimodal Applications › Summary

This chapter examined four major application domains where multimodal AI creates practical value beyond what either images or text alone can give, and drew out the technical principles that make each domain's challenges tractable.

Document understanding benefits from layout-aware representations that encode the spatial position of text alongside its content. LayoutLM-style models stand for each token with a 2D bounding box embedding, letting the model to use spatial proximity as a signal for semantic relationships between labels and values. The coordinate normalization to a 0-1000 range gives resolution invariance, while the separate width and height embeddings capture document typography as a semantic signal. Document question answering and table understanding build on these representations to let structured information extraction from PDFs, scanned forms, and complex formatted documents at scale. The key insight is that layout is not decoration: in structured documents, where something appears on the page is as informative as what it says.

Medical imaging presents the highest-stakes multimodal application, requiring specialized architectures for each imaging modality and careful attention to distribution shift and evaluation rigor. Radiology report generation uses attention-based grounding to connect generated text to specific image regions, reducing hallucination and giving interpretability that clinicians can audit. Whole-slide pathology analysis uses multiple-instance learning with attention pooling to handle gigapixel images too large for standard transformers, and the per-patch attention weights give spatial interpretability for understanding which tissue regions drove the model's conclusion. Clinical note integration combines imaging with EHR data to produce contextually aware diagnostic assistance that outperforms image-only models on tasks where clinical history is directly informative.

Video understanding introduces the temporal dimension and the computational challenge of processing dozens to hundreds of frames efficiently. Factored spatial-temporal attention reduces the quadratic complexity of full spatiotemporal attention by decomposing it into separate spatial and temporal passes, with complexity $O(N^2 T + N T^2)$ instead of $O((NT)^2)$. Adaptive frame sampling concentrates the frame budget on moments of high temporal change, capturing scene transitions and key events more reliably than uniform sampling. Long-form video understanding requires memory, hierarchy, or retrieval to handle videos too long for direct processing, with retrieval-based approaches showing strong practical performance on Needle-in-a-Haystack-style queries.

Multimodal reasoning is the frontier of the field, requiring models to integrate visual and textual information for compositional tasks. Scientific figure understanding requires quantitative extraction, not just qualitative comparison. Geometric problem solving requires visual parsing, theorem recognition, and symbolic computation in sequence. Multimodal chain-of-thought improves performance by eliciting intermediate steps, though the intermediate chains are not always reliable indicators of the actual computation performed. Multimodal agents push further to action-taking systems that combine visual grounding, task planning, and self-correction to interact with digital environments.

The limitations section highlighted four key challenges: hallucination driven by language priors overriding visual evidence, domain shift requiring careful domain-specific fine-tuning and evaluation, computational cost that constrains real-time deployment, and benchmarking gaps that make benchmark scores imperfect proxies for deployment success. Addressing these limitations requires both continued architectural and training research and careful engineering discipline from multimodal system design through evaluation into deployment.

With these applications understood, we have completed the core arc of Part L. The chapters ahead will extend these ideas to additional modalities and application contexts, including audio-language models and the challenges of aligning multimodal systems with human preferences.