Multimodal LLMs (2023) › Applications and Impact

The emergence of multimodal large language models in 2023 enabled a wide range of applications that were previously impossible or required complex multi-system architectures. These applications used the models' ability to reason about visual and textual information together.

In scientific research and education, multimodal models found immediate applications. Researchers could upload scientific papers containing diagrams and charts and receive explanations that integrated visual and textual information. Students learning from textbooks with illustrations could ask questions about both the text and accompanying figures, receiving explanations that connected visual and textual information. The models could analyze research figures, explain experimental results depicted in graphs, and help users understand how visual elements supported textual arguments. This capability was particularly valuable in biology and chemistry as well as physics and mathematics, where visual information is central to understanding.

Content creation and creative applications benefited significantly from multimodal capabilities. Users could provide images as prompts for creative writing, asking models to generate stories or descriptions based on visual content. The models could analyze photographs and generate detailed captions or articles incorporating visual descriptions. Graphic designers and content creators could use these models to understand design elements, generate text that complemented visual content, and create coherent multimodal content. The ability to understand both images and text enabled more sophisticated creative workflows where visual and textual elements worked together.

Accessibility applications saw substantial improvements with multimodal models. Image description systems could provide basic descriptions or detailed explanations tailored to a user's needs. The models could answer questions about images, describe complex scenes with detail, and help visually impaired users understand visual information in ways that were previously challenging. Language processing allowed descriptions to include relevant context and respond to user queries instead of remaining static and generic.

Data analysis applications used multimodal models to interpret charts and other visualizations. Users could upload a chart and ask questions about trends, relationships, or specific data points. The models could explain what the visualization showed, identify patterns, and help users interpret data. This capability was valuable for business intelligence, scientific research, and educational contexts where visual data representation is common but interpretation requires expertise.

The impact extended beyond individual applications to broader shifts in how AI systems could be deployed and used. Multimodal models reduced the need for complex multi-system architectures that combined separate vision and language models with custom integration logic. Instead, a single model could handle tasks requiring both modalities, simplifying deployment and improving performance through better integration. This shift made multimodal AI more accessible to developers and users, enabling new applications to be built more quickly and efficiently.

The performance of multimodal models on diverse tasks demonstrated their practical value. GPT-4, for example, could analyze medical images with appropriate caveats, explain scientific diagrams, describe photographs in detail, and answer questions about complex visual content. While not replacing specialized systems, these general capabilities enabled new use cases and improved existing applications. The models' ability to handle diverse tasks without task-specific training made them particularly valuable for applications with varied or evolving requirements.

The impact also extended to how AI systems were perceived and used. The ability of multimodal models to process visual information in natural language interactions made AI feel more capable and aligned with human communication patterns. Users could interact with AI systems more naturally. This provided information in whatever form was most convenient, whether text, images, or both together. This naturalness of interaction was central for adoption and practical use.