Multimodal LLMs (2023) › Limitations

Despite their capabilities, multimodal large language models in 2023 had limitations that constrained their practical applications. Understanding those limits supported realistic expectations and appropriate deployment.

One fundamental limitation was the resolution and detail level of visual understanding. While models could process and understand images, their ability to perceive fine details was constrained by the vision encoder's resolution limits and computational requirements. High-resolution images were often downsampled before processing, potentially losing important details. This limitation was particularly problematic for applications requiring precise visual analysis, such as medical imaging, scientific diagram analysis, or reading small text in images. The models could miss subtle visual elements or fail to distinguish between similar visual patterns that humans could easily differentiate.

Reasoning about spatial relationships and geometry remained challenging. While models could describe what they saw and answer questions about an image, their handling of geometry and precise spatial detail lagged behind their textual reasoning. Tasks requiring precise spatial reasoning, understanding of perspective, or geometric calculations based on visual information could be challenging. This limitation reflected that visual understanding, while impressive, had not reached the same level of sophistication as the models' language understanding capabilities.

The training data created biases and gaps in visual understanding. Models learned from image-text pairs available on the internet, so they reflected the biases and coverage gaps in that material. Images from certain cultures, contexts, or domains might be underrepresented, leading to gaps in understanding. The models might struggle with visual content outside their training distribution, such as unusual artistic styles, specialized technical diagrams, or images from underrepresented contexts. These biases were technical limitations with direct implications for fairness and representation in AI systems.

Computational requirements remained substantial, limiting accessibility and scalability. Training multimodal models required significant computational resources, making it challenging for smaller organizations or researchers to develop or fine-tune these systems. Inference also required more computation than text-only models, as processing images added overhead. This computational cost limited deployment options and made it difficult to run these models on consumer hardware or in resource-constrained environments. Real-time applications or applications requiring processing of many images could be impractical.

Vision and language did not always integrate well. The models could struggle with tasks requiring close coordination of visual and textual reasoning, such as understanding complex diagrams with extensive textual annotations, following multi-step visual instructions, or reasoning about temporal sequences of images with accompanying text. The architectural choices made to enable multimodal integration sometimes involved trade-offs that limited capabilities in specific domains or applications.

Safety and reliability concerns were particularly important for multimodal systems. Vision models could be fooled by adversarial images, fail to detect subtle but important visual elements, or misinterpret visual content in ways that could have serious consequences. The combination of visual and textual understanding created new attack surfaces and failure modes that needed careful consideration. For applications where errors could have significant consequences, such as medical image analysis or safety-critical systems, these limitations required careful evaluation and appropriate safeguards.

The models' visual understanding remained less sensitive to context and specialized knowledge than human perception. They could describe images and answer questions, but often missed how context or commonsense knowledge changed an interpretation. They might overlook subtle visual cues or struggle with content that depended on cultural or specialist knowledge.