Multimodal Evaluation › Limitations and Impact

The evaluation of multimodal models faces basic challenges that remain unresolved despite years of benchmark development. Understanding these limitations is important for interpreting results and designing better evaluation protocols.

Current metrics suffer from a semantic gap between what they measure and what we care about. N-gram based metrics like BLEU and CIDEr penalize valid linguistic variation in ways that human judges would not. A model that describes a dog as "a canine" receives lower scores than one that says "a dog," even though both are correct and equally informative. SPICE addresses this by parsing semantic content into structured representations, but scene graph parsers themselves make errors and fail to stand for abstract content reliably. Abstract concepts like emotions ("a sad scene"), intentions ("the boy is about to jump"), or social dynamics ("they appear to be arguing") are difficult to stand for in scene graphs, causing SPICE to systematically undervalue descriptions that capture these important dimensions.

CLIPScore and learned metrics introduce different biases rather than eliminating them. CLIP models trained on web data associate certain phrases with visual concepts based on potentially spurious correlations. A high CLIPScore doesn't guarantee that the model understood the image; it only guarantees that the text and image embeddings align in ways similar to training data pairs. This creates an "evaluation anchor" where models optimized to maximize CLIPScore may exploit dataset-specific patterns rather than developing true visual understanding. For example, CLIP might associate certain fonts, watermarks, or image compositions with specific objects based on how these appeared together in web-scraped training data, creating exploitable shortcuts. A model trained to generate captions that maximize CLIPScore might learn to generate captions containing keywords that CLIP associates with the visual domain without describing the specific image content.

The deeper problem is that no automated metric captures the full richness of what makes a good visual description. Salience matters: a description of a birthday scene that mentions the balloons and ignores the birthday person might score well on overlap metrics while missing the most important element. Accuracy matters at the propositional level: a description that gets most details right but one important detail wrong (saying "the car is blue" when it's red) might score well overall while being clearly wrong in a important respect. These judgment-level failures are hard to encode in quantitative metrics.

As we observed with VQA v1, benchmarks often contain exploitable biases that allow models to reach high performance without real visual understanding. Even carefully constructed datasets like VQA v2 may have subtle statistical patterns: questions about "tennis" often involve green courts, questions about "kitchen" often answer "yes" to questions about whether food is present. Models trained on internet-scale data develop strong priors about these co-occurrences that can substitute for real visual reasoning in many cases.

A more insidious problem is data contamination. Models trained on internet-scale datasets may have seen benchmark images or closely related images during pre-training. The COCO images used in VQA, for example, were publicly available before many foundation models were trained, which makes it plausible that captions, annotations, or discussions of these images appeared in training data. When a model reaches near-perfect scores on a benchmark, we must ask whether it has learned general visual reasoning or simply memorized the test set. This question is difficult to answer definitively: we cannot fully audit what a model has been trained on, and even without direct memorization, a model might generalize from related training data in ways that inflate benchmark performance.

The pace of multimodal benchmark development struggles to keep up with model capabilities. When large multimodal models reach 90%+ on standard VQA benchmarks, those tasks transition from "research challenges" to "solved prerequisites," forcing the community to design harder evaluations like MMMU. This creates a Red Queen dynamic: benchmarks must continuously increase in difficulty to remain relevant, requiring ever more specialized knowledge and complex reasoning to challenge state-of-the-art models. The challenge is that harder benchmarks require more expensive data collection (hiring domain experts to annotate medical images, chemistry diagrams, and legal documents) and may cover narrower capabilities, which makes it harder to interpret performance as a measure of general visual intelligence.

Generative evaluation remains particularly fraught with basic tensions between what is measurable and what matters. FID measures statistical similarity to real images but says nothing about whether the generated image matches its text prompt. Two equally realistic images (one of a cat, one of a dog) receive the same contribution to FID regardless of whether the prompt asked for "a cat" or "a dog." A generation system could reach excellent FID by creating only the most common image types in its training distribution, completely ignoring compositional or unusual prompts.

CLIPScore addresses text alignment but is insensitive to image quality: a blurry, artifact-ridden image may score highly if its semantic content matches the text. This is a significant limitation for evaluating diffusion models and other generative systems where the visual quality of outputs varies substantially. Combining FID and CLIPScore gives a more complete picture, but the combination still misses compositional accuracy (getting all elements right but in the wrong spatial configuration) and fine-grained attributes (getting the object right but the wrong color).

Human evaluation, while authoritative, is expensive and slow, as well as subject to rater variability. Different annotators prioritize different aspects: photorealism versus creativity, literal adherence versus artistic interpretation, technical quality versus emotional resonance. This makes it difficult to compare models across studies conducted by different research groups or to track progress consistently over time. The standard response is to use specific evaluation interfaces (like HIVE or ELO-based preference rankings) that structure the comparison task to minimize noise, but these methods have their own limitations and are still substantially more expensive than automated metrics.

The scalability problem is acute in the current period of rapid model development, where new text-to-image models are released frequently and each requires evaluation. Automated metrics let rapid iteration during development, but final publication-quality evaluation requires human studies that can take weeks to complete and thousands of dollars to run. This creates pressure to rely on automated metrics that might not fully capture the quality dimensions that matter most.

Multimodal benchmarks often reflect cultural and demographic biases present in their source data, and evaluation metrics typically ignore these biases in their aggregate statistics. Images sourced primarily from Western websites depict Western contexts, leading to models that perform better on questions about snow sports than tropical agriculture, or on questions about Christian wedding traditions than Hindu or Jewish ceremonies. Evaluation metrics report aggregate accuracy that masks poor performance on underrepresented groups and contexts.

This representation bias extends to the visual concepts tested. Standard benchmarks test object recognition for the hundreds of objects most common in Western internet photography: cars, dogs, pizza, wine. They rarely test recognition of objects common in other cultural contexts: specific types of traditional clothing, regional foods, or culturally specific tools and implements. A model that reaches high benchmark accuracy while failing to recognize common objects in underrepresented cultural contexts may perform well in Western deployment contexts but fail for users from other backgrounds.

Bias also affects the metrics themselves in subtle ways. CLIP models trained on English internet text may produce better embeddings for English captions than captions in other languages, causing CLIPScore to systematically undervalue accurate non-English descriptions. Object detectors trained on Western datasets may fail to recognize culturally specific objects or to handle image styles (photography conventions, color processing, composition norms) that differ across cultures. Without careful geographic and cultural, plus linguistic, diversification of evaluation data, we risk building evaluation frameworks that implicitly define "good performance" in culture-specific ways.

Addressing these biases requires systematic effort in benchmark construction: explicit geographic and cultural diversity requirements in image collection, involvement of annotators from diverse backgrounds, translation and cross-lingual evaluation components, and reporting of performance stratified by demographic variables. Some recent benchmarks like CROSSMODAL-3600 have begun addressing linguistic diversity by collecting captions in 36 languages, but this remains the exception rather than the norm.

Despite these limitations, multimodal evaluation frameworks have driven substantial progress. The move from VQA v1 to v2 taught the community that balanced answer distributions are needed for valid evaluation, leading to more rigorous dataset construction practices that are now standard. GQA demonstrated the value of compositional, structured evaluation over aggregate accuracy, inspiring subsequent benchmarks that test specific reasoning primitives and letting targeted model improvements. CLIPScore enabled automated evaluation of text-to-image models at scale, accelerating iteration cycles and letting the rapid progress in generative modeling seen in diffusion models.

The tension between automated metrics and human judgment has pushed the field toward hybrid evaluation pipelines. During model development, automated metrics (VQA accuracy, CLIPScore, FID) give fast feedback that supports rapid iteration on training procedures and architectures alongside data curation. During validation and publication, human evaluation studies give authoritative quality assessments that capture the nuances automated metrics miss. This division uses the speed of automated metrics for the many decisions made during development while preserving the nuance of human judgment for the few decisions that matter most for research claims.

The evolution of benchmarks also shows improving understanding of what multimodal AI can and should do. Early benchmarks tested perception and simple reasoning, mirroring the capabilities of the models available at the time. Modern benchmarks test expert-level reasoning, instruction following, and integrated multi-skill tasks, mirroring the capabilities of current foundation models. As we move toward more capable multimodal agents, evaluation must continue evolving to assess interactive reasoning, tool use, and long-horizon task completion. Future benchmarks may evaluate whether models can use visual information to operate interfaces, interpret scientific data for research purposes, or collaborate with humans on complex creative projects, moving beyond question-answering to assess real multimodal intelligence operating in realistic, dynamic environments.