Multimodal Evaluation › Visual Question Answering Benchmarks

Visual Question Answering (VQA) tasks ask models to answer natural language questions about images, requiring both visual perception and linguistic understanding. This seemingly simple formulation masks significant complexity: a model must parse the question, identify relevant image regions, perform reasoning (counting, comparing, attributing), and generate or select an appropriate answer.

This difficulty arises because VQA combines multiple AI challenges. The model must handle syntactic ambiguity in questions (does "What is the man eating?" refer to the food or the action?), resolve visual coreference (determining which object "the large one" refers to when multiple objects are present), and integrate commonsense knowledge about physical properties and social contexts. Unlike pure object detection or image classification, VQA requires dynamic, task-specific visual processing guided by the linguistic query. The question directs attention: "What is the color of the car?" and "How many windows does the building have?" both refer to the same image but require entirely different visual processing strategies.

The success of a VQA model therefore tests more than either perception or language understanding in isolation. It tests the model's ability to orchestrate multiple cognitive processes based on a linguistic instruction, making it a useful proxy for general visual intelligence. This is why VQA became the first dominant benchmark for early vision-language models and continues to serve as a foundation for more advanced evaluations.

In standard VQA, we have an image $I$ and a question $Q$ (a sequence of tokens $q_1, q_2, \ldots, q_n$). The model produces an answer $A$ from a predefined set or as open-ended text. The benchmark gives a set of triplets $\{(I_i, Q_i, A_i)\}_{i=1}^N$ where multiple human annotators give answers for each question.

This triplet structure shows the dataset collection process. Researchers present images to human annotators and ask them to generate questions that another person could answer by looking at the image. Then, different annotators answer these questions, creating a distribution of valid responses rather than a single ground truth. This distribution captures the natural variability of human language and perception, acknowledging that different people attend to different aspects of an image and express the same concept using different words.

The decision to collect multiple answers per question was itself a deliberate design choice, motivated by the realization that language is inherently variable. Early datasets that collected a single ground-truth answer found that reasonable model outputs were incorrectly penalized for paraphrasing. Collecting 10 answers per question creates a richer supervision signal that better shows the breadth of valid human responses.

The evaluation metric must account for the variability in how humans phrase identical concepts. Consider the question "What color is the car?" Valid answers include "blue," "dark blue," "navy," and "azure." Simple string matching would penalize synonymous responses, creating a mismatch between the metric and human judgment. A model that says "azure" when humans said "blue" has understood the image correctly but would receive a failing grade under exact match criteria. This mismatch between metric and understanding was one of the central problems that shaped VQA evaluation from the outset.

The standard VQA accuracy metric handles answer variability through consensus. Given that 10 human annotators give answers, the score for a candidate answer $A$ is:

where:

• $A$: the candidate answer being evaluated (a string)
• $\text{count}(A \text{ in human answers})$: the number of human annotators (out of 10) who provided answer $A$ exactly
• $3$: the consensus threshold, meaning at least 3 matching human answers are needed for full credit
• $\min(\cdot, 1)$: caps the score at 1.0 (100%), so 3 or more matches yield perfect credit

The threshold of 3 is a balance between rewarding consensus and penalizing rare answers. Researchers determined this value empirically to ensure that answers receiving full credit reflect common human expression while still letting for some linguistic diversity. Fewer than 3 matches yields partial credit proportional to the fraction, acknowledging that the answer is valid but not the most common expression.

This "soft accuracy" gives partial credit when an answer matches at least one of multiple human responses. If 3 out of 10 annotators said "blue," an answer of "blue" receives accuracy 1.0 (capped). If only 2 annotators said "azure," the model receives $2/3 \approx 0.67$. This grading scheme acknowledges that "azure" is semantically correct even if less common, but recognizes that extremely rare answers might indicate misunderstanding or hallucination.

The design of this metric shows a broader insight: evaluation should measure what we care about, not what is easy to compute. String matching is computationally trivial, but human judgment is the true standard. Consensus scoring approximates human judgment by aggregating multiple annotator perspectives, trading off the cost of human evaluation against the quality of automated assessment.

For multiple-choice VQA, standard classification accuracy applies because the answer space is constrained and each option is distinct. For open-ended generation, exact match against the most common human answer or automated metrics like BLEU serve as proxies, though both have well-documented shortcomings that we discuss throughout this chapter.

VQA has evolved through several generations, each addressing limitations of its predecessors. Understanding this progression illuminates the core challenges of multimodal evaluation and the design decisions required to measure what we care about.

VQA v1 and VQA v2 established the foundational protocol. These datasets contain approximately 200K real images from COCO and 50K abstract scenes. VQA v2's complementary pairs mechanism ensures visual grounding is necessary for correct answers. Questions span three categories: yes/no questions (binary), number questions (counting), and "other" questions (open-ended descriptions). These datasets established the practice of collecting multiple answers per question and using consensus-based evaluation, setting the standard for subsequent benchmarks. The three-category breakdown proved useful for diagnosing model weaknesses: early models often performed well on yes/no questions (which could be gamed by frequency heuristics) but struggled with precise counting and open-ended description.

GQA (Graphical Question Answering) took a fundamentally different approach by using scene graphs to generate questions programmatically. Scene graphs stand for images as structured data: nodes stand for objects (with attributes like color and size), and edges stand for relationships (spatial or semantic). By traversing these graphs, GQA creates precise compositional questions testing specific reasoning skills: transitive relations ("What is left of the object behind the red cube?"), attribute comparisons, and logical operations. The structured generation lets fine-grained analysis of model capabilities across 20+ reasoning types. This allows researchers to identify whether failures stem from object recognition, relationship understanding, or logical inference. This diagnostic precision was a measurable advance over aggregate accuracy numbers.

The insight behind GQA is that natural VQA questions are noisy. Human annotators ask about whatever catches their eye, creating a dataset that tests diverse but uncontrolled reasoning types. GQA's programmatic generation creates a controlled evaluation environment where each reasoning type is systematically represented, letting targeted improvement. If a model scores 90% on color questions but 40% on spatial relationship questions, developers know exactly where to focus their efforts.

OK-VQA (Outside Knowledge VQA) recognized that real-world visual reasoning often requires information not present in the image itself. Answering "Why is this person wearing a helmet?" requires understanding bicycle safety norms or construction regulations, not just recognizing the helmet. This bridges visual perception with world knowledge, anticipating the capabilities of modern large multimodal models. The benchmark tests whether models can retrieve and apply external facts to interpret visual situations, moving beyond pure perception to real understanding. A model that sees a person in a white coat holding a stethoscope must know about medical professions to correctly answer "What is this person's job?"

A-OKVQA extends OK-VQA with rationales and multiple-choice formats. This makes possible evaluation of chain-of-thought reasoning in visual contexts. This allows researchers to assess whether models answer correctly and whether they can articulate their reasoning. Collecting rationales turns the benchmark from a simple accuracy measure to a window into model cognition. This gives insight into how multimodal models combine visual evidence with background knowledge. A model that gets the right answer for the wrong reason (guessing based on superficial image features rather than real reasoning) can be identified through rationale analysis.

TextVQA and ST-VQA focus on reading and reasoning about text that appears within images: signs and labels as well as receipts and menus. They require OCR capabilities integrated with visual reasoning, testing whether models can read "Stop" on a sign and answer "What should the driver do here?" These tasks are important for applications like assistive technology for the visually impaired and autonomous navigation. Early vision-language models largely ignored text within images because they treated images as broad semantic content rather than structured visual scenes containing multiple information types. TextVQA exposed this blind spot and drove development of models that treat in-image text as a first-class signal.

ScienceQA is a multimodal science question-answering dataset with detailed lectures and explanations, targeting educational applications and testing multimodal reasoning on scientific diagrams. Questions often require interpreting complex figures and charts alongside experimental setups from textbooks, demanding both domain expertise and visual understanding. The educational context also lets evaluation of explanation quality: a model that correctly identifies a graph's trend should also be able to explain why that trend occurs. ScienceQA laid groundwork for the more complete expert-level benchmarks that followed.