Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multimodal Evaluation › Worked Example: Evaluating VQA Predictions

Michael BrenndoerferFebruary 14, 20263 min read

Language AI HandbookAll chapters

Part 50 of 67Multimodal Models

Chapter 388Multimodal Evaluation › Worked Example: Evaluating VQA Predictions

Reading section 5 of 10 · Worked Example: Evaluating VQA Predictions · 3 minSection 5 of 10 · 3 min

Section 5 of 10 · 3 min read

Multimodal Evaluation › Worked Example: Evaluating VQA PredictionsOne section · Sections in this chapter5 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's work through a concrete example of VQA evaluation, computing accuracy with consensus scoring and analyzing error patterns across different question types.

Consider a VQA example with the question "What color is the fire hydrant?" and human answers: ["red", "red", "red", "crimson", "dark red", "scarlet", "red", "red", "orange-red", "red"]. A model predicts "red".

In[7]:

Code

# VQA Consensus Accuracy Calculation
human_answers = [
    "red",
    "red",
    "red",
    "crimson",
    "dark red",
    "scarlet",
    "red",
    "red",
    "orange-red",
    "red",
]
model_answer = "red"

# Count matching answers
matching = sum(1 for ans in human_answers if ans == model_answer)
total_humans = len(human_answers)

# VQA accuracy: min(matching / 3, 1.0)
accuracy = min(matching / 3, 1.0)

Out[8]:

Console

Human answers: ['red', 'red', 'red', 'crimson', 'dark red', 'scarlet', 'red', 'red', 'orange-red', 'red']
Model answer: 'red'
Matching human answers: 6
VQA Accuracy: 1.000

Out[9]:

Visualization

Bar chart of human answer counts for fire hydrant color question, with red highlighted and reaching count 7 while synonyms each score 1. — Distribution of human answers for the fire hydrant color question. The consensus answer ''red'' appears 7 times, while valid synonyms like ''crimson'' and ''scarlet'' appear once each, illustrating the challenge of exact-match evaluation: semantically correct synonyms receive only partial credit under the VQA consensus scoring scheme.

The model receives a perfect score of 1.0 because "red" matches the majority answer. But what if the model answered "crimson"?

In[10]:

Code

model_answer_variant = "crimson"
matching_variant = sum(
    1 for ans in human_answers if ans == model_answer_variant
)
accuracy_variant = min(matching_variant / 3, 1.0)

Out[11]:

Console

If model answered 'crimson':
Matching human answers: 1
VQA Accuracy: 0.333

The result reveals a basic limitation: "crimson" is semantically correct but receives partial credit of approximately 0.33 because only one human used that specific word. This penalizes valid synonymy, motivating the development of metrics like WUPS (Wu-Palmer Similarity) that use WordNet semantic hierarchy to score related words appropriately. WUPS would recognize that "crimson" and "red" occupy nearby positions in the color taxonomy, awarding similarity scores based on their distance in the hierarchy rather than requiring string identity. The trade-off is that WUPS requires maintaining a semantic taxonomy and defining distance thresholds, introducing its own hyperparameters and assumptions about what constitutes acceptable synonymy.