Multimodal Evaluation › Key Metric Parameters

The key parameters for multimodal evaluation metrics are:

• consensus_threshold: The minimum number of human annotators (default 3 in VQA) required for full accuracy credit. Answers matching fewer humans receive partial credit proportional to the match count divided by this threshold. Higher thresholds require more consensus but penalize valid synonyms, while lower thresholds reward diversity but may accept incorrect answers. The value of 3 was chosen empirically in the original VQA paper.
• max_n: Maximum n-gram order for BLEU calculation (typically 4). Higher values require matching longer phrase sequences, increasing strictness for exact lexical overlap. BLEU-4 requires matching four-word sequences, which makes it sensitive to word order and exact phrasing. Lower n values (BLEU-1, BLEU-2) are more tolerant but less discriminating.
• scaling_factor: For CLIPScore, the multiplier (typically 100) that maps cosine similarity from $[-1, 1]$ to a readable percentage scale. This linear transformation makes scores interpretable without changing the underlying ranking of model outputs.
• reference_count: The number of human reference captions collected per image for captioning evaluation. More references give better coverage of valid descriptions, reducing false negatives where correct but unusual phrasings receive low scores. Standard datasets typically collect 5 references per image, balancing coverage against collection cost.