Flamingo Architecture › Empirical Results and Evaluation

Flamingo was evaluated on a complete suite of vision-language benchmarks, establishing state-of-the-art performance at the time of publication across multiple tasks. Understanding these results helps clarify what the architecture reaches in practice and where its strengths and weaknesses lie.

On the VQAv2 benchmark, which tests visual question answering on a diverse set of natural images, the 80B Flamingo model achieved 82.0% accuracy in a 32-shot setting, surpassing the best fine-tuned models at the time despite not using any task-specific training data. This is a remarkable result: a model that has never been explicitly trained to answer questions about images outperforms models that have been fine-tuned for exactly this task. The performance advantage grows with the number of in-context examples. This shows that Flamingo uses the provided context rather than relying purely on patterns memorized during pretraining.

The few-shot VQA results also scale cleanly with model size. The 3B, 9B, and 80B Flamingo variants show consistent improvement as scale increases, suggesting that the visual reasoning capabilities benefit from larger language model capacity in ways consistent with the scaling laws we discussed in Part XXIII. This scaling behavior validates that the frozen language model's increased capacity contributes directly to multimodal reasoning, not just text generation quality.

On COCO Captions, Flamingo reaches strong CIDEr scores in few-shot settings even without access to ground-truth captions during evaluation. The captions generated by Flamingo tend to be more descriptive and contextually appropriate than simpler captioning models. This shows the language model's advanced linguistic generation capabilities. However, the model does not always match specialized fine-tuned captioning models on standard automatic metrics like BLEU and METEOR, partly because the language model's style of captioning (influenced by web text patterns) does not perfectly match the crowd-sourced caption style that these metrics reward.

One surprising strength of Flamingo is its performance on tasks requiring text reading from images, such as reading charts and signs as well as documents. Because the vision encoder captures fine-grained spatial features and the language model has extensive exposure to structured text formats, Flamingo can often correctly read and reason about text embedded in images using only a few in-context examples showing the task format. This capability, sometimes called visual text recognition, was not explicitly targeted during training but emerges from the combination of capable visual and linguistic components.