LLaVA Architecture › Visual Instruction Data Generation

A key innovation enabling LLaVA's capabilities is the methodology for creating visual instruction-following data at scale. Manually annotating millions of multimodal conversations would be prohibitively expensive, requiring thousands of human hours to craft diverse, high-quality examples covering the full spectrum of visual reasoning tasks. Since manual annotation is infeasible at the scale required for training modern multimodal models, LLaVA uses existing language models to bootstrap the dataset, using the strong zero-shot capabilities of GPT-4V to generate training supervision.

The original LLaVA paper used GPT-4V (or GPT-4 with caption and bounding boxes) to generate training data. This bootstrapping approach exploits the fact that while GPT-4V is expensive to use at inference time for all users, it can be used cost-effectively to generate high-quality training data that teaches smaller, more efficient models to replicate its capabilities. The process works as follows:

1. Image Captioning: Generate detailed captions describing image contents, including objects and attributes as well as activities
2. Bounding Box Annotation: Identify objects and their locations (when using text-only models, bounding boxes give spatial grounding)
3. Prompt Engineering: Feed captions and bounding boxes to GPT-4 with prompts requesting specific types of interactions

For example, to generate conversational data:

This produces natural, diverse dialogues that capture the distribution of real user queries. The approach generates three types of instruction data:

• Conversation (58K samples): Multi-turn QA about image content, simulating realistic user-assistant interactions
• Detailed Description (23K samples): Long-form image descriptions that capture detailed visual details beyond simple captioning
• Complex Reasoning (77K samples): Questions requiring inference and external knowledge, testing the model's ability to reason about causality and intent alongside abstract concepts

The diversity of these categories ensures that the trained model can handle simple recognition tasks, open-ended dialogue, and analytical reasoning about visual content.

The synthetic data approach allows for systematic coverage of visual reasoning skills across multiple dimensions of understanding:

• Object recognition: Identifying items in the scene, from common objects to rare or specialized items
• Attribute detection: Colors and sizes, materials and textures, plus other physical properties
• Spatial relationships: Positional reasoning ("left of," "behind," "above") and compositional scene understanding
• Action understanding: What people or animals are doing, including subtle activities and interactions
• Abstract reasoning: Intentions, emotions, likely future events, and causal relationships

The complex reasoning category deserves particular attention because it is the hardest to obtain through human annotation. Questions like "Why might this storefront have bars on its windows?" or "What time of day does this photo most likely stand for?" require drawing on world knowledge beyond what is literally visible. GPT-4V's strong common-sense reasoning makes it well-suited to generate such examples, and these samples are precisely what teach LLaVA to go beyond object detection toward real visual understanding.

However, synthetic data has limitations that must be carefully managed. GPT-4V may hallucinate details not present in the image, generating confident descriptions of objects that do not exist, or generate questions that assume knowledge the image does not give (such as asking about a person's name when only their back is visible). The original LLaVA paper included examples where the generated questions and answers were inconsistent with the provided image metadata, illustrating that this noise is real even with a powerful generator.

Later versions of LLaVA addressed this by incorporating human-verified academic datasets and more advanced filtering mechanisms to remove low-quality or hallucinated examples from the training mix. The key insight from this evolution is that the quality distribution of synthetic data matters more than its raw volume: a smaller, carefully curated synthetic dataset outperforms a much larger but noisy one.

One subtle aspect of the original data generation pipeline is that text-only GPT-4 was used in some cases (without direct image access), relying on image captions and bounding box coordinates as proxies for visual content. This is a clever workaround for cases where GPT-4V access is limited or expensive: since COCO images already have detailed bounding box annotations, one can describe the spatial layout of an image entirely in text and ask GPT-4 to generate conversational data about it.

For example, given a bounding box annotation saying "person at (120, 80, 200, 300), umbrella at (100, 40, 220, 120)," GPT-4 can generate the question "The person in the image is carrying an umbrella. What might this suggest about the weather?" without ever seeing the actual image. This approach lets data generation at scale using text-only models, which were far cheaper and more accessible than GPT-4V when LLaVA was originally developed.

The limitation is that bounding box proxies cannot capture aesthetic qualities, lighting, emotional tone, or fine visual details that a human or GPT-4V would notice. These proxies work well for object-centric questions but struggle for questions about overall mood, artistic style, or subtle interactions. This limitation motivated LLaVA 1.5 to use richer image descriptions and incorporate GPT-4V directly for higher-quality data generation.