Multimodal Training Data › Visual Instruction Data: Teaching Multimodal Reasoning

Pre-training on image-text pairs and interleaved documents yields a model that can stand for visual content and generate needed text. However, like their text-only counterparts discussed in Part XXXVI: Instruction Tuning, these models do not naturally follow instructions or engage in helpful dialogue. They require fine-tuning on visual instruction data: curated examples of multimodal conversations, reasoning chains, and task completions that teach the model to act as a helpful assistant rather than merely a passive describer.

The gap between pre-training and instruction following is particularly pronounced in multimodal settings. A model pre-trained on alt-text learns to describe images, but not to answer questions about them, follow complex directives involving multiple reasoning steps, or engage in back-and-forth dialogue that references specific visual details. If you ask a model trained purely on captioning data "What would happen if I added more water to the mixture shown in the beaker?" it has no framework for this hypothetical, reasoning-intensive query. It can describe the beaker and its contents, but not reason about chemical processes or answer counterfactual questions.

Instruction tuning bridges this gap by exposing the model to the format and style of assistant interactions. The model learns what to generate and how to respond to specific types of requests: questions and instructions alongside comparisons and multi-step reasoning tasks. This move from passive description to active assistance is not automatic; it requires carefully designed training data that covers the full range of tasks users will request.

Visual instruction datasets structure interactions as conversations or task completions, each serving different pedagogical purposes. The format determines what skills the model develops, from simple recognition to complex multimodal reasoning.

Single-turn QA is the simplest format, consisting of an image, a question, and an answer. For example:

• Image: [photo of ingredients on a counter]
• Question: "What can I make with these ingredients?"
• Answer: "With tomatoes, basil, mozzarella, and olive oil, you can make a Caprese salad. This Italian dish highlights fresh ingredients and requires no cooking."

This format teaches the model to extract specific information from images and map it to domain knowledge. It is useful for visual question answering but does not teach conversational coherence or multi-step reasoning. The model learns to process a single image-question pair and generate a needed answer, but has no mechanism for maintaining context across turns or building on prior exchanges.

Multi-turn conversation extends this to dialogue format:

• User: [image of graph] "What trend does this show?"
• Assistant: "The graph shows increasing global temperatures from 1880 to 2020, with particular acceleration in recent decades."
• User: "What caused the sharp rise after 1980?"
• Assistant: "This acceleration correlates with increased industrial CO2 emissions, deforestation, and greenhouse gas concentrations. The slope change around 1980 marks when global industrialization intensified."

This format teaches the model to maintain context across exchanges, resolve pronouns and references, and give information incrementally based on user follow-up questions. It is needed for chatbot applications and interactive assistance. Notice that in the second user turn, there is no image reference, yet the model must remember which graph was discussed and connect the follow-up question to the visual information from earlier in the conversation.

Visual reasoning chains give explicit intermediate steps:

• Image: [geometry problem diagram]
• Question: "Find the value of angle x."
• Reasoning: "First, I notice that lines AB and CD are parallel. Therefore, angle y equals angle z due to alternate interior angles. Since angle x and angle z form a linear pair, they must sum to 180 degrees. Given that angle z is 60 degrees based on the provided measurement..."
• Answer: "120 degrees"

Chain-of-thought formatting teaches the model to decompose complex problems, verbalize its reasoning process, and check intermediate steps before giving final answers. This materially improves performance on mathematical and logical reasoning tasks involving visual inputs. The key insight from research on chain-of-thought prompting, which we explored in Part XXX: Emergent Capabilities, carries over directly to multimodal settings: models that explain their reasoning steps make fewer errors because intermediate steps constrain and check each other.

Grounded instruction following links language to specific visual regions:

• Image: [city street scene]
• Instruction: "Describe the safety hazard visible in the lower-left quadrant."
• Response: "A manhole cover is slightly raised in the lower-left area, creating a tripping hazard for pedestrians. The uneven surface also poses a risk for cyclists and could cause a wheel to catch."

This format teaches spatial localization: the model must understand the image globally and identify specific regions referenced by coordinates, directions, or object identifiers. This capability is needed for assistive technologies, quality inspection systems, and any application requiring precise visual localization.

Creating high-quality visual instruction data is expensive and labor-intensive. Three approaches dominate the field, each with distinct trade-offs between cost and scale as well as quality.

Human annotation has expert annotators write questions and answers for images, often following detailed guidelines to ensure diversity and depth. This yields high quality but limited scale, typically creating hundreds of thousands rather than millions of examples. Datasets like VQAv2 and OK-VQA use this approach. The advantages include high accuracy, creative diversity, and the ability to capture fine-grained cultural contexts. Skilled annotators ask questions that naturally arise when viewing an image, including questions about relationships and comparisons as well as counterfactuals and implied information that automated systems miss. The disadvantages are cost and time, plus potential annotator biases in how questions are framed and answered. Annotators from the same cultural background may share biases about what questions are interesting or what the "obvious" answer is.

GPT-4V assisted generation uses powerful multimodal models to generate instruction data at scale. The LLaVA approach illustrates this pipeline: collect image captions from CC3M or similar datasets, use GPT-4 to generate diverse instructions and conversations alongside reasoning questions based solely on the caption text (without seeing the image), then optionally use GPT-4V to verify that the generated questions are answerable from the image content. This bootstrapping creates millions of training examples at low cost, letting rapid scaling of instruction datasets.

However, this approach inherits the biases and hallucination tendencies of the teacher model. If GPT-4V tends to make certain types of errors or exhibits specific cultural biases, these propagate into the training data. Also, questions generated from captions rather than actual images may ask about details not visible in the image, creating misaligned training signals. A caption might mention "a red car," but the car may be partially occluded in the actual image, making questions about its color unanswerable from the visual content alone. When the student model trains on these examples, it learns to answer confidently about visual details that cannot be seen, encoding hallucination behavior from the start.

Conversion of academic datasets reformats existing computer vision benchmarks (VQA, captioning, object detection, OCR) as instruction-following examples. For instance, a detection annotation becomes an instruction: "Locate all the cars in this image and give their bounding boxes," with the response containing structured coordinates in JSON format. A segmentation mask becomes a color description task. An OCR annotation becomes a text extraction instruction.

This approach uses carefully curated academic datasets with high-quality annotations. The advantage is accuracy and diversity of task types: academic benchmarks were designed to test specific visual capabilities, and converting them to instruction format injects these capabilities into the instruction-tuned model. The disadvantage is that these datasets often contain distributional biases (specific photography styles, object categories, or compositional patterns from benchmark design) and may not stand for natural user queries. They also lack the conversational flow of real human interactions, potentially making the model overly formal or structured in its responses.

Visual instruction data requires careful formatting to use pre-trained text templates. As we covered in Part XXXVI, Chapter 4: Instruction Format, chat templates structure conversations with special tokens that indicate speaker turns and message boundaries.

For multimodal models, templates typically insert image tokens at the start of sequences to establish the visual context before the conversation begins:

The <image> token (often represented as special token IDs or as placeholder tokens replaced by visual features during the forward pass) signals where to inject the image encoder's output into the language model's context. Different architectures handle this injection differently. Some models prepend image features to the text embeddings, treating them as additional context tokens. Others interleave them at specific positions marked by the token.

System prompts may also specify the model's role: "You are a helpful assistant that can see and analyze images. Answer questions accurately and concisely." This framing helps the model transition from pre-training (where it simply described images) to instruction following (where it assists users with specific tasks). The system prompt also helps constrain undesirable behaviors inherited from pre-training: a model that learned from diverse web text might generate inappropriate content unless the system prompt establishes appropriate assistant behavior.

The choice of template format also affects what edge cases the model handles gracefully. If instruction tuning data always places the image at the beginning of the conversation, the model may struggle when images appear mid-conversation or when multiple images appear at different points in a dialogue. The diversity of instruction template formats in the training data determines how reliably the model handles varied real-world usage patterns.