Instruction Data Creation › Combining Data Sources

Real instruction-tuning datasets rarely use a single creation method. Instead, they combine multiple sources to balance their respective strengths and weaknesses. This combination recognizes that no single approach excels at everything: human annotation provides quality but not scale, templates provide scale but not naturalness, and seed expansion provides a middle ground that still requires careful curation. The art of building a good mixed dataset lies in understanding what each source contributes and calibrating the mix accordingly.

Think of a mixed instruction dataset as a recipe with multiple ingredients. Each ingredient serves a purpose: human-written examples provide the flavor that makes the dish distinctive, template-converted examples provide the volume that makes it filling, and seed-expanded examples add the variety that keeps it interesting. Getting the proportions right requires understanding the role of each component. Too much of any single ingredient overwhelms the others and produces a lopsided result.

The decision of how to mix sources should be driven by the capabilities you want the final model to have. If the primary goal is strong performance on structured tasks like question answering and classification, template-converted data should dominate. If the primary goal is natural, conversational behavior, human-written examples should be proportionally larger. If you want strong creative writing capabilities, seed-expanded creative examples need sufficient representation to teach that skill effectively.

The FLAN-T5 model, for instance, combines template-converted versions of 62 existing datasets with chain-of-thought examples and dialogue data. This mixture teaches both task-specific skills and more general conversational abilities. The template-converted data provides broad coverage of NLP tasks, while the chain-of-thought and dialogue data teach more sophisticated reasoning and interaction patterns. The specific choice of which task datasets to include was guided by the goal of broad generalization. This keeps the resulting model could handle a wide range of user requests without task-specific fine-tuning.

More recent models like Llama 2 Chat and Claude use instruction datasets that combine human demonstration data from annotators, synthetic data generated by the models themselves, and filtered internet data containing instruction-like content. Each source serves a different purpose: annotator data for quality and safety alignment, synthetic data for scale and coverage, and filtered internet data for diverse phrasing and topics. The relative proportions are carefully tuned through ablation experiments that measure the contribution of each source to downstream performance.

This visualization highlights the strategic balance between volume (template-converted) and quality (human-written/seed-expanded) required for effective instruction tuning. While automated methods provide the bulk of the data, the smaller high-quality components are essential for steering the model's style and capabilities. The human-written demonstrations, though small in number, have outsized influence on how the model behaves, establishing patterns that the larger automated data then reinforces.

An important consideration in mixing sources is weighting during training. Simply concatenating all sources and training on the concatenation gives each example equal weight, but this means the large template-converted component dominates the gradient signal regardless of its relative quality. Production training pipelines often over-sample smaller high-quality sources, presenting each human-written example multiple times per epoch while presenting each template-converted example fewer times. This reweighting ensures that quality guides the model's behavior even when quality examples are outnumbered by volume examples.