Instruction Data Creation

As we discussed in the previous chapter on instruction following, the key innovation that turns a language model into an instruction-following assistant is not architectural. It is the training data. A model learns to follow instructions by seeing thousands of examples that demonstrate the desired behavior: clear tasks paired with helpful, accurate responses. This chapter explores the practical challenge of creating such data at scale, examining the techniques that have enabled researchers to build the datasets that power modern AI assistants.

Think of instruction data creation as teaching someone a new role through worked examples rather than rules. You could write a thousand pages of guidelines for how a helpful assistant should behave, or you could simply show them a thousand real conversations where someone helped effectively. The second approach is more concrete, more generalizable, and ultimately more effective. Instruction data works the same way: instead of encoding desired behavior as explicit rules in the model's architecture, we encode it implicitly through examples of the behavior we want. The model learns the pattern by immersion.

Creating high-quality instruction data is both an art and an engineering challenge. You need sufficient volume to cover diverse tasks, enough quality to teach proper behavior, and adequate diversity to generalize beyond memorized patterns. The field has developed several complementary approaches to meet these requirements: human annotation for quality, template-based generation for scale, seed task expansion for diversity, and quality filtering to maintain standards across all sources. Each approach has its own strengths and limitations, and understanding these tradeoffs is essential when you design or evaluate instruction-tuning pipelines.

The challenge is not unlike the challenge of designing a curriculum for a student. A curriculum that covers only a narrow set of topics produces a specialist, not a generalist. One that includes badly explained examples teaches bad habits. One that repeats the same material endlessly wastes time without building new skills. Good instruction data strikes a balance: broad coverage, clear demonstrations, and enough variety that the student learns principles rather than memorized responses.

One of the most important insights from years of instruction-tuning research is that quality often matters more than quantity, at least up to a point. The original InstructGPT model from OpenAI used only about 13,000 human-written demonstrations, yet it produced dramatically more helpful behavior than the much larger pre-trained model it was built on. This finding has been replicated across many projects: a relatively small but carefully curated dataset can elicit capabilities that require far more examples to develop when those examples are poor. We will see how this tradeoff plays out across every approach covered in this chapter.

This chapter covers the complete lifecycle of instruction data creation, from gathering initial examples through quality control and final assembly. We begin with human annotation, the gold standard against which all other methods are compared, then move through template-based generation and seed task expansion as methods for achieving scale. We conclude with quality filtering, which turns raw data of mixed quality into a clean training set. Understanding each component prepares you to both build your own instruction datasets and critically evaluate those released by others.

Human annotation remains the gold standard for instruction data quality. When a skilled annotator writes an instruction and its response, they bring world knowledge, common sense, and an intuitive understanding of what makes a response helpful. The earliest instruction-tuned models, including InstructGPT from OpenAI, relied heavily on human-written demonstrations. This reliance on human expertise highlights that instruction following depends on human communication patterns and preferences that automated methods struggle to capture.

Think of human annotation as setting the ceiling for data quality. Every other method in this chapter attempts to approach that ceiling more efficiently, accepting some quality degradation in exchange for scale. Human annotators can handle ambiguous instructions, produce careful multi-step reasoning, catch errors in their own output, and write natural responses. These capabilities are difficult to replicate programmatically. Annotation programs aim to collect data while preserving the tacit knowledge that skilled communicators apply automatically.

The key insight is that annotation quality depends critically on the annotation process itself, not just on the skill of individual annotators. Even highly capable annotators produce inconsistent, low-quality data when given poor guidelines, inadequate feedback, or incentive structures that prioritize speed over quality. The most successful annotation programs invest heavily in guideline development, annotator training, and ongoing quality control, treating the annotation process as a product that requires iteration and improvement.

The challenge of human annotation is ultimately economic. Human attention is expensive, and the volume of instruction data needed to train a capable assistant is large. This tension between quality and scale motivates everything that follows in this chapter.

Creating effective annotation guidelines requires balancing specificity with flexibility. Guidelines that are too rigid produce formulaic responses that lack naturalness. Guidelines that are too loose produce inconsistent data where annotators interpret tasks differently. Finding the right balance is itself a design challenge that requires iteration and feedback from annotators who work with the guidelines in practice. This feedback loop is essential: annotators discover ambiguities in guidelines that guideline writers never anticipated, and incorporating that feedback improves the quality of subsequent annotation.

Effective guidelines typically specify the following dimensions of response quality:

• Response format expectations: Should responses include explanations, or just direct answers? Should they acknowledge uncertainty? These choices shape the personality and communication style of the resulting model.
• Tone and style: Formal or conversational? Verbose or concise? The tone established in training data carries through to the model's behavior with you.
• Handling edge cases: What should annotators do with ambiguous requests, harmful queries, or questions outside their expertise? Clear guidance prevents annotators from improvising inconsistently.
• Quality thresholds: What makes a response "good enough" versus requiring revision? Concrete examples of acceptable and unacceptable responses help calibrate annotator judgment.

The Dolly dataset from Databricks provides a useful example. Its guidelines asked annotators to write creative and brainstorming tasks as well as question-answering tasks. Classification and summarization covered additional instruction types; information extraction did too. Annotators retained freedom within each category. This approach recognized that creativity flourishes within constraints, and that annotators need enough structure to be consistent while retaining enough freedom to produce natural, varied examples. Databricks also encouraged annotators to write about topics they personally found interesting or knew well, producing more authentic and detailed responses than tasks assigned at random would generate.

Good annotation guidelines also specify what to avoid. Instructions that duplicate content from well-known copyrighted sources, instructions designed to elicit harmful responses, and responses that reflect personal opinions as facts all represent pitfalls that explicit guidance can prevent. Annotators are more likely to avoid problematic patterns when those patterns are named and explained than when they are expected to deduce the right approach from general principles.

Human annotation introduces human variability. Different annotators bring different writing styles, knowledge levels, and interpretations of guidelines. Quality control mechanisms help maintain consistency across a diverse pool of contributors and ensure that the resulting dataset reflects the intended quality standards rather than the idiosyncrasies of individual annotators.

The most important quality control mechanism is the feedback loop. Annotators need timely, specific feedback on their work to improve and to understand how the guidelines apply to specific cases. Feedback that arrives days after annotation is completed is less useful than immediate review. Many annotation programs use a tiered model where senior annotators review a sample of each contributor's work. This provides written feedback on specific examples. This feedback is then discussed in group calibration sessions where annotators can ask questions and ensure they share a common interpretation of the guidelines.

The main quality control practices in annotation programs include:

• Inter-annotator agreement: Having multiple annotators label the same examples reveals disagreements that indicate unclear guidelines or difficult cases. When two annotators produce very different responses to the same instruction, it signals either an ambiguous task or a need for guideline refinement.
• Expert review: Senior annotators review a sample of outputs. This provides feedback and identifying systematic issues. This creates a feedback loop where common problems are caught early and addressed through guideline updates or additional training.
• Iterative refinement: Guidelines evolve based on observed problems, with regular calibration sessions to realign annotator understanding. This ongoing process acknowledges that annotation guidelines are living documents that improve through use.
• Rejection and revision cycles: Rather than accepting all submitted work, high-quality annotation programs reject examples that fall below quality thresholds, requiring revision or replacement. This creates accountability and prevents the dataset from gradually accumulating low-quality examples.

Inter-annotator agreement deserves special attention because it serves two purposes simultaneously. Measuring agreement identifies areas where annotators disagree, signaling guideline gaps. But the process of having annotators attempt the same tasks also reveals the inherent difficulty of some instructions: if even expert annotators cannot agree on the best response, that instruction type may not belong in the training set at all.

Human annotation is expensive. Professional annotators cost $15-50+ per hour depending on the task complexity and required expertise. A single high-quality instruction-response pair might take 5-15 minutes to create, putting the cost per example between $1-15. These costs accumulate quickly when building datasets of any meaningful size. When you factor in guideline development, annotator training, quality review overhead, and revision cycles, the true cost per usable example is often two to three times the direct annotation cost.

This cost creates a fundamental tradeoff: you can have either a small dataset of exceptional quality or a larger dataset of variable quality. The InstructGPT paper used only about 13,000 human demonstrations, a tiny dataset by pre-training standards, but each example was carefully crafted by selected contractors who underwent extensive training. This choice reflected a bet that quality would matter more than quantity for teaching instruction-following behavior. That bet turned out to be correct: the model trained on 13,000 high-quality demonstrations substantially outperformed the model trained on pre-training data alone, despite the enormous size difference.

These costs explain why purely human-annotated instruction datasets remain relatively small. Even well-funded projects typically cap at tens of thousands of examples, not the millions common in pre-training. This scale limitation motivates the alternative approaches described in the following sections, each of which trades some aspect of human quality for the ability to generate data more efficiently.

An important practical point is that annotation costs are often dominated not by the per-example production cost but by the upfront investments: guideline development, annotator recruitment and screening, training programs, quality review infrastructure, and iteration cycles after discovering problems. These fixed costs mean that small annotation projects are often more expensive per example than large ones, creating economies of scale that favor organizations that can commit to building substantial datasets.

Template-based generation addresses the scale limitation of human annotation by programmatically converting existing NLP datasets into instruction format. The insight is that tasks like sentiment classification, question answering, and summarization already exist in large annotated datasets. They just need to be reframed as natural language instructions. This reframing turns supervised learning labels into the kind of instruction-response pairs that teach a model how to follow your requests.

Think of template-based generation as translating an old textbook into a new language. The knowledge is already there, carefully organized and validated by years of use. The work is not to create new knowledge but to present the existing knowledge in a form that the student can engage with. Existing NLP datasets contain a large reservoir of structured knowledge about language tasks. Template-based generation makes it usable by translating structured annotations into natural language instructions.

The key insight here is that the diversity of existing NLP benchmarks maps almost perfectly onto the diversity of tasks that users want from instruction-following models. Question answering datasets contain questions and answers. Summarization datasets contain documents and summaries. Entailment datasets contain premise-hypothesis pairs and logical relationships. Each of these can be cast as an instruction-following task with minimal transformation. The challenge is not conceptual but practical: writing enough templates to cover the variety of ways users might phrase the same underlying task.

A deeper benefit of template-based generation is that it simultaneously creates instruction data and tests the model's ability to generalize across task formulations. When a model trained on template-converted data encounters a novel phrasing at inference time, it can succeed only if it has learned the underlying task concept rather than specific template patterns. This requirement forces the model to build representations of what tasks mean that generalize across phrasing.

Consider the Stanford Sentiment Treebank, a dataset with sentences labeled as positive or negative. In its original form, an example looks like:

With templates, we can transform this into an instruction-following example:

This transformation preserves the supervised signal while presenting it in a format that teaches instruction following. The model learns to classify sentiment and to respond appropriately when you ask it to perform classification. The distinction matters: instruction following requires understanding what is being asked, not just producing the correct label. A model trained purely on classification labels has no reason to produce natural language responses. A model trained on instruction-formatted examples learns to interpret requests and respond in kind.

The same transformation applies across the entire NLP ecosystem. Named entity recognition datasets can become "identify all persons mentioned in this text" tasks. Semantic textual similarity datasets can become "tell me whether these two sentences mean the same thing" tasks. Coreference resolution datasets can become "what does 'it' refer to in this sentence" tasks. Every NLP task that has been formulated as a supervised learning problem can, in principle, be reframed as an instruction-following task.

The output shows how a single input "The acting was superb..." generates diverse training examples. By seeing the same content formatted differently, the model learns that sentiment analysis is independent of the specific prompt wording. This tolerance for phrasing variations matters in deployment, where users express the same intent in many different ways.

Notice also that the output format varies across templates. Some produce a single word, others produce a full sentence. This variation teaches the model to adapt its response format to the phrasing of the request. If you ask a simple yes/no question, the model should give a concise answer. If you ask for analysis, the model should provide more context. Template design choices have lasting effects on model behavior, making them more consequential than they might initially appear.

Effective templates share several characteristics that make them valuable for instruction tuning. Understanding these principles helps you create better templates and recognize quality issues in existing template collections. A template collection that violates these principles will produce training data that teaches undesirable patterns, which can be difficult to detect until the model is deployed and users begin noticing unusual behavior.

Templates should vary in phrasing to prevent the model from overfitting to specific wordings. If every sentiment classification instruction starts with "Classify the sentiment," the model may fail when encountering "What's the tone of this text?" Variation teaches the model to recognize the underlying task regardless of surface-level differences in how it is expressed. The goal is to teach the concept of sentiment classification, not the specific phrasing pattern "Classify the sentiment of..."

Templates should sound natural, as if you wrote them in the course of ordinary work. Stilted or overly formal phrasing creates a mismatch between training data and your queries. The goal is to expose the model to the kinds of requests it will receive in practice, not to construct perfectly grammatical but unnaturally precise instructions. Real users make grammatical errors, use informal language, and express themselves imprecisely. Templates that are too polished may hurt generalization to real-world queries.

Templates should include diverse output formats: sometimes a single word, sometimes a full sentence, to teach flexible response styles. You have different preferences for response verbosity, and a well-trained model should be able to adapt. You might want a quick answer, or you might appreciate context and explanation. Training on templates that produce varied output formats teaches the model to calibrate its verbosity to the style of the request.

The FLAN collection demonstrates these principles at scale. It includes templates for over 60 datasets spanning question answering, sentiment analysis, natural language inference, coreference resolution, and many other tasks. Each dataset has multiple templates, creating millions of instruction examples from existing supervised data. This massive scale would be impossible to achieve through human annotation alone, and the resulting model showed strong zero-shot generalization that validated the template diversity approach.

These examples illustrate how the same underlying logical relationship can be cast as different instructional tasks, ranging from simple classification to more complex reasoning queries. By varying the prompt structure, the model learns to identify the core task regardless of how it is phrased. The logical relationship between premise and hypothesis remains constant, but the way we ask about that relationship changes. This teaches the model that natural language inference is a concept, not a specific prompt format.

To make template-based generation concrete, let us walk through the complete process of converting a small NLP dataset. We will use a hypothetical named entity recognition dataset, which identifies people, organizations, and locations in text.

Step 1: Understand the original data format. An NER dataset typically contains sentences paired with labeled spans. A single example might be: "Barack Obama was born in Hawaii" with labels indicating that "Barack Obama" is a PERSON and "Hawaii" is a LOCATION.

Step 2: Design templates covering different phrasings of the same task. We want templates that ask for entity extraction in several natural ways. The templates should vary in how they name the entity types and how they ask for the output.

Step 3: Design the output format for each template. Some templates might expect a comma-separated list; others might expect a structured listing by entity type. The output format should match what a user would naturally expect given the way the instruction is phrased.

Step 4: Apply the templates to the full dataset. Each original example gets converted into multiple instruction-following examples, one per template. The resulting dataset is several times larger than the original.

Step 5: Validate the conversions. Spot-check converted examples to ensure the templates produce natural-sounding instructions and correct outputs. Pay special attention to edge cases, such as examples with no entities of a particular type, which might require special handling in the templates.

This five-step process generalizes to any NLP dataset. The key judgment calls are in template design: choosing phrases that sound natural, covering enough variation, and designing output formats that are both correct and teachable.

While template-based generation provides scale, it has significant limitations that you must understand and address. Recognizing these limitations helps you make informed decisions about when template-based data is appropriate and when other approaches are needed. A dataset built entirely from template conversions will produce a model with predictable failure modes that reflect the structural limitations of the approach.

The resulting instructions tend to be formulaic, lacking the variety of real queries. Templates follow patterns, and no matter how many templates you create, they cannot capture the full messiness and creativity of how humans phrase requests. Users make typos, use slang, provide unclear context, and phrase things in unexpected ways. Template-based data, by contrast, is clean and predictable. This creates a distribution shift between training data and inference: the model sees perfectly formed instructions during training but encounters imperfect ones at inference time.

Templates also inherit the task distribution of existing datasets, which skew heavily toward classification and extraction tasks rather than open-ended generation. The NLP community has historically focused on tasks with clear right answers that enable straightforward evaluation. This means template-converted data overrepresents tasks like sentiment classification, named entity recognition, and question answering, while underrepresenting creative writing, advice-giving, and open-ended exploration. A model trained primarily on template-converted data will excel at structured tasks but may struggle when asked to write a persuasive essay or explain a concept in accessible terms.

Perhaps most importantly, template-based data does not teach creative or conversational abilities. A model trained only on converted classification datasets will not learn to write poetry, explain concepts in simple terms, or engage in multi-turn dialogue. These capabilities require training data that demonstrates them explicitly. Template-based generation works best as a complement to other data sources, not a replacement for them. The FLAN approach achieved strong zero-shot performance on classification-style tasks precisely because those were the kinds of tasks represented in the template-converted data, but contemporary models need broader capabilities than FLAN delivered.

Seed task expansion bridges the gap between expensive human annotation and mechanical template conversion. The approach starts with a small set of carefully crafted seed examples, then uses various techniques to expand this seed into a larger, more diverse dataset. This method preserves the quality and naturalness of human-written examples while achieving scale through systematic augmentation. Think of it as using master-crafted templates as a starting point and then having skilled craftspeople produce variations that respect the spirit of the original.

The motivation for seed expansion comes directly from the limitations we identified in both human annotation and template conversion. Human annotation produces excellent examples but at prohibitive cost. Template conversion produces many examples but lacks naturalness and creative breadth. Seed expansion attempts to get the best of both approaches: by starting from human-quality seed examples and expanding them through principled techniques, we can build datasets that are larger than pure human annotation can produce while retaining more naturalness than pure template conversion.

The key insight in seed expansion is that a well-chosen example encodes information about an entire class of similar examples. A seed that demonstrates how to answer a question about a historical event implicitly teaches the model about the structure of historical questions more generally. A seed that demonstrates creative brainstorming implies a space of similar creative tasks. By expanding from carefully designed seeds, we can efficiently populate the space of possible instructions without needing to enumerate every possible instance explicitly.

A good seed set is small but carefully curated. The philosophy behind seed expansion recognizes that a few excellent examples, chosen to cover the space of possible instructions, can serve as anchors for generating many more examples. The key is ensuring that the seeds represent the diversity you want in the final dataset. A seed set that covers only question answering and summarization will produce expansions that cover only those tasks, no matter how many expansion techniques you apply.

A well-designed seed set should demonstrate the following properties:

• Task diversity: Including examples from many task categories (creative writing, question answering, coding, analysis, etc.) ensures that expansion can generate examples across the full range of desired capabilities.
• Format diversity: Showing different input/output structures (single-turn, multi-turn, with/without context) teaches the model that instructions can take many forms.
• Complexity range: Spanning from simple lookups to multi-step reasoning ensures the model learns to handle both easy and difficult requests.
• Style variety: Demonstrating different appropriate tones and verbosity levels teaches the model to adapt its communication style to different contexts.

The Self-Instruct paper, which the next chapter explores in detail, started with just 175 seed tasks. Each seed was written to be distinct, covering a different aspect of instruction-following capability. This small but diverse seed set enabled the generation of tens of thousands of additional examples while maintaining quality and coverage. The success of Self-Instruct demonstrated that the seed set, not the expansion volume, is the critical determinant of final dataset quality. Expanding from 175 excellent seeds produced better results than expanding from 500 mediocre seeds.

Designing seed sets is itself a creative challenge that rewards careful thought. A practical approach is to enumerate the capabilities you want the final model to have, then write seeds that clearly demonstrate each capability. Creative writing, factual question answering, logical reasoning, code generation, summarization, translation, advice-giving, and task decomposition are examples of capabilities worth explicit seed coverage. Within each capability, seeds should cover different topics and difficulty levels, with varied response lengths to signal the range that expansion should produce.

Once you have a seed set, several techniques can expand it without requiring additional human annotation. These techniques use the structure and content of existing examples to generate new variations that preserve the essential characteristics while adding diversity. The goal is to fill in the space around each seed example, populating the neighborhood of each capability cluster with many closely related examples so the model encounters that capability in many forms.

Paraphrasing generates alternative wordings for existing instructions. The goal is to teach the model that the same task can be expressed in many different ways. You might say "Write a poem" or "Compose a poem" or "Create a poem," and the model should recognize these as equivalent requests. Paraphrasing is particularly valuable for instructions that users are likely to phrase in many different ways, such as creative writing tasks or general knowledge questions. When paraphrasing is done by another language model rather than through rules, it can produce remarkably natural variations that capture stylistic subtleties a rule-based system would miss.

The coding variations demonstrate a pattern common in real instruction datasets: a single structural template ("Write a Python function that...") applied to many different problems. Each resulting instruction is distinct and useful as a training example, but the cost of creation is minimal because the structure is reused. The creative writing variations work similarly, substituting different topics into a fixed poetic form. For both types, the variation is in the content, not the format, so the model learns the task structure from the seed and content generalization from the variations.

A third augmentation technique is complexity scaling, where we take simple seed examples and create more complex versions by adding constraints, combining multiple sub-tasks, or requiring multi-step reasoning. A simple "Summarize this article in one sentence" can be scaled to "Summarize this article in one sentence, then identify the three most important supporting points." A simple "Classify this email as spam or not spam" can be scaled to "Classify this email as spam or not spam, explain your reasoning, and suggest how to improve the spam filter to catch similar messages." Complexity scaling ensures the dataset covers the full range of difficulty that users might require.

When expanding from seeds, maintaining diversity matters. Without careful sampling, expansion tends to produce many similar examples clustered around popular seed patterns while neglecting rarer but important task types. This clustering reduces the effective size of the dataset and can lead to models that perform well on common tasks but poorly on less frequent ones. The problem is analogous to class imbalance in supervised learning: if 80% of your training examples are sentiment classification tasks, the model will be very good at sentiment classification but mediocre at everything else.

Diversity sampling techniques help ensure balanced coverage across task types and linguistic patterns. The goal is not to produce equal numbers of every type, but to prevent any type from becoming so dominant that it crowds out others. A dataset that covers many task types with adequate representation for each produces more capable, general-purpose models than one that over-specializes in any particular direction.

The key diversity mechanisms used in practice include:

• Category balancing: Ensuring expanded data maintains representation across all task categories prevents any single category from dominating the training signal. This often requires actively sampling underrepresented categories more heavily during expansion.
• Embedding-based filtering: Using text embeddings to detect and remove examples too similar to existing ones keeps the dataset diverse at a semantic level, catching redundancy that surface-level metrics would miss.
• Verb/topic tracking: Monitoring the distribution of instruction verbs and topics to prevent overconcentration helps identify when expansion is gravitating toward particular patterns that need to be counterbalanced.

The diversity score captures how evenly distributed the instruction patterns are. A dataset dominated by "Write..." instructions scores lower than one with balanced representation across different task types. This metric provides a quantitative way to track diversity during expansion and flag potential problems before they affect model training. Monitoring diversity metrics throughout the expansion process, rather than only at the end, allows course correction before the imbalance becomes entrenched.

Diversity measured by instruction verb is a crude proxy for the full diversity we care about. Two instructions that both start with "Explain" may cover completely different knowledge domains and require very different capabilities. More sophisticated diversity measures use embedding-based methods to compare instructions in a semantic space, identifying redundancy at the level of meaning rather than surface form. The verb-based entropy measure above is a useful quick check, but it should be supplemented with semantic diversity analysis.

Whether data comes from human annotation, template conversion, or seed expansion, quality filtering is essential. Raw data inevitably contains errors and duplicates, along with low-quality examples that can harm model training. A systematic filtering pipeline removes these problems while preserving valuable examples. The goal is to maximize the signal-to-noise ratio in the training data. This keeps every example the model sees contributes positively to its learning.

Think of quality filtering as the editorial process for a textbook. Even if every chapter is written by a qualified expert, the final textbook benefits from editing to remove redundancy, fix errors, standardize formatting, and improve clarity. The editing process does not diminish the textbook; it makes it more effective as a teaching tool. Quality filtering serves the same function for instruction datasets: it turns a collection of examples with mixed quality into a coherent, high-quality training set.

Quality filtering operates on several levels simultaneously. Some problems are structural (the instruction is too short to be meaningful), some are semantic (the output does not answer the instruction), and some are relational (this example is nearly identical to one already in the dataset). A complete filtering pipeline addresses all three levels, using different techniques suited to each type of problem. Applying only one level of filtering leaves the others unaddressed, producing a dataset that may look clean on surface metrics while still containing significant problems.

Basic heuristics catch many obvious problems. These filters are fast to compute and can be applied to millions of examples without significant computational cost. While simple, they remove a substantial portion of problematic data before more expensive filtering stages. Running cheap filters first and expensive filters later is a key design principle for production-scale filtering pipelines, because it reduces the number of examples that need to pass through the costly stages.

The filter rules above capture several common failure modes. Too-short instructions often arise from truncation errors during data processing or from annotators who did not complete their work. Error markers in outputs indicate cases where the model used to generate responses refused or failed, producing training examples that teach refusal rather than capability. Instructions that are simply repeated in the output represent a degenerate case where no actual task was performed. Each of these failure modes, if left unfiltered, teaches the model a bad behavior pattern that will manifest during inference.

The specific thresholds (minimum 3 words for instructions, maximum 2000 words for outputs) require calibration based on the expected task distribution. A dataset focused on creative writing might need higher maximum output length thresholds, while one focused on classification might allow shorter outputs. Understanding the data well enough to set appropriate thresholds is itself a form of quality work.

Duplicate or near-duplicate examples waste training compute and can cause the model to memorize specific examples rather than learn general patterns. Deduplication operates at multiple levels, from exact string matching to semantic similarity detection. Each level catches different types of redundancy, and a complete deduplication strategy applies multiple levels in sequence. Different types of duplicates have different effects: exact duplicates cause straightforward overfitting, while near-duplicates can make the model generalize too rigidly from a family of similar examples.

Exact deduplication removes identical instruction-output pairs. This is the simplest form of deduplication but catches surprisingly many duplicates, especially in data generated through automated expansion or scraped from multiple sources. When multiple data pipelines contribute to the same dataset, it is common for the same examples to appear multiple times under slightly different formatting. Exact deduplication on normalized text (lowercased, whitespace-standardized) catches these cases efficiently.

The high similarity scores identify pairs that differ only slightly in wording, allowing us to flag potential redundancies that exact matching misses. Filtering these near-duplicates ensures the training compute is spent on distinct examples. The threshold for near-duplicate detection requires careful tuning: too low and you remove legitimate variations; too high and redundant examples slip through. TF-IDF similarity is a useful approximation, but production systems typically use learned text embeddings that better capture semantic similarity across diverse phrasings of the same task.

Human-written heuristics can only catch obvious problems. For subtler quality issues, including unclear instructions, factually incorrect outputs, or stylistically poor responses, model-based filtering provides a more detailed assessment. These approaches use the judgment capabilities of language models to evaluate aspects of quality that are difficult to capture in simple rules. Think of model-based scoring as hiring a skilled editor to review each example after the initial mechanical checks have been completed.

The approach uses a language model to score or classify the quality of each example. The model can be prompted to evaluate various dimensions of quality: clarity of the instruction, helpfulness of the response, correctness of any factual claims, and appropriateness of the tone. This multi-dimensional evaluation captures quality aspects that a single score would obscure. An instruction might be perfectly clear but receive a poor response; a response might be helpful but slightly off-topic; a factually correct response might be formatted in a way that makes it hard to read.

The key insight is that a language model used for quality scoring does not need to be the same model being trained. A smaller but capable model, perhaps the best publicly available open model at the time, can serve as an effective quality filter for data that will train a larger model. This asymmetry makes model-based quality scoring practical even when the target model is very large and expensive to evaluate.

The simulated scores reflect the heuristic that detailed, well-structured responses (like the relativity explanation) are more valuable for instruction tuning than brief or generic outputs. By scoring examples, we can filter out low-value data or curriculum-train the model on high-quality examples first. This curriculum approach exposes the model to the best examples early in training, establishing good patterns before introducing noisier data. Research suggests that curriculum learning, where training proceeds from simpler, higher-quality examples to more complex ones, can improve both convergence speed and final performance.

In production systems, quality filtering often uses a tiered approach: fast heuristic filters remove obvious problems, then more expensive model-based scoring handles the remaining examples. This balances thoroughness with computational cost. Running a language model over every example is expensive, so reserving model-based filtering for examples that pass initial heuristic checks makes the pipeline practical at scale. A typical production pipeline might reduce the dataset by 20-30% through heuristic filters, then apply model-based scoring only to the remaining examples, reducing the overall compute burden by an equivalent proportion.

Combining all filtering stages into a unified pipeline ensures consistent quality across the dataset. A well-designed pipeline applies filters in order of computational cost, with cheap heuristic filters first and expensive model-based scoring last. This design minimizes the total compute required while still achieving thorough filtering. The pipeline should also track statistics at each stage, enabling retrospective analysis of where data quality problems originate, which informs improvements to the upstream data creation processes.

Real instruction-tuning datasets rarely use a single creation method. Instead, they combine multiple sources to balance their respective strengths and weaknesses. This combination recognizes that no single approach excels at everything: human annotation provides quality but not scale, templates provide scale but not naturalness, and seed expansion provides a middle ground that still requires careful curation. The art of building a good mixed dataset lies in understanding what each source contributes and calibrating the mix accordingly.

Think of a mixed instruction dataset as a recipe with multiple ingredients. Each ingredient serves a purpose: human-written examples provide the flavor that makes the dish distinctive, template-converted examples provide the volume that makes it filling, and seed-expanded examples add the variety that keeps it interesting. Getting the proportions right requires understanding the role of each component. Too much of any single ingredient overwhelms the others and produces a lopsided result.

The decision of how to mix sources should be driven by the capabilities you want the final model to have. If the primary goal is strong performance on structured tasks like question answering and classification, template-converted data should dominate. If the primary goal is natural, conversational behavior, human-written examples should be proportionally larger. If you want strong creative writing capabilities, seed-expanded creative examples need sufficient representation to teach that skill effectively.

The FLAN-T5 model, for instance, combines template-converted versions of 62 existing datasets with chain-of-thought examples and dialogue data. This mixture teaches both task-specific skills and more general conversational abilities. The template-converted data provides broad coverage of NLP tasks, while the chain-of-thought and dialogue data teach more sophisticated reasoning and interaction patterns. The specific choice of which task datasets to include was guided by the goal of broad generalization. This keeps the resulting model could handle a wide range of user requests without task-specific fine-tuning.

More recent models like Llama 2 Chat and Claude use instruction datasets that combine human demonstration data from annotators, synthetic data generated by the models themselves, and filtered internet data containing instruction-like content. Each source serves a different purpose: annotator data for quality and safety alignment, synthetic data for scale and coverage, and filtered internet data for diverse phrasing and topics. The relative proportions are carefully tuned through ablation experiments that measure the contribution of each source to downstream performance.

This visualization highlights the strategic balance between volume (template-converted) and quality (human-written/seed-expanded) required for effective instruction tuning. While automated methods provide the bulk of the data, the smaller high-quality components are essential for steering the model's style and capabilities. The human-written demonstrations, though small in number, have outsized influence on how the model behaves, establishing patterns that the larger automated data then reinforces.

An important consideration in mixing sources is weighting during training. Simply concatenating all sources and training on the concatenation gives each example equal weight, but this means the large template-converted component dominates the gradient signal regardless of its relative quality. Production training pipelines often over-sample smaller high-quality sources, presenting each human-written example multiple times per epoch while presenting each template-converted example fewer times. This reweighting ensures that quality guides the model's behavior even when quality examples are outnumbered by volume examples.

Instruction data creation faces several fundamental challenges that shape what is possible and what tradeoffs are unavoidable. Understanding these limitations matters for building datasets and for interpreting the behavior of models trained on them. When a model behaves unexpectedly, the root cause can often be traced back to a specific property of its training data, and recognizing that connection is the first step toward fixing the problem.

Quality-diversity tradeoffs are unavoidable. Human annotation produces high-quality data but struggles to cover the full space of possible instructions. Template-based methods achieve coverage but produce data that lacks the naturalness of your queries. Seed expansion can balance these concerns but requires careful monitoring to prevent quality degradation as the dataset grows. No single approach solves all problems, which is why production systems invariably combine multiple methods. The right balance depends on the target use case: a model intended primarily for structured tasks can lean more heavily on templates, while one intended for creative assistance requires more human-quality creative examples.

Dataset biases propagate to model behavior. If your instruction data overrepresents certain topics, phrasings, or response styles, the fine-tuned model will reflect those biases. This is particularly insidious because biases in training data are often invisible until they manifest as unexpected model behaviors in deployment. For instance, a dataset heavy on formal business writing might produce a model that struggles with casual conversation, while one focused on American English idioms might confuse you if you live in other English-speaking regions. Annotation programs that draw from a narrow demographic pool will produce models that reflect the communication preferences of that pool, which may not generalize to all users. Addressing demographic bias in instruction data requires deliberate efforts to recruit diverse annotators and audit data for representation imbalances.

Scaling data creation remains expensive despite advances in automation. Template-based and model-assisted generation have reduced the cost dramatically compared to pure human annotation, but creating diverse, high-quality instruction data still requires significant human effort. The cost estimates we calculated earlier for human annotation represent a floor, not a ceiling. Real projects often spend considerably more on iteration, quality control, and fixing problems discovered during training. This expense motivates ongoing research into more efficient data creation methods, including the Self-Instruct approach described in the next chapter, which uses a language model to generate new instruction examples from a small seed set, reducing the human annotation burden substantially.

Evaluation lags behind creation. While many techniques exist for creating instruction data, evaluating its quality at scale remains challenging. Automated metrics capture surface-level properties but miss deeper quality issues. Human evaluation is expensive and slow. This asymmetry means data problems often remain undiscovered until after a model has been trained and its behavior observed, creating costly iteration cycles where the root cause of a behavioral problem must be traced back through layers of data processing to its origin. Investing in better data evaluation methods is as important as investing in better data creation methods, though it receives less research attention.

The problem of "instruction distribution shift" deserves explicit attention. The distribution of instructions in a training dataset never perfectly matches the distribution of instructions at deployment. Real users ask questions the dataset authors never anticipated, phrase familiar requests in unfamiliar ways, and combine tasks in unexpected combinations. The gap between training distribution and deployment distribution is a fundamental challenge that no amount of data collection can completely eliminate. The practical response is to design datasets that are diverse enough to cover the relevant space broadly, accept that some distribution shift is inevitable, and build mechanisms for discovering and addressing gaps once the model is deployed.

Instruction data creation bridges the gap between pre-trained language models and instruction-following assistants. This chapter covered four complementary approaches that form the foundation of modern instruction-tuning pipelines.

Human annotation provides the highest quality data but is constrained by cost. Effective annotation requires clear guidelines, quality control mechanisms, and acceptance that scale will be limited. Human-written demonstrations remain valuable for establishing quality standards and covering tasks that require expert knowledge. The InstructGPT result, where 13,000 high-quality demonstrations produced dramatically improved behavior, illustrates that quality can substitute for quantity up to a point. Annotation programs that invest in guideline quality, annotator training, and feedback loops produce substantially better data than those that simply maximize throughput.

Template-based generation converts existing supervised datasets into instruction format. This provides scale and task diversity. Well-designed templates vary in phrasing and output format to teach flexible instruction following. The FLAN collection demonstrated that template-converted data, applied at enough scale, enables zero-shot generalization to unseen tasks. However, templates produce formulaic data that lacks the naturalness of your real interactions, and the task distribution of existing NLP datasets skews toward classification rather than the open-ended generation tasks that characterize modern AI assistants.

Seed task expansion starts with a small set of carefully curated examples and uses various techniques to grow the dataset while maintaining diversity. Good seed sets demonstrate variety across task categories, formats, complexity levels, and styles. Expansion techniques include paraphrasing, input variation, complexity scaling, and diversity sampling to prevent clustering around popular patterns. The Self-Instruct approach, covered in the next chapter, extends this idea by using a language model to generate the expansion itself, enabling dataset growth at a scale that purely human expansion cannot match.

Quality filtering removes problematic examples regardless of their source. A complete pipeline combines fast heuristic filters for obvious problems, deduplication to prevent memorization, and model-based scoring for subtler quality issues. Filtering statistics help identify systematic problems in data creation processes, enabling targeted improvements upstream. The ordering of filters from cheapest to most expensive is a practical necessity at production scale, where millions of examples must be processed efficiently.

The next chapter explores Self-Instruct, a specific seed expansion technique that uses a language model to generate new instruction examples from a small seed set. This approach has enabled the creation of instruction datasets at scale without massive human annotation efforts, democratizing access to instruction-tuning beyond well-funded research labs. Understanding the data creation foundations covered here will help you appreciate both the power of Self-Instruct and its limitations, which are directly connected to the quality and diversity properties we have examined throughout this chapter.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about instruction data creation methods.