Instruction Data Creation › Template-Based Generation

Template-based generation addresses the scale limitation of human annotation by programmatically converting existing NLP datasets into instruction format. The insight is that tasks like sentiment classification, question answering, and summarization already exist in large annotated datasets. They just need to be reframed as natural language instructions. This reframing turns supervised learning labels into the kind of instruction-response pairs that teach a model how to follow your requests.

Think of template-based generation as translating an old textbook into a new language. The knowledge is already there, carefully organized and validated by years of use. The work is not to create new knowledge but to present the existing knowledge in a form that the student can engage with. Existing NLP datasets contain a large reservoir of structured knowledge about language tasks. Template-based generation makes it usable by translating structured annotations into natural language instructions.

The key insight here is that the diversity of existing NLP benchmarks maps almost perfectly onto the diversity of tasks that users want from instruction-following models. Question answering datasets contain questions and answers. Summarization datasets contain documents and summaries. Entailment datasets contain premise-hypothesis pairs and logical relationships. Each of these can be cast as an instruction-following task with minimal transformation. The challenge is not conceptual but practical: writing enough templates to cover the variety of ways users might phrase the same underlying task.

A deeper benefit of template-based generation is that it simultaneously creates instruction data and tests the model's ability to generalize across task formulations. When a model trained on template-converted data encounters a novel phrasing at inference time, it can succeed only if it has learned the underlying task concept rather than specific template patterns. This requirement forces the model to build representations of what tasks mean that generalize across phrasing.

Consider the Stanford Sentiment Treebank, a dataset with sentences labeled as positive or negative. In its original form, an example looks like:

With templates, we can transform this into an instruction-following example:

This transformation preserves the supervised signal while presenting it in a format that teaches instruction following. The model learns to classify sentiment and to respond appropriately when you ask it to perform classification. The distinction matters: instruction following requires understanding what is being asked, not just producing the correct label. A model trained purely on classification labels has no reason to produce natural language responses. A model trained on instruction-formatted examples learns to interpret requests and respond in kind.

The same transformation applies across the entire NLP ecosystem. Named entity recognition datasets can become "identify all persons mentioned in this text" tasks. Semantic textual similarity datasets can become "tell me whether these two sentences mean the same thing" tasks. Coreference resolution datasets can become "what does 'it' refer to in this sentence" tasks. Every NLP task that has been formulated as a supervised learning problem can, in principle, be reframed as an instruction-following task.

The output shows how a single input "The acting was superb..." generates diverse training examples. By seeing the same content formatted differently, the model learns that sentiment analysis is independent of the specific prompt wording. This tolerance for phrasing variations matters in deployment, where users express the same intent in many different ways.

Notice also that the output format varies across templates. Some produce a single word, others produce a full sentence. This variation teaches the model to adapt its response format to the phrasing of the request. If you ask a simple yes/no question, the model should give a concise answer. If you ask for analysis, the model should provide more context. Template design choices have lasting effects on model behavior, making them more consequential than they might initially appear.

Effective templates share several characteristics that make them valuable for instruction tuning. Understanding these principles helps you create better templates and recognize quality issues in existing template collections. A template collection that violates these principles will produce training data that teaches undesirable patterns, which can be difficult to detect until the model is deployed and users begin noticing unusual behavior.

Templates should vary in phrasing to prevent the model from overfitting to specific wordings. If every sentiment classification instruction starts with "Classify the sentiment," the model may fail when encountering "What's the tone of this text?" Variation teaches the model to recognize the underlying task regardless of surface-level differences in how it is expressed. The goal is to teach the concept of sentiment classification, not the specific phrasing pattern "Classify the sentiment of..."

Templates should sound natural, as if you wrote them in the course of ordinary work. Stilted or overly formal phrasing creates a mismatch between training data and your queries. The goal is to expose the model to the kinds of requests it will receive in practice, not to construct perfectly grammatical but unnaturally precise instructions. Real users make grammatical errors, use informal language, and express themselves imprecisely. Templates that are too polished may hurt generalization to real-world queries.

Templates should include diverse output formats: sometimes a single word, sometimes a full sentence, to teach flexible response styles. You have different preferences for response verbosity, and a well-trained model should be able to adapt. You might want a quick answer, or you might appreciate context and explanation. Training on templates that produce varied output formats teaches the model to calibrate its verbosity to the style of the request.

The FLAN collection demonstrates these principles at scale. It includes templates for over 60 datasets spanning question answering, sentiment analysis, natural language inference, coreference resolution, and many other tasks. Each dataset has multiple templates, creating millions of instruction examples from existing supervised data. This massive scale would be impossible to achieve through human annotation alone, and the resulting model showed strong zero-shot generalization that validated the template diversity approach.

These examples illustrate how the same underlying logical relationship can be cast as different instructional tasks, ranging from simple classification to more complex reasoning queries. By varying the prompt structure, the model learns to identify the core task regardless of how it is phrased. The logical relationship between premise and hypothesis remains constant, but the way we ask about that relationship changes. This teaches the model that natural language inference is a concept, not a specific prompt format.

To make template-based generation concrete, let us walk through the complete process of converting a small NLP dataset. We will use a hypothetical named entity recognition dataset, which identifies people, organizations, and locations in text.

Step 1: Understand the original data format. An NER dataset typically contains sentences paired with labeled spans. A single example might be: "Barack Obama was born in Hawaii" with labels indicating that "Barack Obama" is a PERSON and "Hawaii" is a LOCATION.

Step 2: Design templates covering different phrasings of the same task. We want templates that ask for entity extraction in several natural ways. The templates should vary in how they name the entity types and how they ask for the output.

Step 3: Design the output format for each template. Some templates might expect a comma-separated list; others might expect a structured listing by entity type. The output format should match what a user would naturally expect given the way the instruction is phrased.

Step 4: Apply the templates to the full dataset. Each original example gets converted into multiple instruction-following examples, one per template. The resulting dataset is several times larger than the original.

Step 5: Validate the conversions. Spot-check converted examples to ensure the templates produce natural-sounding instructions and correct outputs. Pay special attention to edge cases, such as examples with no entities of a particular type, which might require special handling in the templates.

This five-step process generalizes to any NLP dataset. The key judgment calls are in template design: choosing phrases that sound natural, covering enough variation, and designing output formats that are both correct and teachable.

While template-based generation provides scale, it has significant limitations that you must understand and address. Recognizing these limitations helps you make informed decisions about when template-based data is appropriate and when other approaches are needed. A dataset built entirely from template conversions will produce a model with predictable failure modes that reflect the structural limitations of the approach.

The resulting instructions tend to be formulaic, lacking the variety of real queries. Templates follow patterns, and no matter how many templates you create, they cannot capture the full messiness and creativity of how humans phrase requests. Users make typos, use slang, provide unclear context, and phrase things in unexpected ways. Template-based data, by contrast, is clean and predictable. This creates a distribution shift between training data and inference: the model sees perfectly formed instructions during training but encounters imperfect ones at inference time.

Templates also inherit the task distribution of existing datasets, which skew heavily toward classification and extraction tasks rather than open-ended generation. The NLP community has historically focused on tasks with clear right answers that enable straightforward evaluation. This means template-converted data overrepresents tasks like sentiment classification, named entity recognition, and question answering, while underrepresenting creative writing, advice-giving, and open-ended exploration. A model trained primarily on template-converted data will excel at structured tasks but may struggle when asked to write a persuasive essay or explain a concept in accessible terms.

Perhaps most importantly, template-based data does not teach creative or conversational abilities. A model trained only on converted classification datasets will not learn to write poetry, explain concepts in simple terms, or engage in multi-turn dialogue. These capabilities require training data that demonstrates them explicitly. Template-based generation works best as a complement to other data sources, not a replacement for them. The FLAN approach achieved strong zero-shot performance on classification-style tasks precisely because those were the kinds of tasks represented in the template-converted data, but contemporary models need broader capabilities than FLAN delivered.