Self-Instruct › Instance Generation
Language AI HandbookAll chapters Self-Instruct › Instance GenerationOne section · Sections in this chapter6 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Once instructions are classified, the pipeline generates complete task instances consisting of inputs and outputs. This stage uses one of two strategies depending on the task type. The choice of strategy directly impacts the quality and balance of the resulting training data. Getting this right is what separates a Self-Instruct pipeline that produces useful training data from one that produces plausible-looking but poorly calibrated examples.
Think of instance generation as the difference between writing an exam question by starting with the answer versus starting with the scenario. If you need to write a question whose correct answer is "photosynthesis," you design a scenario that naturally leads to that answer, creating a coherent and well-motivated question. If you start with the scenario instead, you might accidentally write a question whose obvious answer is something other than what you intended. Output-first generation for classification tasks applies exactly this logic: starting with the intended label ensures that the generated input warrants that label.
The dual-strategy approach also helps with coherence in a more subtle way. For generation tasks, the input context determines what constitutes a good output. A summarization task with a specific article as input calls for a summary that accurately reflects that article's content. If the input and output are generated independently, they may be inconsistent with each other: the summary might not match the article. Input-first generation avoids this problem by establishing the context before generating the response, maintaining a natural dependency between the two.
Instance generation is also where the scale of Self-Instruct becomes most apparent. For each instruction in the pipeline, the model must make an additional API call (or inference pass) to generate the instance. This means that for a dataset of 52,000 instructions, the original paper required roughly 52,000 additional model calls just for instance generation, on top of the calls required for instruction generation and task classification. Understanding this computational cost is important for planning large-scale Self-Instruct runs.
Input-First Approach
For open-ended generation tasks, the model first creates a plausible input, then generates the corresponding output. This approach works well when the input constrains what kind of output makes sense: given a specific article to summarize, there are many valid summaries, but they should all reflect the article's content. By generating the input first, we establish a concrete context that grounds the output generation. The causal arrow runs from input to output, just as it would in a real task scenario.
The input-first approach also tends to produce more natural-sounding instances. When a model generates an input and then responds to it, the result has the coherence of a question-and-answer exchange rather than the slightly artificial feeling of a fabricated example. This naturalness is valuable for training because it teaches the model to handle the kinds of inputs that real users provide, rather than abstract or overly formal inputs that would never appear in practice.
def create_input_first_prompt(instruction, demonstrations):
"""
Generate input first, then output.
Used for open-ended generation tasks.
"""
prompt = f"""For the following task, first generate an appropriate input,
then provide the expected output.
Task: {instruction}
"""
# Add demonstrations
for demo in demonstrations:
prompt += f"Input: {demo['input']}\n"
prompt += f"Output: {demo['output']}\n\n"
prompt += "Input:"
return prompt# Example: generating an instance for a summarization task
summarization_instruction = "Summarize the main points of the given article."
# Mock demonstrations
summarization_demos = [
{
"input": "Scientists discovered a new species of deep-sea fish that produces its own light...",
"output": "Researchers found a bioluminescent fish species in deep ocean waters.",
}
]
input_first_prompt = create_input_first_prompt(
summarization_instruction, summarization_demos
)=== Input-First Prompt === For the following task, first generate an appropriate input, then provide the expected output. Task: Summarize the main points of the given article. Input: Scientists discovered a new species of deep-sea fish that produces its own light... Output: Researchers found a bioluminescent fish species in deep ocean waters. Input:
The prompt explicitly instructs the model to generate the input context first, establishing a scenario that makes the subsequent output generation more natural and coherent. The demonstrations show the model what kind of inputs are appropriate for this task type and how outputs should relate to those inputs. This sequential structure mimics how humans would approach such tasks: first understand the context, then produce a response.
Output-First Approach
For classification tasks, generating labels first and then creating matching inputs produces more balanced datasets. This ordering might seem counterintuitive at first, but it addresses an important practical problem. If you generate inputs first, the model might repeatedly create inputs that warrant the same label, leading to imbalanced class distributions. For example, when generating sentiment examples, a model might produce mostly positive reviews simply because they are easier to write or more common in training data.
The output-first approach inverts the natural language production order to serve a data engineering goal. We are not trying to mimic how humans naturally produce text; we are trying to ensure that our training data has specific distributional properties. By explicitly specifying which label to generate an example for, we force the model to construct inputs that warrant that label, and we can cycle through labels systematically to achieve balance. This is a case where the needs of the training data pipeline diverge from the natural flow of language generation, and the design of the instance generation strategy accommodates that divergence.
def create_output_first_prompt(instruction, possible_labels, demonstrations):
"""
Generate output (label) first, then create matching input.
Used for classification tasks to ensure balanced labels.
"""
prompt = f"""For the following classification task, generate an example
for each possible output label.
Task: {instruction}
Possible labels: {", ".join(possible_labels)}
"""
for demo in demonstrations:
prompt += f"Label: {demo['output']}\n"
prompt += f"Input: {demo['input']}\n\n"
# Request a new example
prompt += "Label:"
return prompt# Example: generating instances for sentiment classification
sentiment_instruction = "Classify the sentiment of the given text."
sentiment_labels = ["Positive", "Negative", "Neutral"]
sentiment_demos = [
{
"output": "Positive",
"input": "This product exceeded all my expectations!",
},
{"output": "Negative", "input": "Terrible service, would not recommend."},
]
output_first_prompt = create_output_first_prompt(
sentiment_instruction, sentiment_labels, sentiment_demos
)=== Output-First Prompt === For the following classification task, generate an example for each possible output label. Task: Classify the sentiment of the given text. Possible labels: Positive, Negative, Neutral Label: Positive Input: This product exceeded all my expectations! Label: Negative Input: Terrible service, would not recommend. Label:
The output-first approach cycles through labels, requesting inputs for each one in turn. This ensures roughly equal representation of each class in the final dataset. By explicitly specifying which label to generate an example for, we force the model to create inputs that warrant that label rather than defaulting to the most common or easiest case. The result is a more balanced dataset that trains models to distinguish between all categories effectively.
