Self-Instruct › Worked Example: End-to-End Self-Instruct Run

To solidify the concepts, let us trace through a complete mini Self-Instruct run step by step, with concrete numbers and decisions at each stage. We will use three seed tasks and run one full iteration of the pipeline, tracking every decision from instruction generation through final acceptance or rejection.

Setup. We begin with three seed tasks:

1. "Classify the sentiment of the movie review." (input: "The film was boring.", output: "Negative")
2. "Write a haiku about spring." (input: "", output: "Cherry blossoms fall / Soft petals on the warm breeze / New life just begins")
3. "Convert 100 degrees Fahrenheit to Celsius." (input: "100 F", output: "37.78 C")

We sample all three as demonstrations and prompt the model to generate new instructions.

Step 1: Instruction Generation. The model receives a prompt showing the three seed tasks and produces the following candidates:

• "4. Determine if the given statement is true or false."
• "5. Write a short poem about autumn."
• "6. Translate the phrase to German."
• "7. Classify the given news headline as sports, politics, or entertainment."

Step 2: Similarity Filtering. We compute ROUGE-L scores between each candidate and all existing instructions. The scores for "Write a short poem about autumn" against our haiku instruction are:

The two instruction strings are "Write a haiku about spring" (reference, 5 tokens) and "Write a short poem about autumn" (candidate, 6 tokens).

The longest common subsequence of these two token lists is ["Write", "a", "about"] with length 3. We compute:

$$\text{Precision}_{\text{LCS}} = \frac{3}{6} = 0.500$$ $$\text{Recall}_{\text{LCS}} = \frac{3}{5} = 0.600$$ $$\text{ROUGE-L} = \frac{2 \times 0.500 \times 0.600}{0.500 + 0.600} = \frac{0.600}{1.100} \approx 0.545$$

where:

• $3$ is the LCS length (tokens "Write", "a", "about" appear in both instructions in the same relative order)
• $6$ is the number of tokens in the candidate instruction
• $5$ is the number of tokens in the reference instruction

A score of 0.545 is below our threshold of 0.7, so "Write a short poem about autumn" passes the similarity filter. If the candidate had been "Write a haiku about autumn," the LCS would be ["Write", "a", "haiku", "about"] with length 4, giving ROUGE-L of approximately 0.86, which would trigger rejection. The other three candidates score below 0.3 against all existing instructions and pass easily.

Step 3: Task Classification. We prompt the model to classify each passing candidate:

• "Determine if the given statement is true or false." → Classification (binary labels: true/false)
• "Write a short poem about autumn." → Generation
• "Translate the phrase to German." → Generation
• "Classify the given news headline as sports, politics, or entertainment." → Classification

Step 4: Instance Generation. For each passing instruction, we apply the appropriate strategy:

For "Classify the given news headline as sports, politics, or entertainment" (output-first): We first request a "Sports" example. The model generates: Label: Sports, Input: "Local team wins championship in overtime thriller." We then request "Politics": Label: Politics, Input: "Senate votes on new infrastructure bill." Finally "Entertainment": Label: Entertainment, Input: "Pop star announces world tour dates."

For "Write a short poem about autumn" (input-first): There is no separate input needed for this task (it is an instruction-only task), so the model generates only an output. The model produces: "Golden leaves descend / The earth prepares for its rest / Quiet beauty fades."

Step 5: Validation. All four instances pass length and format filters. The final accepted instances join the task pool, growing it from 3 to 7 tasks. In the next iteration, all 7 tasks are available as demonstration candidates. This provides richer context for the next round of instruction generation.

This walkthrough reveals the compounding nature of the pipeline: each accepted task makes future generations richer. After 100 such iterations, the pool contains hundreds of tasks spanning many domains, and the model's demonstrations at each step reflect that accumulated diversity.