Self-Instruct › Filtering Strategies

Raw generated data contains significant noise: duplicate instructions, low-quality outputs, formatting errors, and examples that are too similar to existing ones. Filtering removes these problematic cases before they contaminate the task pool. Without rigorous filtering, the iterative nature of Self-Instruct would cause errors to compound over time, degrading the quality of each successive generation. Think of the filtering stage as the immune system of the pipeline: it cannot prevent all errors from being generated, but it prevents them from taking hold and multiplying.

The filtering pipeline operates across multiple dimensions simultaneously, each catching a different class of problematic data. ROUGE similarity filtering addresses redundancy. This keeps the dataset grows in new directions rather than repeatedly covering the same ground. Length and format filters address structural quality, catching cases where the model's generation drifted from the expected format. Keyword filters address semantic quality, removing instructions that reference capabilities the model lacks or contexts that cannot be reliably completed without additional information.

A necessary design principle in the filtering architecture is that each filter should be fast and cheap. The entire point of Self-Instruct is to automate data creation at scale; if the filters require expensive model calls, the cost savings evaporate. ROUGE similarity computation is a simple string comparison algorithm. Length checking is a word count. Keyword filtering is substring matching. These cheap filters catch the majority of problematic cases before any expensive model-based evaluation is applied. Reserve expensive filtering for edge cases that heuristics cannot handle.

The composition of filters also matters. Applying them in the right order maximizes efficiency. Keyword filtering should come first because it is the cheapest operation: a simple string check that eliminates instructions referencing images or URLs before any further processing. Length filtering comes next, catching structurally invalid instances before the more expensive ROUGE similarity computation. Similarity filtering comes last because it requires comparing against the entire existing pool, which makes it the most computationally intensive operation. This ordering ensures that the expensive operations see only the candidates that have already passed cheaper checks.

The most important filter prevents near-duplicate instructions from entering the pool. New instructions that are too similar to existing ones add no diversity and waste annotation budget. If two instructions are nearly identical, including both provides little additional signal during training. Self-Instruct uses ROUGE-L similarity to measure instruction overlap, comparing the longest common subsequence between instruction texts.

ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation, using Longest Common Subsequence) was originally developed as an automatic evaluation metric for text summarization, but it serves a different purpose here: measuring lexical overlap between instructions to detect near-duplicates. The LCS-based approach is more sensitive to word order and phrase-level matching than bag-of-words alternatives, which makes it effective for catching paraphrases and minor rewrites of existing instructions. An instruction like "Summarize the article in three sentences" scores high ROUGE-L against "Provide a three-sentence summary of the article" because both share most of the same key words in similar order.

The ROUGE-L score between two texts $A$ and $B$ is computed as an F1 measure that combines precision and recall of the longest common subsequence. Let $\text{LCS}(A, B)$ denote the length of the longest common subsequence between the token sequences of $A$ and $B$, $|A|$ denote the number of tokens in $A$, and $|B|$ denote the number of tokens in $B$. The score is:

$$\text{ROUGE-L}(A, B) = \frac{2 \cdot \text{Precision}_{\text{LCS}} \cdot \text{Recall}_{\text{LCS}}}{\text{Precision}_{\text{LCS}} + \text{Recall}_{\text{LCS}}}$$

where:

• $\text{Precision}_{\text{LCS}} = \dfrac{|\text{LCS}(A, B)|}{|B|}$: the fraction of $B$'s tokens that participate in the longest common subsequence
• $\text{Recall}_{\text{LCS}} = \dfrac{|\text{LCS}(A, B)|}{|A|}$: the fraction of $A$'s tokens that participate in the longest common subsequence
• The F1 combination of the two balances the tendency of precision alone to reward short matches and recall alone to reward long matches

Intuitively, a high ROUGE-L score means the two texts share a long sequence of tokens in the same relative order, suggesting they convey similar content. A score close to 1.0 means the texts are nearly identical. A score close to 0.0 means they share very few tokens in common order.

Simple heuristics catch many problematic generations without requiring complex analysis. These filters check basic structural properties that valid instructions should satisfy. A well-formed instruction must be long enough to communicate a meaningful task but short enough to be actionable without becoming a paragraph-length specification. Outputs must exist and be within a reasonable length range. These seemingly obvious constraints catch a surprising number of failures in practice, because language models occasionally generate degenerate outputs: a single word, an incomplete sentence, or a multi-page stream-of-consciousness.

Length filters also enforce indirect quality signals. Instructions shorter than three words are almost certainly too vague to be useful: "Write." or "Explain something." provide no meaningful task specification. Instructions longer than 150 words are typically compound tasks that would be better split into separate, more focused instructions. Outputs longer than 1000 words suggest the model has generated a lengthy text in response to what should have been a shorter task, possibly because the prompt was misinterpreted or the model entered a generation loop.

The filters successfully accept valid instructions while rejecting those that are too short, repetitive, or empty, preventing low-quality data from polluting the pool. These simple checks are computationally cheap but catch a surprising number of problematic cases, making them an efficient first line of defense. The repetition filter in particular catches a common failure mode where models fall into generation loops, repeating the same word or phrase indefinitely. These outputs look valid superficially (they have the right format) but would provide no training signal.

Certain keywords indicate problematic instructions that should be excluded. These include self-references, meta-commentary, and requests for capabilities the model lacks. Keyword filtering acts as a semantic safety net, catching issues that structural filters would miss. An instruction that passes all length and format checks can still be semantically invalid: "Describe the image shown above" has a perfectly reasonable length and format but cannot be completed by a text-only model.

The keyword list reflects the types of instructions that appear reasonable in isolation but become problematic in the context of a text-only instruction-following dataset. References to images and audio, as well as video indicate multimodal tasks that cannot be completed with text alone. URLs and links indicate tasks that require accessing external resources, which the model cannot do at inference time. References to specific model names or previous conversations indicate tasks that assume context not present in the instruction-input-output structure. Filtering all these cases ensures that every instance in the final dataset represents a self-contained, completable task.