Instruction Following › Instruction Quality

Not all instruction examples are equally valuable. The quality of training data directly impacts model behavior, often in ways that may not be immediately apparent but compound over time. High-quality instruction data exhibits several characteristics that distinguish it from noise or low-value examples. The relationship between data quality and model quality is one of the most consistently validated findings in the instruction tuning literature: quality matters more than quantity past a certain threshold, and low-quality data can actively harm a model's performance by teaching it patterns you don't want.

Think of the instruction dataset as a curriculum. A student learns from their coursework, and if that coursework contains errors, teaches poor habits, or covers an unbalanced selection of topics, the student graduates with corresponding deficiencies. The language model is no different. Every training example is an implicit lesson about what good instruction following looks like. Low-quality examples teach poor lessons, and the model has no mechanism to distinguish good lessons from bad ones unless quality is enforced during data creation.

The response must correctly fulfill the instruction. An instruction asking for the capital of France paired with a response saying "London" teaches the wrong behavior. The model learns to produce confident but incorrect answers because that's what it sees in training. Quality control processes must verify that responses accurately address their instructions, especially for factual or deterministic tasks where correctness can be objectively assessed.

For subjective tasks like creative writing, accuracy means appropriateness: does the response reasonably satisfy what a human would expect from the instruction? A poem about autumn should be about autumn, not spring. A formal letter should sound formal, not casual. These judgments require human evaluation to ensure the training data teaches appropriate behavior. The difficulty is that accuracy evaluation doesn't scale easily: verifying that thousands of instruction-response pairs are factually correct requires significant human effort or automated verification tools with their own error rates.

The problem of accuracy degradation compounds over time in an interesting way. If a model is used to generate instruction data (as in Self-Instruct and similar approaches), and that data is then used to train a new model, any inaccuracies in the generated data propagate forward. The trained model may be confidently wrong in the same ways its training data was wrong, and if that model is then used to generate more data, those errors compound further. This cascading inaccuracy problem is a core challenge in synthetic data generation, and it motivates the continued importance of human verification even as automation becomes more prevalent.

Partial or truncated responses teach models to stop prematurely. If an instruction asks for "five examples" but the response only provides three, the model learns incomplete behavior. It learns that three examples constitute an acceptable response to a request for five. Over many training examples, such patterns accumulate, teaching the model that incomplete responses are acceptable. Quality instruction data includes complete responses that fully address all aspects of the instruction.

Completeness has a symmetric failure mode: responses that go substantially beyond what was asked teach the model to be verbose and unfocused. An instruction asking for a "one-sentence summary" paired with a three-paragraph response teaches the model that constraint violation is acceptable. The model learns to pad its responses, hedging, adding caveats, and elaborating beyond what the instruction specified. Both truncation and over-generation represent completeness failures, and both require attention during data quality control.

Ambiguous or confusing instructions produce training noise. If an instruction could reasonably be interpreted multiple ways, the response may align with one interpretation while you have another in mind. When the model encounters similar ambiguous instructions during deployment, it may produce responses that don't match your expectations because it learned a different interpretation during training. High-quality instructions minimize ambiguity.

Consider the instruction "Make this better." Better in what way? More formal? More concise? More engaging? Such vague instructions should either be clarified or paired with responses that interpret the intent reasonably and explicitly state their interpretation. A high-quality response might begin, "I'll improve this text by which makes it more concise and formal:" before giving the revised version, thereby showing how to handle ambiguity constructively.

The tricky aspect of instruction clarity is that ambiguity isn't always obvious to the person writing the instruction. Instructions that seem clear to the author may be ambiguous to the model, which has no context about the author's intent beyond the words on the screen. Effective quality control includes reviewing instructions from the perspective of someone who doesn't share the author's background knowledge and asking whether the instruction uniquely specifies the desired output or admits multiple reasonable interpretations.

Instructions should reflect how you make requests. Overly formal or artificial instructions create a distribution mismatch between training and deployment. If all training instructions follow a rigid template like "Please perform the following action: [task description]" but you write "hey can you help me with something," the model may struggle with natural language inputs. It has learned to expect a particular format that you don't employ.

Effective instruction datasets include a mix of formal and informal phrasings, complete sentences and fragments, explicit requests and implied needs. This variety prepares the model to handle the messy, inconsistent ways humans communicate. The goal is that the model should feel like it's interacting with a person who has a wide range of communication styles, not a system that expects perfectly formatted API calls.

Naturalness also matters for the response side. Overly rigid, template-like responses may be technically correct but feel robotic and unhelpful. If training data responses all follow the same structure (restate the question, provide the answer, summarize the key point), the model learns this rigid pattern and applies it even when a more direct, natural response would be better. Effective instruction data mixes response styles to teach the model to adapt its output to the context rather than following a fixed template.

Training data shapes model behavior, including potentially harmful behaviors. If the training data includes examples of generating harmful content, the model learns to produce such content when asked. Quality instruction data excludes examples that would teach models to:

• Generate harmful or dangerous content
• Reveal private information
• Exhibit biases or discriminatory patterns
• Bypass safety considerations

This concern becomes particularly important in the alignment work we'll explore in Part XXXVII, where human preferences guide models toward helpful, harmless, and honest behavior. Safety filtering during instruction data creation is necessary but insufficient: it prevents the model from explicitly learning harmful behaviors from training examples, but it does not guarantee refusal of harmful instructions in deployment. Subsequent chapters cover more reliable safety measures, including RLHF, constitutional AI, and refusal training.