Instruction Following

Imagine you've just finished pre-training a large language model on trillions of tokens from the internet. The model can complete sentences, generate coherent text, and even demonstrate surprising capabilities like answering questions when prompted correctly. But when you ask it to "Summarize this article in three bullet points," it continues writing more of the article instead. When you request "Translate the following sentence to French," it sometimes translates and sometimes just keeps generating English. The model has learned language, but it hasn't learned to follow instructions.

This gap between language modeling capability and practical utility motivated one of the most important developments in modern LLMs: instruction tuning. As we explored in Part XXVIII with GPT-3's in-context learning, pre-trained models can perform tasks when given the right examples in their context. But this requires careful prompt engineering, wastes context space on demonstrations, and fails unpredictably on novel phrasings. Instruction tuning offers a different approach: teaching models to understand and execute your requests directly, without requiring examples at inference time.

Think of the gap between a base language model and an instruction-following assistant like the difference between a brilliant but socially oblivious expert and a skilled consultant. The expert possesses all the knowledge and reasoning ability you need, but they don't recognize when you're asking them for help versus when you're just making conversation. The consultant has learned a social contract: when someone presents a problem, they engage with it directly, provide a concrete recommendation, and stop when they've addressed what was asked. Instruction tuning teaches language models this same social contract, transforming their raw capability into practical utility.

Understanding why this transformation is necessary, how the training process achieves it, and what factors determine its success requires examining the basic nature of language model pre-training and its relationship to the tasks you want models to perform. The mismatch isn't a bug in the pre-training procedure; it's an inevitable consequence of optimizing for one objective while wanting behavior aligned with a different one. Recognizing this helps you understand what instruction tuning teaches and why certain design choices in instruction data matter so much.

This chapter introduces the motivation behind instruction tuning, explores how to design effective instruction formats, examines why diversity in training data matters for generalization, and discusses what distinguishes high-quality instruction data from noise. Understanding these foundations prepares you for the subsequent chapters on data creation, training procedures, and evaluation methods. You'll also encounter the key concepts that recur throughout the field: the pre-training gap, format consistency, diversity as a proxy for generalization, and the tension between data quality and scale.

Pre-trained language models like those we studied in Parts XVI through XX excel at one specific task: predicting the next token given a context. During causal language modeling, the model learns to assign high probability to continuations that match the statistical patterns in its training data. This training paradigm produces models with remarkable capabilities, but it creates a basic mismatch with how humans want to use these systems. The core issue is that predicting likely continuations and following your instructions are fundamentally different objectives, even though they both involve generating text.

The pre-training objective is both powerful and narrowly defined. When a model trains on web text and books, along with code, it learns to model the distribution of human-written content. It learns grammar, facts, reasoning patterns, and the conventions of countless writing styles. But none of this training explicitly rewards instruction compliance. The model has no concept of "requests" versus "documents" or "assistant responses" versus "document continuations." From the model's perspective during training, your question is simply one more piece of text to be continued in the most statistically plausible way.

Consider what happens when you prompt a base GPT-style model with "Write a poem about autumn leaves." The model doesn't interpret this as a request requiring action. Instead, it treats the text as a prefix to be continued in the most likely way given its training distribution. The model asks itself, in effect, what text would most plausibly follow these words based on everything it learned during pre-training. Depending on what similar text appeared in pre-training, it might:

• Complete the sentence: "Write a poem about autumn leaves falling gently..."
• Generate a meta-discussion: "Write a poem about autumn leaves. This is a common creative writing prompt..."
• Produce the poem directly if similar instructional content appeared frequently in training

This unpredictability stems from the fact that the pre-training objective optimizes for continuation probability, not instruction compliance. The model has no explicit training signal that says "when text looks like a request, generate a response that fulfills that request." From the model's perspective, there is no meaningful distinction between your instruction and any other piece of text. Everything is simply context to be continued. This basic design characteristic explains why base models, despite their impressive language understanding, often fail to behave as useful assistants.

Beyond the objective mismatch, base models don't understand conversational conventions. When you interact with an assistant, there's an implicit structure: you provide a request, and the assistant responds. This turn-taking pattern requires the model to know when to stop generating (it has produced a complete response) and what role it should adopt (helpful assistant rather than document author). These conventions, which humans learn through years of social interaction, must be explicitly taught to language models.

Base models trained on web text have seen conversations, but they've also seen novels, code, advertisements, forum threads, and countless other formats. Without additional training, the model cannot reliably distinguish "this is an instruction I should execute" from "this is text I should continue in its style." A base model might generate a response to your question, or it might generate more questions, or it might continue as if writing a FAQ document, or it might shift into a completely different register. The model simply doesn't have a consistent understanding of its role in the interaction.

The format problem goes beyond mere stylistic inconsistency. When a base model encounters the beginning of an exchange that looks like a user asking a question, it has been trained on text where such exchanges appear in many different contexts: interview transcripts where questions are followed by more questions, Socratic dialogues where answers prompt further inquiry, news articles that quote questions and long analytical responses, and actual chat logs where responses vary wildly in length and style. The model must somehow choose among all these possibilities with no explicit mechanism for doing so. Instruction tuning resolves this ambiguity by establishing a clear, consistent convention: instruction-like input maps to response-like output, full stop.

The pre-training gap can seem surprising because a model trained on so much text still needs explicit instruction data to learn this seemingly simple behavior. The answer lies in what next-token prediction learns and what it cannot learn on its own.

Next-token prediction learns correlational patterns: what tokens tend to follow other tokens. It captures the texture of language, the flow of arguments, the structure of documents. What it cannot learn from text alone is intentionality. When a text contains the phrase "Write me a poem about autumn," the model learns that these words tend to appear before poems. But it doesn't learn that this phrase represents a request that should be honored, as opposed to a sentence that should be continued. The distinction requires understanding the communicative intent behind language, not just its statistical regularities.

Humans understand communicative intent from a rich context of social interaction, shared conventions, and pragmatic reasoning that goes far beyond the words themselves. Language models, trained purely on the textual record of human communication, learn a great deal about the content of that communication but relatively little about its pragmatics. Instruction tuning provides the pragmatic training signal that pure text prediction cannot supply: here is what "following an instruction" looks like when done correctly, and here is how it differs from simply continuing text.

The key insight behind instruction tuning is deceptively simple: if you want models to follow instructions, train them on examples of instructions being followed. By fine-tuning a pre-trained model on a dataset of (instruction, response) pairs, you explicitly teach it the behavior you want. Rather than hoping the model infers the correct behavior from ambiguous pre-training data, you directly demonstrate what good instruction-following looks like.

This approach was pioneered by several research efforts in 2021-2022, including Google's FLAN (Fine-tuned LAnguage Net), OpenAI's InstructGPT, and various academic projects. Despite using relatively small amounts of instruction data compared to pre-training corpora (thousands to millions of examples versus trillions of tokens), instruction tuning produces dramatic improvements in model usability. Models go from being powerful but unpredictable text generators to being cooperative assistants that understand and respond to your needs.

The effectiveness of instruction tuning reveals something important about large pre-trained models: they already possess the underlying capabilities needed to follow instructions. Pre-training gives models broad knowledge about language and facts, including reasoning patterns. Instruction tuning doesn't teach new capabilities so much as it teaches models to activate existing capabilities in response to your requests. Think of it like this: a pre-trained model is like a highly skilled professional who doesn't know they're supposed to be helping you. Instruction tuning teaches them to recognize when they're being asked to do something and to respond accordingly.

If instruction tuning is fundamentally about activating pre-existing capabilities rather than creating new ones, then the quality of the base model places a hard ceiling on what instruction tuning can achieve. A base model with weak reasoning cannot become a strong reasoner through instruction tuning alone. A base model with limited factual knowledge cannot become more knowledgeable. Instruction tuning makes the capabilities that exist more reliably accessible. This distinction affects evaluation: when instruction-tuned models fail, ask whether the failure reflects a missing pre-training capability or instruction-following behavior, because these require different remedies.

Given the scale of pre-training (hundreds of billions to trillions of tokens), it might seem surprising that instruction tuning works with datasets that are orders of magnitude smaller. Several factors explain this efficiency, and understanding them illuminates why instruction tuning is so effective.

Capability already exists. The model has already learned to summarize, translate, answer questions, and perform other tasks during pre-training. It encountered examples of summaries and translations, along with question-answer pairs throughout the web data it was trained on. What it lacks is the understanding that when you present an instruction, it should deploy these existing capabilities. Instruction tuning provides this missing link, teaching the model to recognize and respond to requests.

Format is learnable quickly. The instruction-following format (receive request, generate response, stop) is a relatively simple pattern compared to the full complexity of language. Unlike learning grammar, world knowledge, or reasoning, which requires exposure to large amounts of text, learning to recognize and respond to instructions is a much more constrained problem. Models can learn this convention quickly because it's fundamentally a formatting task rather than a capability-building task.

Transfer across instructions multiplies value. Training on "summarize this article" helps the model understand "condense this text" and "give me the main points" even without explicit examples. The model generalizes from the specific examples it sees to the broader concept of summarization requests. This transfer effect multiplies the value of each training example, as one example can inform the model's behavior across many related phrasings and variations.

Alignment with generation makes fine-tuning efficient. Following instructions aligns with the model's core capability: generating text. The model simply learns to generate the kind of text that represents a helpful response to the given instruction. Unlike training objectives that require the model to perform fundamentally new operations, instruction tuning asks the model to do what it already does (generate text) but with a different intent (fulfill your requests rather than continue documents).

An important nuance in the "small datasets work" story is that the relationship between instruction dataset size and model quality is not flat: more diverse, higher-quality instruction data generally produces better models. The FLAN paper showed that scaling from 62 to 1,836 task types dramatically improved zero-shot generalization. Later work like the LIMA paper ("Less Is More for Alignment") argued that 1,000 carefully curated, high-quality examples could produce surprisingly capable models, but this only held when those examples were exceptionally well-chosen. The key insight from this line of research is that for instruction tuning, quality and diversity matter more than raw quantity. A dataset of 10,000 high-diversity, accurate instruction examples will typically outperform a dataset of 100,000 noisy, repetitive ones.

This scaling story also has an important conditional: the benefits of more instruction data plateau as instruction following improves. Once a model has firmly learned the instruction-following convention and seen sufficient diversity across task types, additional instruction examples yield diminishing returns. At that point, the bottleneck shifts back to pre-training quality and the breadth of capabilities the base model learned from text. Understanding this saturation effect helps you allocate your data creation budget wisely, putting effort on diversity and quality rather than indefinitely scaling up volume.

Creating effective instruction-tuning data requires careful attention to format. The structure of training examples teaches models what inputs to expect and what outputs to produce. A well-designed format makes the distinction between instruction, input, and response crystal clear, helping the model learn the appropriate boundaries and behaviors. Several components define an instruction format, and each plays a useful role in the training signal.

The format affects model behavior. Because language models learn by pattern-matching, the consistent structural markers in your instruction format become reliable behavioral cues. When a model is trained on thousands of examples that always place the task description after "### Instruction:" and the expected response after "### Response:", these tokens activate instruction-following behavior. The specific delimiter tokens matter: a model tuned on one format may perform poorly with a different format because the activating cues are absent. This is why production systems preserve their instruction format at inference time and why deviating from the training format often degrades performance.

The instruction component tells the model what to do. This is the core directive that specifies the task the model should perform. Effective instructions are clear and specific enough to act on:

• Clear: "Translate the following English sentence to Spanish" leaves no ambiguity about the task. The model knows exactly what language pair to work with and what direction the translation should go.
• Specific: "Summarize in exactly three sentences" provides concrete constraints that guide the model's output. Without such specificity, the model must guess at the appropriate length and format.
• Actionable: The instruction describes something the model can generate as text output. Instructions like "feel happy about this text" aren't actionable because they don't translate into specific textual output.

Instructions can range from single-word commands ("Translate:") to detailed multi-sentence specifications explaining exactly what output is expected. The level of detail often depends on the task complexity and the desired output format. Simple tasks may need only brief instructions, while complex tasks benefit from more elaborate specifications that reduce ambiguity and set clear expectations.

The quality of the instruction component has an asymmetric effect on what models learn. A vague instruction paired with a high-quality response still teaches the model to produce good responses, but it doesn't teach the model to consistently recognize what type of response is being requested. Conversely, a precise, clear instruction paired with a mediocre response teaches the model a pattern for recognizing requests but produces suboptimal output quality. This asymmetry suggests that both instruction quality and response quality matter, but they matter for different aspects of the model's behavior.

Many instructions operate on provided content. The input component contains the material the model should process, serving as the raw data that the instruction acts upon:

Instruction: Summarize the following article.
Input: [article text here]

Some instructions don't require separate input (e.g., "Write a haiku about spring") while others are meaningless without it (e.g., "Translate the following sentence"). The format must clearly distinguish the instruction from the input to prevent confusion. Without clear separation, the model might incorporate parts of the instruction into its understanding of the input, or vice versa, leading to incorrect responses.

The presence or absence of an explicit input field also carries meaning. When your training data uses a consistent format where the input field is present for some task types and absent for others, the model learns that "tasks with input fields require processing external content" versus "tasks without input fields require generation from scratch." This distinction helps the model correctly interpret instruction-only prompts at inference time without getting confused about where to find content to process.

The output (or response) component contains the desired model behavior. During training, this is the target text the model learns to generate. The output should directly address the instruction and, where applicable, properly process the input. This component is the ground truth that the model optimizes toward during training.

Quality outputs exhibit several properties:

• Responsiveness: The output addresses exactly what the instruction asks. A summarization instruction should produce a summary, not additional commentary or tangential information.
• Completeness: All parts of the instruction are fulfilled. If the instruction asks for three points, the output should contain three points, not two or four.
• Conciseness: No unnecessary elaboration beyond what's requested. Verbose responses that pad the output with irrelevant information teach the model to be similarly unfocused.
• Correctness: Factually accurate and logically sound. Training on incorrect outputs teaches the model to produce errors confidently.

The output component is where training data quality has the most direct impact on deployed model behavior. If you train on incorrect outputs, the model learns to be confidently wrong. If you train on verbose, padded outputs, the model learns to pad. If you train on outputs that technically address the instruction but miss the spirit of what was asked, the model learns similarly evasive responses. The model is, in a very literal sense, learning to imitate the outputs in your training set. This makes output quality the most consequential investment in instruction data creation.

Instruction-tuning datasets typically use consistent templates to structure examples. A common template might look like:

### Instruction:
{instruction_text}

### Input:
{input_text}

### Response:
{output_text}

The specific delimiters and formatting vary across datasets, but consistency within a dataset helps models learn the pattern reliably. When the model sees consistent markers like "### Instruction:" and "### Response:", it learns to associate these patterns with the instruction-following behavior. This consistency reduces ambiguity and accelerates learning. We'll explore format templates in depth in the upcoming chapter on instruction format.

Different template designs carry different tradeoffs. The minimal template above (used in Alpaca and many derivative datasets) is compact and easy to parse, but provides no system-level context about the model's role. Templates that include a preamble like "Below is an instruction that describes a task. Write a response that appropriately completes the request" add context that may help the model maintain its role as an assistant, at the cost of extra tokens in every training example. Chat-oriented templates that distinguish between user and assistant turns (as in OpenAI's ChatML format) are better suited for multi-turn conversations but require the training data to include appropriate turn boundaries. The choice of template should match how you intend to use the model in deployment.

The diversity of instructions in training data critically affects how well the tuned model generalizes. A model trained only on translation examples won't learn to summarize, because it has never seen the pattern of summarization requests and responses. A model trained only on formal phrasings won't understand casual requests, because it has learned to expect a specific register of language. Diversity operates along multiple dimensions, and each dimension contributes to the model's flexibility.

The main point about diversity is that it is a proxy for generalization. When we say we want a model to "generalize well to new instructions," we mean we want it to handle instruction types and phrasings across contexts it hasn't seen during training. Diversity in training data is how we prepare the model for this challenge: the more varied the examples the model sees during training, the more likely it is to extract underlying principles rather than memorizing surface patterns. A model that has seen only one phrasing of a request has learned a trigger; a model that has seen 50 different phrasings has learned the underlying intent. Intent-level understanding is far more reliable at deployment time.

Task diversity means covering many different types of instructions. Rather than specializing in one particular capability, an instruction-following model should handle the range of tasks you might request. Broad instruction datasets include examples across categories like:

• Text generation: Writing stories, poems, emails, code, essays. These tasks require the model to create content from scratch based on specifications.
• Text transformation: Summarization, paraphrasing, style transfer, translation. These tasks require the model to take existing text and modify it in specific ways.
• Information extraction: Named entity recognition, relation extraction, key point identification. These tasks require the model to identify and isolate specific information from provided text.
• Question answering: Factual questions, reasoning questions, reading comprehension. These tasks require the model to provide information in response to queries.
• Analysis: Sentiment classification, topic identification, text comparison. These tasks require the model to evaluate and categorize text according to various criteria.
• Conversation: Dialogue responses, follow-up handling, context maintenance. These tasks require the model to participate in multi-turn exchanges while tracking context.

Training across diverse tasks teaches models the general skill of instruction following rather than specific task patterns. A model that has learned to follow summarization and classification instructions as well as generation instructions develops a meta-understanding of what it means to receive and execute an instruction. It learns the abstract pattern: "receive a directive, understand what's being asked, generate appropriate output." This meta-learning is more valuable than mastering any single task, and it separates an instruction-following model from a narrow task-specific fine-tune.

There is an important caveat about task diversity: adding more task types doesn't always help equally. Research consistently finds that certain task types are more important for establishing general instruction-following behavior than others. Tasks that require free-form, open-ended responses (writing, explanation, reasoning) tend to be especially valuable because they teach the model to produce varied output in response to varied input, rather than learning the relatively constrained mapping from input to categorical label. Tasks that require following complex multi-part specifications are similarly valuable because they teach constraint satisfaction. A dataset heavily weighted toward simple classification tasks may produce a model that's good at classifying but poor at following complex generation instructions.

Even within a single task, instructions can be phrased countless ways. Human language is remarkably flexible, and you will express the same intent using vastly different words and structures. Consider these equivalent summarization requests:

• "Summarize this text"
• "What are the main points?"
• "Give me a brief overview"
• "TL;DR"
• "Can you condense this?"
• "I need a quick summary"
• "Break this down for me"

If training data only contains one phrasing, the model may fail on others because it has learned to pattern-match on specific words rather than understanding the underlying intent. Phrasing diversity ensures the model learns the underlying intent rather than pattern-matching on specific words. The model should recognize that all these phrasings point to the same underlying request for summarization, even though they use completely different vocabulary.

Phrasing diversity also matters because users have different communication styles, and a deployed model will encounter all of them. A software engineer might write terse, precise instructions. A student might use informal language with unclear structure. A business analyst might write elaborate, multi-clause specifications. An effective instruction-following model must handle all these styles without requiring users to learn the "right" way to phrase requests. Training data therefore needs examples spanning the full range of communication styles.

The challenge is that generating phrasing diversity manually is tedious and error-prone. Humans asked to "rephrase this instruction five different ways" tend to produce variations that are more similar than they realize, clustering around a few familiar patterns. This is one area where language model assistance for data generation proves particularly valuable: models are surprisingly good at generating diverse phrasings of the same underlying intent, and these model-generated paraphrases can be used to augment manually created instruction data. We'll explore model-assisted data generation in detail in the upcoming chapter on Self-Instruct and synthetic data.

Instructions vary in complexity from simple single-step requests to detailed multi-part specifications. A reliable instruction-following model must handle this entire spectrum, from straightforward commands to fine-grained directives with several requirements. Training data should span this spectrum:

Simple instructions:

• "Translate 'hello' to German"
• "What is 2 + 2?"

Moderate instructions:

• "Summarize this article in 3 bullet points, focusing on the economic implications"
• "Rewrite this paragraph in a more formal tone while preserving the key information"

Complex instructions:

• "Compare and contrast these two research papers, identifying their methodological differences and evaluating the strength of their conclusions"
• "Write a response to this customer complaint that acknowledges their frustration, explains our policy, offers a reasonable solution, and maintains a professional tone"

Exposure to varying difficulty levels helps models handle the full range of requests they'll encounter in deployment. If training data contains only simple instructions, the model may struggle to decompose and address complex multi-part requests. Conversely, training only on complex instructions may not help the model learn the basic patterns of instruction following that simpler examples demonstrate clearly.

Difficulty diversity also connects to curriculum learning. Some researchers have explored training models on instruction data ordered from simple to complex, allowing the model to establish basic instruction-following patterns before encountering examples that require chaining several capabilities. While the evidence for strict curriculum ordering is mixed, complex instructions do build on simple ones. Simple examples handle common deployment cases and provide the foundation for complex instruction following.

Instructions span many knowledge domains: science, history, technology, arts and law. Medicine and many other fields are also represented. Domain diversity ensures the model doesn't overfit to particular subject areas and can handle requests across the full breadth of human knowledge. A model trained predominantly on technology-related instructions might struggle with history questions, not because it lacks historical knowledge (that comes from pre-training) but because it hasn't learned how historical inquiries are typically phrased.

This doesn't mean the model needs domain expertise (that comes from pre-training knowledge), but it needs to understand how instructions are phrased in different domains. A medical question might use terminology differently than a legal question, even if both are asking for clarification. Scientific instructions might favor precise, technical language, while creative writing instructions might use more evocative, open-ended phrasing. Domain diversity teaches the model to handle these variations in communication style and to recognize that the same underlying instruction type (explain, summarize, evaluate) looks different across domains.

Domain diversity also interacts with safety and alignment in important ways. Instructions about sensitive domains such as medicine and law, including finance require special handling: the model should be helpful while acknowledging its limitations and the importance of professional expertise. Including domain-diverse instruction data with appropriate responses teaches models to handle these domain-specific conventions. A model that has only seen technology instructions may produce confidently assertive responses when confronted with medical questions, having never learned the appropriate epistemic humility that medical domains require.

Not all instruction examples are equally valuable. The quality of training data directly impacts model behavior, often in ways that may not be immediately apparent but compound over time. High-quality instruction data exhibits several characteristics that distinguish it from noise or low-value examples. The relationship between data quality and model quality is one of the most consistently validated findings in the instruction tuning literature: quality matters more than quantity past a certain threshold, and low-quality data can actively harm a model's performance by teaching it patterns you don't want.

Think of the instruction dataset as a curriculum. A student learns from their coursework, and if that coursework contains errors, teaches poor habits, or covers an unbalanced selection of topics, the student graduates with corresponding deficiencies. The language model is no different. Every training example is an implicit lesson about what good instruction following looks like. Low-quality examples teach poor lessons, and the model has no mechanism to distinguish good lessons from bad ones unless quality is enforced during data creation.

The response must correctly fulfill the instruction. An instruction asking for the capital of France paired with a response saying "London" teaches the wrong behavior. The model learns to produce confident but incorrect answers because that's what it sees in training. Quality control processes must verify that responses accurately address their instructions, especially for factual or deterministic tasks where correctness can be objectively assessed.

For subjective tasks like creative writing, accuracy means appropriateness: does the response reasonably satisfy what a human would expect from the instruction? A poem about autumn should be about autumn, not spring. A formal letter should sound formal, not casual. These judgments require human evaluation to ensure the training data teaches appropriate behavior. The difficulty is that accuracy evaluation doesn't scale easily: verifying that thousands of instruction-response pairs are factually correct requires significant human effort or automated verification tools with their own error rates.

The problem of accuracy degradation compounds over time in an interesting way. If a model is used to generate instruction data (as in Self-Instruct and similar approaches), and that data is then used to train a new model, any inaccuracies in the generated data propagate forward. The trained model may be confidently wrong in the same ways its training data was wrong, and if that model is then used to generate more data, those errors compound further. This cascading inaccuracy problem is a core challenge in synthetic data generation, and it motivates the continued importance of human verification even as automation becomes more prevalent.

Partial or truncated responses teach models to stop prematurely. If an instruction asks for "five examples" but the response only provides three, the model learns incomplete behavior. It learns that three examples constitute an acceptable response to a request for five. Over many training examples, such patterns accumulate, teaching the model that incomplete responses are acceptable. Quality instruction data includes complete responses that fully address all aspects of the instruction.

Completeness has a symmetric failure mode: responses that go substantially beyond what was asked teach the model to be verbose and unfocused. An instruction asking for a "one-sentence summary" paired with a three-paragraph response teaches the model that constraint violation is acceptable. The model learns to pad its responses, hedging, adding caveats, and elaborating beyond what the instruction specified. Both truncation and over-generation represent completeness failures, and both require attention during data quality control.

Ambiguous or confusing instructions produce training noise. If an instruction could reasonably be interpreted multiple ways, the response may align with one interpretation while you have another in mind. When the model encounters similar ambiguous instructions during deployment, it may produce responses that don't match your expectations because it learned a different interpretation during training. High-quality instructions minimize ambiguity.

Consider the instruction "Make this better." Better in what way? More formal? More concise? More engaging? Such vague instructions should either be clarified or paired with responses that interpret the intent reasonably and explicitly state their interpretation. A high-quality response might begin, "I'll improve this text by which makes it more concise and formal:" before giving the revised version, thereby showing how to handle ambiguity constructively.

The tricky aspect of instruction clarity is that ambiguity isn't always obvious to the person writing the instruction. Instructions that seem clear to the author may be ambiguous to the model, which has no context about the author's intent beyond the words on the screen. Effective quality control includes reviewing instructions from the perspective of someone who doesn't share the author's background knowledge and asking whether the instruction uniquely specifies the desired output or admits multiple reasonable interpretations.

Instructions should reflect how you make requests. Overly formal or artificial instructions create a distribution mismatch between training and deployment. If all training instructions follow a rigid template like "Please perform the following action: [task description]" but you write "hey can you help me with something," the model may struggle with natural language inputs. It has learned to expect a particular format that you don't employ.

Effective instruction datasets include a mix of formal and informal phrasings, complete sentences and fragments, explicit requests and implied needs. This variety prepares the model to handle the messy, inconsistent ways humans communicate. The goal is that the model should feel like it's interacting with a person who has a wide range of communication styles, not a system that expects perfectly formatted API calls.

Naturalness also matters for the response side. Overly rigid, template-like responses may be technically correct but feel robotic and unhelpful. If training data responses all follow the same structure (restate the question, provide the answer, summarize the key point), the model learns this rigid pattern and applies it even when a more direct, natural response would be better. Effective instruction data mixes response styles to teach the model to adapt its output to the context rather than following a fixed template.

Training data shapes model behavior, including potentially harmful behaviors. If the training data includes examples of generating harmful content, the model learns to produce such content when asked. Quality instruction data excludes examples that would teach models to:

• Generate harmful or dangerous content
• Reveal private information
• Exhibit biases or discriminatory patterns
• Bypass safety considerations

This concern becomes particularly important in the alignment work we'll explore in Part XXXVII, where human preferences guide models toward helpful, harmless, and honest behavior. Safety filtering during instruction data creation is necessary but insufficient: it prevents the model from explicitly learning harmful behaviors from training examples, but it does not guarantee refusal of harmful instructions in deployment. Subsequent chapters cover more reliable safety measures, including RLHF, constitutional AI, and refusal training.

Let's create a few instruction examples to illustrate format and quality considerations. We'll develop examples for a summarization task, showing how different format choices affect the training signal. By examining these examples closely, you can see how small variations in instruction phrasing lead to different expected outputs.

Step 1: Basic format with no constraints. The simplest instruction format provides the task and input with minimal structure:

Instruction: Summarize the following paragraph.

Input: The James Webb Space Telescope, launched in December 2021, represents
the largest and most powerful space telescope ever built. Its primary mirror,
composed of 18 hexagonal gold-plated beryllium segments, spans 6.5 meters in
diameter. Operating primarily in the infrared spectrum, JWST can observe
objects too old, distant, or faint for its predecessor, the Hubble Space
Telescope. The telescope orbits the Sun at the second Lagrange point,
approximately 1.5 million kilometers from Earth.

Response: The James Webb Space Telescope is the largest space telescope ever
built, featuring a 6.5-meter segmented mirror. Launched in 2021, it observes
in infrared from a position 1.5 million kilometers from Earth, enabling
detection of objects beyond Hubble's capabilities.

This example teaches the model the basic summarization pattern. The response is shorter than the input, preserves the key facts, and doesn't add any new information. Training on many such examples helps the model understand what "summarize" means in terms of expected output structure.

Step 2: Constrained format with explicit requirements. Adding explicit constraints tests a different aspect of instruction following:

Instruction: Summarize the following paragraph in exactly two sentences.
Focus on the telescope's key capabilities.

Input: [same paragraph]

Response: The James Webb Space Telescope features a 6.5-meter gold-plated
mirror that enables infrared observations of objects too distant or faint
for previous telescopes. Operating from the L2 Lagrange point, it represents
humanity's most powerful tool for deep space observation.

Notice how the response now strictly adheres to the two-sentence constraint and focuses specifically on capabilities rather than construction details. This example teaches the model to honor explicit constraints, a more sophisticated skill than basic summarization.

Step 3: Informal phrasing with stylistically matched response.

Instruction: What's the gist of this?

Input: [same paragraph]

Response: It's about the James Webb Space Telescope, which is basically the
biggest and most powerful space telescope we've ever made. It launched in
late 2021 and can see much farther into space than Hubble could, using
infrared detection from its position way out past Earth's orbit.

Notice how the response style matches the instruction style. Formal instructions receive formal responses; casual questions get conversational answers. This stylistic alignment teaches models to adapt their output to your preferences. The model learns the content to provide and the register and tone to use for your request. Training all three variants teaches the model that "summarize," "focus on key capabilities," and "what's the gist of this" are related requests that share an underlying intent but require different output styles.

Step 4: Evaluating the examples against quality criteria.

Looking back at these three examples through the lens of the quality criteria we discussed:

Accuracy: All three responses correctly represent the source material without introducing false claims. The key facts (infrared detection, 6.5-meter mirror, L2 orbit, Hubble comparison) appear accurately in appropriate forms across all responses.

Completeness: Each response fully addresses the instruction. The constrained example provides exactly two sentences, not one or three. The informal example covers the key aspects without being truncated.

Clarity: Each instruction clearly specifies what's needed. The second instruction is the clearest because it explicitly states both the format constraint (two sentences) and the content focus (key capabilities).

Naturalness: The informal example feels like a natural human question. The constrained example sounds like a precise specification that a careful user might write. Both represent how people communicate.

Let's implement a basic instruction data processor that validates and formats instruction examples. This prepares raw data for instruction tuning.

The key parameters for the instruction processing pipeline are:

• template: The formatting style for instructions (e.g., "default", "alpaca"). Different templates provide different context cues to the model. Consistency between training and deployment is needed: a model trained on the Alpaca template should receive Alpaca-formatted prompts at inference time to activate the instruction-following behavior it learned.
• max_length: The maximum sequence length allowed for training examples. Filtering long sequences ensures data fits within the model's context window. Setting this too low discards valuable complex examples; setting it too high may cause memory issues during training.
• min_instruction_length: The minimum character count for instructions. This threshold helps filter out noise or trivial inputs. Very short instructions like "Hi" or "Thanks" don't teach meaningful instruction-following patterns.
• min_output_length: The minimum character count for responses to ensure meaningful training signals. Filtering near-empty outputs prevents the model from learning to produce terse, unhelpful responses to legitimate requests.

Instruction tuning represents a significant advance in making language models practically useful, but it comes with important limitations that you must understand to avoid overestimating what it achieves and to diagnose failures when they occur.

The most basic limitation is that instruction tuning cannot teach capabilities the model doesn't already possess. If a base model lacks knowledge about a particular domain or cannot perform certain reasoning, no amount of instruction tuning will create that capability from nothing. Instruction tuning is more about eliciting and formatting existing capabilities than creating new ones. This means the quality of the pre-trained base model fundamentally limits what instruction tuning can achieve. When an instruction-tuned model fails on a task, you should ask whether the base model possesses the underlying capability, because if it doesn't, more instruction data will not help.

Data quality presents persistent challenges. Creating high-quality instruction data requires significant human effort, and scaling this effort is expensive. Crowdsourced data may contain errors, inconsistencies, or biases. Model-generated instruction data (which we'll explore in the chapter on Self-Instruct) can propagate and amplify the generating model's mistakes. The old principle of "garbage in, garbage out" applies with full force: models trained on low-quality instruction data learn low-quality behaviors. Perhaps more insidiously, models trained on subtly biased instruction data learn subtly biased behaviors that may be hard to detect until the model is deployed and causing harm at scale.

Instruction tuning also introduces the risk of capability degradation. As discussed in Part XXXIV on catastrophic forgetting, fine-tuning on instruction data can cause the model to forget some pre-training knowledge. This is particularly concerning when instruction datasets don't cover the full breadth of capabilities the base model possessed. A model fine-tuned heavily on English instruction data may perform worse on other languages. A model fine-tuned primarily on conversational tasks may regress on specialized technical domains. Careful dataset design and training procedures can mitigate but not eliminate this risk.

The generalization boundaries of instruction tuning remain unclear and are an active area of research. While instruction-tuned models often generalize impressively to instruction phrasings they've never seen, they can also fail unexpectedly on seemingly simple variations. A model trained extensively on summarization may still fail when the instruction is phrased as "condense" or "distill" if those specific words didn't appear in training. A model that handles single-turn instructions well may struggle with multi-turn conversations where instructions build on each other. You should not assume that instruction tuning creates reliable, failure-free instruction following across all possible input variations.

There is also a subtler limitation around instruction tuning and alignment. Teaching a model to follow instructions efficiently also means it will follow harmful instructions efficiently, if no safeguards are in place. A model that has learned to execute requests reliably will execute malicious requests reliably unless separately trained to refuse them. This is why instruction tuning is typically combined with safety training and RLHF alongside other alignment techniques. The subsequent work on preference learning and RLHF, which we'll cover in Part XXXVII, addresses this limitation by teaching models to follow instructions while also avoiding harmful outputs. Instruction following and safety are distinct properties that require distinct training interventions.

Finally, the distribution mismatch between instruction training data and deployment is a persistent practical challenge. No matter how diverse your training data, real users will find phrasings and tasks in combinations you didn't anticipate. The instruction-tuned model will have varying reliability across this long tail of unusual requests, and predicting where it will fail is difficult. Regular evaluation on held-out instruction sets, red-teaming to find failure modes, and ongoing data collection from deployment are all necessary to maintain and improve model reliability over time.

Instruction tuning bridges the gap between language models that can generate text and assistants that respond helpfully to your requests. Pre-trained models optimize for next-token prediction, not instruction compliance, creating a basic mismatch with your expectations. Instruction tuning addresses this by fine-tuning on datasets of instruction-response pairs, teaching models to recognize and execute requests.

Effective instruction data requires careful attention to format, with clear structure distinguishing instructions, inputs, and expected outputs. The consistency of format markers across training examples creates the cues that activate instruction-following behavior at inference time. Diversity across task types, phrasings, difficulty levels, and domains enables models to generalize beyond their training examples to handle the varied requests you'll send in deployment. Quality considerations including response accuracy, completeness, and naturalness determine whether training produces reliable or erratic behavior, and they matter more than raw data volume past a certain scale.

The efficiency of instruction tuning, which changes behavior with relatively small datasets, indicates that pre-trained models already possess many of the necessary capabilities. Instruction tuning teaches them when and how to deploy these capabilities in response to your needs. These capabilities are often latent in the base model; instruction tuning provides access rather than creating them from scratch.

The next chapter examines how instruction data is created at scale, exploring both human annotation approaches and the increasingly useful role of model-generated synthetic data. Understanding data creation methods will help you evaluate and create instruction datasets for your own applications.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about instruction tuning and instruction-following in large language models.