Instruction Following › Limitations and Practical Considerations

Instruction tuning represents a significant advance in making language models practically useful, but it comes with important limitations that you must understand to avoid overestimating what it achieves and to diagnose failures when they occur.

The most basic limitation is that instruction tuning cannot teach capabilities the model doesn't already possess. If a base model lacks knowledge about a particular domain or cannot perform certain reasoning, no amount of instruction tuning will create that capability from nothing. Instruction tuning is more about eliciting and formatting existing capabilities than creating new ones. This means the quality of the pre-trained base model fundamentally limits what instruction tuning can achieve. When an instruction-tuned model fails on a task, you should ask whether the base model possesses the underlying capability, because if it doesn't, more instruction data will not help.

Data quality presents persistent challenges. Creating high-quality instruction data requires significant human effort, and scaling this effort is expensive. Crowdsourced data may contain errors, inconsistencies, or biases. Model-generated instruction data (which we'll explore in the chapter on Self-Instruct) can propagate and amplify the generating model's mistakes. The old principle of "garbage in, garbage out" applies with full force: models trained on low-quality instruction data learn low-quality behaviors. Perhaps more insidiously, models trained on subtly biased instruction data learn subtly biased behaviors that may be hard to detect until the model is deployed and causing harm at scale.

Instruction tuning also introduces the risk of capability degradation. As discussed in Part XXXIV on catastrophic forgetting, fine-tuning on instruction data can cause the model to forget some pre-training knowledge. This is particularly concerning when instruction datasets don't cover the full breadth of capabilities the base model possessed. A model fine-tuned heavily on English instruction data may perform worse on other languages. A model fine-tuned primarily on conversational tasks may regress on specialized technical domains. Careful dataset design and training procedures can mitigate but not eliminate this risk.

The generalization boundaries of instruction tuning remain unclear and are an active area of research. While instruction-tuned models often generalize impressively to instruction phrasings they've never seen, they can also fail unexpectedly on seemingly simple variations. A model trained extensively on summarization may still fail when the instruction is phrased as "condense" or "distill" if those specific words didn't appear in training. A model that handles single-turn instructions well may struggle with multi-turn conversations where instructions build on each other. You should not assume that instruction tuning creates reliable, failure-free instruction following across all possible input variations.

There is also a subtler limitation around instruction tuning and alignment. Teaching a model to follow instructions efficiently also means it will follow harmful instructions efficiently, if no safeguards are in place. A model that has learned to execute requests reliably will execute malicious requests reliably unless separately trained to refuse them. This is why instruction tuning is typically combined with safety training and RLHF alongside other alignment techniques. The subsequent work on preference learning and RLHF, which we'll cover in Part XXXVII, addresses this limitation by teaching models to follow instructions while also avoiding harmful outputs. Instruction following and safety are distinct properties that require distinct training interventions.

Finally, the distribution mismatch between instruction training data and deployment is a persistent practical challenge. No matter how diverse your training data, real users will find phrasings and tasks in combinations you didn't anticipate. The instruction-tuned model will have varying reliability across this long tail of unusual requests, and predicting where it will fail is difficult. Regular evaluation on held-out instruction sets, red-teaming to find failure modes, and ongoing data collection from deployment are all necessary to maintain and improve model reliability over time.