Instruction Tuning Training: Data Mixing & Loss Masking

Pre-training a language model produces a powerful text predictor, but that predictor does not yet understand what you want. Ask a raw pre-trained model to "summarize this article" and it may continue the article instead, or repeat the phrase "summarize this article" in several stylistic variations, because the pre-training objective simply rewarded predicting the next token. Instruction tuning bridges this gap. By fine-tuning on curated examples that pair instructions with ideal responses, we teach the model to interpret requests and act on them, transforming a statistical text engine into a useful assistant.

But instruction tuning is not as simple as collecting examples and running a standard fine-tuning loop. The training process introduces several decisions that determine whether the model learns what you intend or learns something subtly different. Get the data mixing wrong and the model will excel at common tasks while failing on rare-but-important ones. Train without loss masking and the model spends gradient budget recreating instructions rather than learning to respond. Set hyperparameters incorrectly and you risk catastrophic forgetting, where the fine-tuning erases the broad capabilities that made the base model useful in the first place.

Think of instruction tuning as a specialization process applied to a generalist. A brilliant research scientist has large general knowledge. When they join a company and receive onboarding training, the topics covered and the training structure both matter. Too much time on easy procedures they already know dilutes the signal. The training should focus on new behaviors, practiced in proportions that reflect the actual job, without overwriting everything they learned in graduate school. The same logic applies to instruction tuning.

This chapter covers the mechanics of that training process in detail. We explore three interconnected decisions: how to balance task types through data mixing strategies, how to use loss masking to focus the learning signal on responses rather than prompts, and how to choose hyperparameters that guide the model toward instruction-following without destroying its pre-trained capabilities. We also examine multi-task learning dynamics and the practical considerations that determine whether instruction tuning succeeds or falls short of its goals.

Instruction tuning datasets are not collections of a single task type. They aggregate examples from many domains. These may include question answering and summarization alongside code generation or mathematical reasoning. Other examples cover creative writing, translation, or classification. Each task type requires different capabilities. Question answering relies heavily on factual recall and information retrieval. Mathematical reasoning requires multi-step logical inference. Code generation demands precise syntax adherence and algorithmic thinking. Creative writing calls for coherent narrative construction and stylistic flexibility.

The distribution of these task types during training shapes what skills the model develops and how well it generalizes across the variety of requests it will encounter at deployment. Leave the distribution entirely to chance and you will get a model whose capabilities mirror your data collection process rather than your actual goals. A thoughtful data mixing strategy is one of the most impactful levers you control during instruction tuning, and it deserves as much attention as model architecture or learning rate.

Think of data mixing like designing a curriculum for a student. A music student who spends 90% of practice time on scales and only 10% on actual pieces will become technically proficient but may struggle with performance. The curriculum must reflect the actual skills the student needs, in proportions that ensure sufficient practice for each competency. Instruction tuning requires the same deliberate curation.

Training on all available instruction data without adjustment causes task imbalance. Instruction datasets are rarely balanced because some tasks are far easier to collect than others. Question-answering pairs can be scraped at scale from documentation, FAQ pages, or discussion forums. High-quality mathematical reasoning examples require expert annotation, careful verification, and significant human effort. If your dataset contains 120,000 question-answering examples but only 8,000 math examples, the model sees QA tasks 15 times more frequently during training. This imbalance causes the model to perform well on common tasks while struggling with rare ones, even if both are equally important for the application you are building.

The imbalance compounds across training steps. After seeing one million examples during training, a model trained with proportional sampling has received gradient updates from 345,000 QA examples but only 23,000 math examples. The model's parameters are shaped far more by what helps it answer questions than by what helps it solve mathematical problems. The math skill does not simply fail to develop; it may regress if the QA optimization subtly interferes with mathematical reasoning patterns learned during pre-training.

Consider a hypothetical instruction dataset with the following distribution, which shows the kind of imbalance commonly found in real instruction tuning corpora:

In proportional sampling, where examples are drawn according to their natural frequency, the model sees the QA category roughly 15 times more often than math. This imbalance biases the training signal toward the majority task in ways that go beyond simple frequency. The gradient updates from majority tasks dominate the loss landscape, pushing the model's parameters toward a region that is good for QA but potentially suboptimal for reasoning-heavy tasks.

Three main approaches address task imbalance, each with different trade-offs that make them suitable for different situations and goals. Understanding when each approach is appropriate requires thinking carefully about your target deployment scenarios and the nature of the tasks you are balancing.

Proportional sampling uses the natural data distribution without modification. Examples appear according to their frequency in the dataset, meaning that if 34% of your data is question-answering, then 34% of your training batches will contain question-answering examples on average. This works well when the natural distribution matches your goals. A general assistant that users primarily interact with through simple questions might legitimately benefit from heavy QA training. The main drawback is underfitting on minority tasks. The model may not see enough math examples to develop reliable reasoning, even if that skill is important for the use case you care about.

Equal sampling takes the opposite approach, assigning each task equal probability regardless of dataset size. If you have seven task categories, each receives roughly 14.3% of the training examples. This strategy ensures the model sees rare tasks frequently enough to learn from every category. However, it introduces its own problems. It may waste model capacity by over-training on already well-represented tasks. More seriously, it can cause severe overfitting when small task datasets must be repeated many times to reach equal representation. If your math dataset has only 8,000 examples but needs to match 120,000 QA examples, each math example must be seen 15 times across a single epoch, risking memorization of specific problem instances rather than generalization to new problems.

Temperature-based sampling uses a temperature parameter to produce a smooth interpolation between proportional and equal sampling. This approach gives you continuous control over how aggressively to upweight minority tasks. The key insight is that you rarely want either extreme. You want minority tasks represented frequently enough for real learning, but you do not need or want perfectly uniform sampling if your tasks vary naturally in importance or volume. Temperature-based sampling lets you find the right balance for your specific situation.

Given raw task proportions $p_i$, the sampling probability becomes:

where:

• $q_i$: the adjusted sampling probability for task $i$, representing how likely you are to draw an example from this task during training
• $p_i$: the raw proportion of task $i$ in the original dataset, calculated as the count of examples in task $i$ divided by the total number of examples
• $p_i^{1/T}$: the proportion for task $i$ raised to the power $1/T$, which compresses the differences between large and small proportions to make the distribution more uniform
• $T$: the temperature parameter that controls how aggressively to flatten the distribution, with larger values creating more uniform sampling
• $\sum_{j} p_j^{1/T}$: the sum of exponentiated proportions across all tasks, which normalizes the adjusted values so they sum to one and form a valid probability distribution

The mathematical mechanism is elegant. When $T = 1$, the exponent is $1/1 = 1$, so $p_i^1 = p_i$, and the formula reduces to $q_i = p_i$. You recover proportional sampling exactly. As $T$ increases, the exponent $1/T$ approaches zero. Any positive number raised to a power near zero approaches one, so all exponentiated proportions converge toward one regardless of the original $p_i$ values. The normalized distribution therefore approaches the uniform distribution. The temperature parameter continuously interpolates between these two extremes.

The three key operating regimes are:

1. When $T = 1$: The formula reduces to proportional sampling, preserving the original natural distribution without any adjustment.
2. When $T \to \infty$: All sampling weights converge to the same value, creating perfectly uniform sampling where each task is equally likely regardless of its original size.
3. When $T \approx 2$ to $3$: The distribution is flattened but not made uniform. Rare tasks are upweighted enough to appear frequently during training, while common tasks are still sampled more often than rare ones. This shows their greater natural occurrence. This intermediate regime gives the best balance for most practical instruction tuning scenarios.

When training for multiple epochs, you must carefully manage how sampling and repetition interact. With proportional sampling across multiple epochs, training remains dominated by common tasks. After three epochs, the model may have seen each QA example three times while still finding math examples sparse. Conversely, with equal sampling, rare task examples must be repeated far more frequently than common ones within each epoch to balance the distribution, and across multiple epochs this repetition compounds sharply, increasing the risk of memorizing specific examples rather than learning general patterns.

The risk of memorization is not hypothetical. A model that has seen the same 8,000 math problems dozens of times may produce correct answers by pattern-matching to memorized problem-answer pairs rather than by executing the underlying reasoning procedure. When it encounters a novel problem that is slightly different from the training examples, this memorized approach fails in ways that are hard to detect without careful evaluation. The model appears to perform well during training but fails to generalize.

To prevent this, you can cap how many times any single example appears during training:

The two central parameters that govern data mixing are worth understanding in concrete terms:

• temperature: Controls the sharpness of the sampling distribution. A higher temperature (for example $T = 2$) upweights minority tasks relative to their natural frequency. This keeps they receive more training attention. Lower temperatures preserve more of the original distribution, while very high temperatures approach uniform sampling. In practice, $T = 2$ is a safe default for multi-task instruction tuning.
• max_repeats: The maximum number of times a single example can appear during balanced dataset construction. This parameter prevents overfitting on small task categories by capping repetition even when equal sampling would require seeing those examples far more often. A value of 3 is commonly used; values above 5 are generally risky for datasets smaller than a few thousand examples.

These two parameters interact. A high temperature combined with a low max_repeats cap means that rare tasks will be sampled often but can only appear a limited number of times, creating a tension that cannot be resolved without collecting more data. When you hit this limit, the practical answer is to invest in data collection for the under-represented tasks rather than trying to compensate through sampling tricks.

Standard language modeling calculates loss for all tokens. In instruction tuning, this behavior wastes gradient capacity on recreating instructions instead of learning to produce helpful responses. Loss masking is the technique that redirects the training signal to where it belongs.

The concept is straightforward once you understand how gradient flow works. Every token that contributes to the loss function produces a gradient update that adjusts the model's parameters. If both the instruction tokens and the response tokens contribute equally to the loss, then the model is simultaneously being trained to reproduce instructions and to generate appropriate responses. These two objectives are related but distinct. The model already knows how to predict text fluently from pre-training. What it needs to learn during instruction tuning is specifically how to transition from a received instruction to an appropriate response, not how to reproduce the instruction itself.

Think of loss masking like a professional training simulation. When training a customer service representative, you play recorded customer calls and ask the trainee to practice the representative's responses. You would not grade the trainee on how accurately they reproduced what the customer said. The customer's words are the context that the trainee receives; the trainee's words are what you are training. Loss masking makes the same distinction: the instruction is input context, and the response is what you are training the model to produce.

Consider an instruction tuning example:

Without loss masking, the model receives gradient updates for every token, including the instruction "Explain photosynthesis in simple terms." But recreating user instructions is not the goal. The model should learn to generate helpful responses, not memorize prompts. Training on prompt tokens dilutes the learning signal with information the model already understands from pre-training. The pre-trained model is already excellent at predicting the token "Explain" given the token "User:" in its context. Spending gradient updates to reinforce this existing capability is a waste.

Loss masking addresses this by zeroing out the loss for prompt tokens. Only the assistant's response contributes to parameter updates. The result is that every gradient step moves the model's parameters specifically in the direction of better response generation, making instruction tuning more sample-efficient and more precisely targeted.

The effect is particularly significant for long prompts. In retrieval-augmented generation scenarios, the prompt may contain hundreds or thousands of tokens of retrieved context. Without loss masking, the model would spend the large majority of its training budget on predicting tokens that appear in documents it was given as context, which is irrelevant to its goal of creating accurate responses. With loss masking, all of that context is ignored in the loss calculation and only the response tokens shape the parameters.

The necessary detail in this implementation is how the reduction works. Standard cross-entropy with reduction="mean" averages over all positions in the sequence. By using reduction="none" first, we get a per-token loss, which we then multiply by the mask (zeroing out prompt positions) and divide by the total number of active mask positions. This ensures the averaged loss shows only the response tokens, not the entire sequence length. If we divided by sequence length instead of by the mask sum, the loss would be artificially diluted by all the zero-loss prompt positions, making the effective gradient much smaller than intended.

Loss masks must align with tokenized sequences. This requires tracking where the prompt ends and the response begins. In practice, this tracking happens during data preprocessing, where the tokenizer produces an attention mask alongside the token IDs, and you augment this with a separate loss mask that follows the same alignment:

A subtle but important detail is that finding the response start index requires care when using sub-word tokenizers. You cannot simply count characters in the prompt and divide by some average token length. The correct approach is to tokenize the prompt and the full sequence separately, then compute response_start_idx = len(tokenized_prompt). This is exact and handles cases where the prompt-response boundary falls inside a multi-character token correctly.

For multi-turn conversations, the masking logic extends naturally. Only tokens that are part of assistant turns contribute to the loss; user turns and system messages are masked throughout. This means that in a long conversation, the active loss positions may be scattered through the sequence, interspersed with masked regions corresponding to each user message.

The following visualization shows how loss masking focuses training on the response portion of each example:

Loss masking materially affects what the model learns during instruction tuning. Without masking, gradients flow from both prompt and response tokens, diluting the signal that teaches response generation. With masking, all gradient information comes from response tokens, making training more efficient and focused.

The effect varies considerably across different types of instruction-following examples. In short question-answering examples, the prompt might constitute 40% of the tokens, so masking reclaims 40% of the gradient budget for response learning. In code generation tasks where the prompt includes a detailed specification and the response is a single function, the ratio might be 30% prompt to 70% response. But in long-context applications where the prompt contains retrieved documents or multi-turn conversation history, prompts can easily constitute 60% to 70% of all tokens in the sequence.

The effect is particularly pronounced in scenarios with long prompts relative to responses. In long-context applications or multi-turn conversations, prompts can comprise 60% to 70% of tokens. Without masking, most of the learning signal would be spent on recreating context rather than learning to respond appropriately. The model would implicitly be trained to be a very good text predictor for the kinds of text that appear in instructions, which is exactly what it already learned to do during pre-training. Loss masking gives instruction tuning a different objective from continued pre-training on another distribution.

Let us trace through a concrete numerical example to see exactly what happens during a masked loss computation. Suppose we have a simple vocabulary of five tokens: "the", "cat", "sat", "mat", "on". Our instruction-response sequence consists of three prompt tokens followed by two response tokens.

The prompt is "the cat sat" (token IDs 0, 1, 2). The response is "on mat" (token IDs 3, 4). The full sequence is [0, 1, 2, 3, 4] with loss mask [0, 0, 0, 1, 1].

During training, the model processes the full sequence autoregressively. For the shifted prediction task, the input is [0, 1, 2, 3] and the target is [1, 2, 3, 4]. The loss mask applied to the shifted targets is [0, 0, 1, 1] (we shift the mask along with the labels).

Suppose the model produces the following logits for the four prediction positions:

• Position 0 (predicting token 1 = "cat"): logits = [0.1, 2.0, 0.3, 0.1, 0.1], cross-entropy = 0.15. Mask = 0, so contribution = 0.
• Position 1 (predicting token 2 = "sat"): logits = [0.2, 0.4, 1.8, 0.2, 0.2], cross-entropy = 0.22. Mask = 0, so contribution = 0.
• Position 2 (predicting token 3 = "on"): logits = [0.1, 0.1, 0.3, 1.9, 0.3], cross-entropy = 0.18. Mask = 1, contribution = 0.18.
• Position 3 (predicting token 4 = "mat"): logits = [0.2, 0.2, 0.2, 0.5, 1.7], cross-entropy = 0.31. Mask = 1, contribution = 0.31.

The total masked loss is:

where the denominator is the sum of mask values (2), not the total sequence length (4). If we had used the total sequence length, the loss would be $0.49 / 4 = 0.1225$, which is half as large. The smaller loss would produce smaller gradients, making training half as efficient. The denominator must be the number of active (unmasked) positions.

This numerical example shows why the implementation detail in compute_masked_loss matters: total_loss = masked_loss.sum() / shift_mask.sum() divides by the mask sum, not by the sequence length. Getting this wrong silently reduces gradient magnitudes and can require doubling your learning rate or training steps to compensate.

Instruction tuning requires careful hyperparameter selection. Unlike pre-training, which processes trillions of tokens over many epochs with gradual learning rate schedules tuned over weeks, instruction tuning typically involves much smaller datasets and far fewer training steps. This concentration amplifies the importance of each hyperparameter choice. A pre-training run that uses a slightly suboptimal learning rate loses only a fraction of a percent in final performance. An instruction tuning run with a suboptimal learning rate might fail entirely, creating a model that either ignores instructions or loses its pre-trained capabilities.

The basic challenge of instruction tuning hyperparameter selection is the tension between learning speed and knowledge preservation. Faster learning rates help the model adapt to the instruction-following objective quickly, but large gradient updates can overwrite the weights learned during pre-training, erasing capabilities that no amount of instruction tuning can recover. Slower learning rates preserve pre-trained knowledge but may require so many steps that the small instruction tuning dataset is exhausted before the model has properly adapted.

Think of this tension like training a highly skilled employee on a new workflow. If you push them too hard and too fast, they may forget important parts of their existing expertise in the process of adapting. If you train them too gently, you may never fully complete the transition. The optimal pace lies in between, and it depends on how different the new workflow is from everything they already know.

The learning rate is typically lower than pre-training, usually in the range of $1 \times 10^{-5}$ to $5 \times 10^{-5}$. Higher learning rates risk catastrophic forgetting, where the model loses pre-trained capabilities while learning to follow instructions. Lower learning rates preserve more of the base model's knowledge but may require more training steps to reach adequate instruction following.

The reason instruction tuning uses lower learning rates than pre-training is not merely caution. It shows the different nature of the optimization problem. During pre-training, the model starts from random initialization and must learn everything from scratch. Large updates early in training make sense because there is nothing to preserve. During instruction tuning, the model already has useful representations learned from billions of tokens. The goal is to refine the output behavior without destabilizing the underlying representations, which calls for smaller, more targeted updates.

A warmup period is particularly important in instruction tuning. During warmup, the learning rate gradually increases from near zero to the target value over the first few percent of training steps. This prevents large gradient updates before the optimizer has accumulated sufficient statistics about the gradient landscape. Without warmup, the first few batches of instruction data can produce outsized updates that disrupt the carefully learned pre-trained representations before the optimizer has had a chance to stabilize. With warmup, the transition from pre-trained to instruction-tuned behavior is more gradual and controlled.

The cosine learning rate schedule then smoothly decays the learning rate from its peak back toward zero over the remaining training steps. This means the most aggressive learning happens in the early middle of training, with gradually more conservative updates as the model converges. The intuition is that early in instruction tuning, large updates are needed to adapt the output behavior; late in instruction tuning, only fine adjustments are needed, and overly large updates risk over-fitting.

Larger batch sizes give more stable gradient estimates by averaging over more examples, which reduces the noise in the update direction. This stability comes at a cost: memory. Larger batches require proportionally more GPU memory, which limits the maximum feasible batch size on any given hardware.

Gradient accumulation is the standard solution to this hardware constraint. Instead of updating the model after every batch, you accumulate gradients over several smaller batches and then perform a single update. The mathematical effect is identical to training with a larger batch, but the peak memory usage is only that of a single small batch:

Effective batch sizes between 64 and 256 are common for instruction tuning. Smaller batches introduce more noise into gradient estimates, which can help generalization but may slow convergence by making the optimization trajectory more erratic. Larger batches converge faster and more predictably but may generalize less well and are more prone to sharp minima. For instruction tuning specifically, batch sizes in the range of 128 to 256 have been found to work reliably across many experimental settings.

One important subtlety with gradient accumulation and loss masking is that the effective averaging interacts with the masked loss computation. If you accumulate gradients over four batches before updating, and the number of active (unmasked) tokens varies considerably between batches, the gradient magnitude will vary as well, which can create instability. Some implementations normalize the accumulated gradient by the total number of active tokens across all accumulation steps rather than averaging within each step separately.

Instruction tuning typically uses only 1 to 3 epochs over the training data. This is sharply fewer epochs than classic fine-tuning scenarios. The reason is that instruction datasets are relatively small compared to the pre-training corpus, which means each epoch exposes the model to far fewer parameter updates. With 100,000 instruction examples and a batch size of 128, a single epoch requires only about 780 gradient updates, compared to millions during pre-training. Three epochs therefore amounts to roughly 2,300 updates total, a modest amount that rarely causes over-fitting if the learning rate is correctly set.

The risk of over-fitting nevertheless increases with each epoch. More epochs risk overfitting to the instruction format rather than learning general instruction-following capabilities. Signs of overfitting include training loss continuing to decrease while validation loss increases, model outputs becoming formulaic or repetitive (the model learns to respond in specific patterns it has seen repeatedly rather than adapting to novel instructions), and performance degrading on held-out tasks that were not part of the training set.

Early stopping based on validation loss is the standard guard against overfitting in instruction tuning. Save checkpoints at regular intervals, for example every 100 or 200 steps, and evaluate on a held-out set of instruction examples that were not included in training. When validation loss begins to rise, select the checkpoint from the step just before the upturn. This checkpoint is the best balance between fitting the training data and generalizing to new instructions.

The validation set should contain examples from all task categories, including minority tasks. If your validation set contains only QA examples, you may select a checkpoint that performs well on QA but has already started over-fitting on the instruction format in ways that harm other capabilities. A diverse validation set gives you a more faithful signal about the model's overall instruction-following ability.

Instruction tuning naturally supports multi-task learning. By training on diverse instruction types simultaneously, the model learns transferable representations that generalize across tasks. This is one of the most important findings in the instruction tuning literature: models trained on a diverse mixture of tasks often outperform models trained on any individual task alone, even when evaluated on that task.

The mechanism behind this benefit is that different tasks share underlying linguistic and reasoning skills. Summarization requires identifying the most important information in a document. Question answering requires locating specific pieces of information in context. These two tasks share the skill of identifying relevance in text. By training on both simultaneously, the model develops a more reliable notion of relevance than it would from training on either task alone. Translation and creative writing both require generating fluent, natural text with appropriate style. Math reasoning and code generation both require multi-step structured inference. These cross-task overlaps create a web of mutually reinforcing skills.

The key benefits include:

• Positive transfer: Skills learned on one task improve performance on related tasks. A model trained on code generation tends to improve on math reasoning as well, because both benefit from precise step-by-step thinking.
• Regularization: Task diversity acts as a natural regularizer. The model cannot simply memorize the output format of a single task type because it is constantly switching between different formats and objectives. This diversity prevents the model from collapsing into narrow response patterns.
• Emergent capabilities: Models sometimes develop abilities that were not explicitly trained in any single task. The combination of skills from different tasks can produce generalization to novel task types. Early instruction tuning papers reported surprising zero-shot performance on tasks that were not part of the training mixture.

However, multi-task learning also introduces the risk of negative transfer, where tasks with conflicting objectives interfere with each other. Tasks requiring very different output styles can conflict. A classification task that expects a single label and a creative writing task that expects hundreds of words of flowing prose have superficially incompatible output distributions. Training on both may cause the model to produce outputs that are oddly short or unexpectedly long when the task format is ambiguous.

Temperature-based sampling and careful task curation help mitigate these conflicts. By using moderate temperatures, you ensure that no single task dominates the gradient signal to the exclusion of others. By curating your task mixture to avoid tasks with fundamentally incompatible objectives, you reduce the chance of harmful interference. In practice, tasks that differ in format but share underlying skills (factual QA vs. reading comprehension vs. summarization) tend to transfer positively, while tasks that differ in both format and the skills they exercise are more likely to interfere.

Despite its effectiveness, instruction tuning training carries several important limitations that practitioners need to understand before deploying instruction-tuned models.

The most basic limitation is that instruction tuning can only teach behaviors that are representable in the training data. The model cannot learn to follow instructions it has never seen examples of, and it cannot learn to produce responses that no human annotator thought to include. If your instruction dataset does not contain examples of the model declining dangerous requests, the model will not learn to decline them. If your dataset does not contain examples of acknowledging uncertainty, the model may confidently produce incorrect answers. The training data sets an absolute ceiling on the behaviors the model can exhibit, regardless of how advanced the training procedure is.

Data quality interacts with training procedure in ways that can be difficult to diagnose. A dataset that contains systematically incorrect responses will teach the model to produce those incorrect responses more confidently as training progresses. Unlike pre-training on text that was already filtered for quality, instruction tuning datasets often rely on human annotation processes that introduce systematic biases. Annotators may favor certain response styles, certain lengths, or certain ways of hedging uncertainty that do not reflect the best response to the instruction. These biases become encoded in the model and are difficult to remove without collecting new data.

The catastrophic forgetting problem remains a challenge even with careful hyperparameter selection. Instruction tuning shifts the model's weight distribution to better serve the instruction-following objective, and this shift can reduce performance on capabilities that were not exercised during instruction tuning. A model instruction-tuned primarily on conversational tasks may lose some of its pre-trained fluency in technical domains if those domains are not represented in the instruction dataset. The learning rate and epoch count can be tuned to minimize forgetting, but they cannot eliminate it entirely. Full fine-tuning of all parameters amplifies this risk. Parameter-efficient methods like LoRA, which we cover in later chapters, partially address this by limiting which parameters are updated during instruction tuning.

Finally, there is a distributional mismatch between instruction tuning and deployment. During training, the model receives well-formed, carefully constructed instructions from the training dataset. During deployment, users send poorly formatted, ambiguous, incomplete, or adversarial instructions that may be quite different from anything in the training distribution. The model may handle well-formed instructions reliably while failing on edge cases that were never represented in training. This is not a flaw in the training procedure per se; it shows the inherent difficulty of generalizing from finite training examples to an in effect unbounded space of possible user instructions. Evaluation on diverse, realistic instruction sets that include edge cases is needed for understanding where the model will and will not perform well.

Instruction tuning training requires balancing multiple competing concerns: so adequate coverage of minority tasks without overfitting, focusing learning on response generation through loss masking, and selecting hyperparameters that preserve pre-trained capabilities while teaching instruction following.

The three major decisions we covered in this chapter work together as an interconnected system:

1. Data mixing strategy: Use temperature-based sampling with $T \approx 2$ to $3$ to balance task representation. This ensures minority tasks receive enough training signal while common tasks remain appropriately frequent. Cap repetition of small datasets at 3 to 5 repeats to prevent memorization.
2. Loss masking: Zero out the loss on all prompt tokens so that every gradient update comes from response generation. This is not optional for instruction tuning to work well. Without it, the model spends significant capacity on pre-training-style text prediction rather than instruction following. Remember to normalize by the number of active (unmasked) tokens, not by sequence length.
3. Hyperparameter selection: Use lower learning rates ($1$ to $5 \times 10^{-5}$) than pre-training to avoid catastrophic forgetting. Use a short warmup period (about 3% of training steps) to stabilize early updates. Train for 1 to 3 epochs and monitor validation loss on a diverse held-out set, using early stopping when validation loss begins to rise. Target effective batch sizes of 128 to 256 through gradient accumulation if memory is limited.

These techniques work together to produce models that follow instructions reliably across diverse task types while maintaining their broader language capabilities. The next chapter examines how to evaluate whether instruction tuning has succeeded, moving from the training mechanics we covered here to the equally important question of measuring the resulting model's behavior.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about instruction tuning training.