Instruction Tuning Training › Training Hyperparameters

Instruction tuning requires careful hyperparameter selection. Unlike pre-training, which processes trillions of tokens over many epochs with gradual learning rate schedules tuned over weeks, instruction tuning typically involves much smaller datasets and far fewer training steps. This concentration amplifies the importance of each hyperparameter choice. A pre-training run that uses a slightly suboptimal learning rate loses only a fraction of a percent in final performance. An instruction tuning run with a suboptimal learning rate might fail entirely, creating a model that either ignores instructions or loses its pre-trained capabilities.

The basic challenge of instruction tuning hyperparameter selection is the tension between learning speed and knowledge preservation. Faster learning rates help the model adapt to the instruction-following objective quickly, but large gradient updates can overwrite the weights learned during pre-training, erasing capabilities that no amount of instruction tuning can recover. Slower learning rates preserve pre-trained knowledge but may require so many steps that the small instruction tuning dataset is exhausted before the model has properly adapted.

Think of this tension like training a highly skilled employee on a new workflow. If you push them too hard and too fast, they may forget important parts of their existing expertise in the process of adapting. If you train them too gently, you may never fully complete the transition. The optimal pace lies in between, and it depends on how different the new workflow is from everything they already know.

The learning rate is typically lower than pre-training, usually in the range of $1 \times 10^{-5}$ to $5 \times 10^{-5}$. Higher learning rates risk catastrophic forgetting, where the model loses pre-trained capabilities while learning to follow instructions. Lower learning rates preserve more of the base model's knowledge but may require more training steps to reach adequate instruction following.

The reason instruction tuning uses lower learning rates than pre-training is not merely caution. It shows the different nature of the optimization problem. During pre-training, the model starts from random initialization and must learn everything from scratch. Large updates early in training make sense because there is nothing to preserve. During instruction tuning, the model already has useful representations learned from billions of tokens. The goal is to refine the output behavior without destabilizing the underlying representations, which calls for smaller, more targeted updates.

A warmup period is particularly important in instruction tuning. During warmup, the learning rate gradually increases from near zero to the target value over the first few percent of training steps. This prevents large gradient updates before the optimizer has accumulated sufficient statistics about the gradient landscape. Without warmup, the first few batches of instruction data can produce outsized updates that disrupt the carefully learned pre-trained representations before the optimizer has had a chance to stabilize. With warmup, the transition from pre-trained to instruction-tuned behavior is more gradual and controlled.

The cosine learning rate schedule then smoothly decays the learning rate from its peak back toward zero over the remaining training steps. This means the most aggressive learning happens in the early middle of training, with gradually more conservative updates as the model converges. The intuition is that early in instruction tuning, large updates are needed to adapt the output behavior; late in instruction tuning, only fine adjustments are needed, and overly large updates risk over-fitting.

Larger batch sizes give more stable gradient estimates by averaging over more examples, which reduces the noise in the update direction. This stability comes at a cost: memory. Larger batches require proportionally more GPU memory, which limits the maximum feasible batch size on any given hardware.

Gradient accumulation is the standard solution to this hardware constraint. Instead of updating the model after every batch, you accumulate gradients over several smaller batches and then perform a single update. The mathematical effect is identical to training with a larger batch, but the peak memory usage is only that of a single small batch:

Effective batch sizes between 64 and 256 are common for instruction tuning. Smaller batches introduce more noise into gradient estimates, which can help generalization but may slow convergence by making the optimization trajectory more erratic. Larger batches converge faster and more predictably but may generalize less well and are more prone to sharp minima. For instruction tuning specifically, batch sizes in the range of 128 to 256 have been found to work reliably across many experimental settings.

One important subtlety with gradient accumulation and loss masking is that the effective averaging interacts with the masked loss computation. If you accumulate gradients over four batches before updating, and the number of active (unmasked) tokens varies considerably between batches, the gradient magnitude will vary as well, which can create instability. Some implementations normalize the accumulated gradient by the total number of active tokens across all accumulation steps rather than averaging within each step separately.

Instruction tuning typically uses only 1 to 3 epochs over the training data. This is sharply fewer epochs than classic fine-tuning scenarios. The reason is that instruction datasets are relatively small compared to the pre-training corpus, which means each epoch exposes the model to far fewer parameter updates. With 100,000 instruction examples and a batch size of 128, a single epoch requires only about 780 gradient updates, compared to millions during pre-training. Three epochs therefore amounts to roughly 2,300 updates total, a modest amount that rarely causes over-fitting if the learning rate is correctly set.

The risk of over-fitting nevertheless increases with each epoch. More epochs risk overfitting to the instruction format rather than learning general instruction-following capabilities. Signs of overfitting include training loss continuing to decrease while validation loss increases, model outputs becoming formulaic or repetitive (the model learns to respond in specific patterns it has seen repeatedly rather than adapting to novel instructions), and performance degrading on held-out tasks that were not part of the training set.