Hyperparameter Selection › Fine-Tuning Hyperparameter Recipes

Pretraining and fine-tuning have fundamentally different hyperparameter needs, and applying pretraining defaults to fine-tuning is one of the most common mistakes in practice. Understanding why they differ helps you adapt appropriately.

During pretraining, the model is learning from scratch on a vast and diverse dataset. The learning rate can be relatively high because the model has no existing knowledge to protect, and large gradient steps efficiently move parameters from their random initial positions to useful regions of weight space. Weight decay at $0.1$ is safe because with trillions of tokens, the risk of catastrophic overfitting is low.

During fine-tuning on a task-specific dataset (which might contain thousands to millions of examples rather than trillions), the situation is inverted. The model already has useful representations from pretraining. Large gradient steps destroy these representations rather than refining them. High weight decay accelerates this destruction by actively pushing weights toward zero.

The community consensus for supervised fine-tuning (SFT) and similar adaptation tasks:

• Learning rate: 1/10 to 1/100 of the pretraining learning rate. For a model pretrained at $3 \times 10^{-4}$, use $3 \times 10^{-5}$ to $3 \times 10^{-6}$ for fine-tuning. Lower rates better preserve pretraining knowledge.
• Schedule: cosine or linear decay over the fine-tuning steps. Total steps are often much fewer (1-3 epochs over the fine-tuning dataset rather than a fixed token budget).
• Warmup: shorter than pretraining, often just 1-3% of fine-tuning steps or a fixed 50-100 steps. The model is already well-initialized, so the second moment estimate stabilizes quickly.
• Weight decay: lower than pretraining, often $0.01$ or even $0.0$. Aggressive weight decay during fine-tuning can erase the pretrained representations that make the model useful.
• Dropout: may be reintroduced at low rates ($0.05$ to $0.1$) when the fine-tuning dataset is very small and overfitting is a concern.
• Batch size: smaller than pretraining, often 16 to 128 samples. Smaller batches increase gradient noise, which provides implicit regularization and can help the model adapt to the new distribution without memorizing.

For parameter-efficient fine-tuning methods like LoRA, the learning rate for the trainable adapters is typically higher than for full fine-tuning, often back in the $1 \times 10^{-4}$ range, because the adapters are small and randomly initialized relative to the frozen pretrained weights. The frozen base model provides stable feature extraction while the adapters learn task-specific modifications.