Full Fine-tuning › Summary

Full fine-tuning adapts all parameters of a pre-trained model to a downstream task. The procedure involves loading a pre-trained checkpoint, adding a task-specific head, and training on labeled data with carefully selected hyperparameters. The configuration choices encode empirical wisdom accumulated from thousands of experiments: learning rates in the $10^{-5}$ range, warmup periods covering the first 6-10% of training, and 2-5 epochs of optimization.

Here are the necessary hyperparameters for fine-tuning:

• Learning rate: typically $1 \times 10^{-5}$ to $5 \times 10^{-5}$ (much lower than pre-training rates)
• Number of epochs: usually 2-5 (just enough for adaptation without overfitting)
• Batch size: 16-32 for consumer GPUs; use gradient accumulation for larger effective sizes
• Warmup ratio: 0.06 to 0.1 (6% to 10%) of training steps to stabilize early training
• Weight decay: around 0.01, applied to most but not all parameters

Learning rate schedules like linear or cosine decay, combined with warmup, give stable training dynamics. The batch size-learning rate relationship suggests scaling both together for consistent behavior. Gradient accumulation decouples memory requirements from effective batch size. This makes possible flexible training across hardware configurations.

Full fine-tuning excels when compute resources are available and maximum task performance is the goal. The complete update of all parameters allows the model to develop deep task-specific specialization that parameter-efficient methods cannot fully replicate. For resource-constrained scenarios or multi-task deployments, parameter-efficient methods offer attractive alternatives that sacrifice some peak performance for large reductions in memory and storage requirements. In the following chapters, we will examine catastrophic forgetting in depth and explore the parameter-efficient fine-tuning methods that have become the practical standard for large-scale deployments.