Fine-tuning Learning Rates › Fine-tuning Learning Rates

When you fine-tune a pre-trained language model, choosing the right learning rate is more important than when training from scratch. A learning rate that works well for pre-training can destroy months of computational effort in minutes of fine-tuning. Conversely, a learning rate that is too small might never adapt the model to your task. The challenge lies in finding the narrow band of learning rates that let real adaptation without erasing the knowledge the model has already acquired.

Think of the pre-trained model as a craftsperson who has spent years building an advanced set of skills. When you hire them for a new project, you want them to apply their expertise to the new context, not forget everything they know and start from scratch. The learning rate is the intensity of their retraining. Too intense, and they forget their foundational skills. Too gentle, and they never adapt to the new role. You need a calibrated approach that respects their existing expertise while allowing specialization.

This analogy captures something real about the mathematics. Pre-trained weights stand for a particular region of parameter space that took enormous computational resources to reach. This region is not just any local minimum; it is a high-quality one that encodes linguistic structure, semantic relationships, and world knowledge accumulated over billions of training tokens. When we fine-tune, we are not searching for a good solution from scratch. We are making careful adjustments around an already-excellent solution. The learning rate controls exactly how careful or aggressive those adjustments are.

The challenge is compounded by the heterogeneous structure of transformer models. A 12-layer BERT model is not a single monolithic entity but a stack of specialized subsystems: word embeddings that encode vocabulary semantics, early transformer layers that capture syntax and local patterns, middle layers that build richer contextual representations, and upper layers that prepare outputs for the specific pre-training task. Each of these subsystems has a different relationship to the fine-tuning objective. Some need minimal adjustment, others need substantial reconfiguration, and the task head needs to be trained almost from scratch. A single learning rate cannot optimally serve all of these needs simultaneously.

The strategies in this chapter address this heterogeneity systematically. Discriminative fine-tuning and layer-wise learning rate decay assign different rates to different layers, respecting the hierarchical structure of learned representations. Warmup schedules protect the model during the dangerous early phase of adaptation. Decay schedules guide convergence toward a stable final solution. Together, these techniques provide a way to manage the fine-tuning objective with care.