Fine-tuning Learning Rates › The Fine-tuning Learning Rate Challenge

As we discussed in the previous chapter on catastrophic forgetting, fine-tuning involves a basic tension. You want the model to learn task-specific patterns, but you do not want to overwrite the rich linguistic knowledge acquired during pre-training. Learning rates sit at the heart of this tension because they directly control how much each gradient update changes the model's parameters.

Understanding why fine-tuning is qualitatively different from pre-training requires thinking carefully about the optimization objective. During pre-training, the model starts from random weights and explores the loss surface from a completely uninformed starting point. The optimizer has to make large moves to escape the random initialization and find real structure in the data. There is nothing to lose, because the starting weights encode no knowledge worth preserving. Large learning rates are not just safe at this stage; they are necessary for making rapid progress across billions of training tokens.

Fine-tuning presents a fundamentally different scenario. You are starting from a carefully optimized point that already captures real language structure. The pre-trained weights stand for a local minimum that took enormous computational resources to reach, and this minimum encodes useful knowledge about syntax and semantics as well as world knowledge. Large updates can push parameters far from this optimum, destroying useful representations that took the original training process millions of gradient steps to construct. The loss surface near a high-quality pre-trained minimum is typically much more sensitive than it is near random initialization: small perturbations in the wrong direction can cascade through the network and destabilize representations that downstream layers depend on.

During pre-training, models typically use learning rates between $10^{-4}$ and $10^{-3}$ with the Adam optimizer. These relatively large rates help the model explore the parameter space efficiently when learning from scratch. But fine-tuning requires a reduction of roughly one to two orders of magnitude, with typical values in the range of $10^{-5}$ to $5 \times 10^{-5}$ for BERT-style models. This reduction shows the different nature of the optimization problem: rather than exploring a large parameter space, we are making careful adjustments around an already-good solution.

Even this simple observation raises deeper questions. Not all parts of the model are equally sensitive to update magnitude. Not all parts need to change by the same amount. And the optimal strategy depends on characteristics of both the pre-trained model and the fine-tuning task. The strategies we explore in this chapter emerged from attempts to answer these questions systematically. Rather than treating fine-tuning as simply "training with a smaller learning rate," we have developed fine-grained approaches that consider which parts of the model need to change and by how much.

• Should all layers receive the same learning rate?
• Should the rate start small and grow, or start larger and decay?
• How do these choices interact with different model architectures?

The answers depend on the hierarchical structure of what transformers learn, which motivates our first major technique.