Fine-tuning Learning Rates › Summary

Fine-tuning learning rates require careful consideration that goes beyond simply selecting a single value. The strategies covered in this chapter address different aspects of the fine-tuning challenge, and together they form a principled framework for protecting pre-trained knowledge while letting real task-specific adaptation.

Layer-wise learning rate decay assigns different learning rates to different layers based on the principle that lower layers capture general features (requiring less adaptation) while upper layers capture task-specific features (requiring more adaptation). Using a decay factor between 0.9 and 0.95 across layers typically gives a good balance between differentiation and flexibility, with the geometric decay formula $\eta_l = \eta \cdot \xi^{L-l}$ so that the ratio between adjacent layers remains constant throughout the network.

Learning rate warmup gradually increases the learning rate from zero during the first 5-10% of training. This stabilizes optimization by letting Adam's moment estimates to accumulate, preventing the randomly initialized task head from sending large, noisy gradients into the pre-trained backbone, and giving early mini-batch gradient estimates time to become more representative of the full data distribution.

Learning rate decay reduces the rate throughout training after warmup, helping the model converge to a stable minimum rather than oscillating around it. Cosine decay is often the preferred choice because its smooth behavior near both the start and end of the decay phase reduces the risk of instability during transitions. Linear decay gives a simpler alternative that works equally well in most cases.

Combining these techniques creates a reliable fine-tuning setup: warmup followed by decay sets the global learning rate trajectory, while LLRD distributes that trajectory differentially across the network hierarchy. The task head typically receives 10x the base learning rate since it starts from random initialization and needs aggressive training from the outset. Understanding why each component works, not just how to configure it, allows you to adapt these strategies intelligently when facing unusual tasks, architectures, or data conditions that fall outside the standard BERT fine-tuning regime.

These strategies stand for one dimension of efficient fine-tuning. In the next chapter, we will examine data efficiency: how to get the most out of limited labeled data when adapting pre-trained models to new tasks.