Fine-tuning Learning Rates › Practical Guidelines

Michael BrenndoerferNovember 27, 20254 min read
Language AI HandbookAll chapters
Part 34 of 67Fine-tuning Fundamentals
Chapter 252Fine-tuning Learning Rates › Practical Guidelines
Reading section 9 of 12 · Practical Guidelines · 4 minSection 9 of 12 · 4 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 252 of 513

Section 9 of 12 · 4 min read

Fine-tuning Learning Rates › Practical GuidelinesOne section · Sections in this chapter9 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

These learning rate strategies have been refined through extensive empirical study across many NLP benchmarks, and certain practical patterns have emerged as reliable defaults. Understanding these defaults and why they differ across model sizes helps you adapt them intelligently to novel scenarios.

For BERT-base and similar models (110M parameters), the relatively modest size means that even aggressive learning rates rarely cause catastrophic failure; the model has enough capacity to recover from moderate perturbations. Recommended settings:

  • Base learning rate: 2×10−52 \times 10^{-5} to 5×10−55 \times 10^{-5}
  • LLRD decay: 0.9 to 0.95
  • Warmup: 6-10% of training steps
  • Task head learning rate: 10x base rate

For larger models (GPT-2, RoBERTa-large, 340M+ parameters), the increased depth means that gradient signals must travel through more layers, amplifying any instabilities in the optimization dynamics. More conservative settings are needed:

  • Base learning rate: 10−510^{-5} to 2×10−52 \times 10^{-5} (lower due to more layers)
  • LLRD decay: 0.85 to 0.95
  • Warmup: 10% of training steps
  • Consider lower weight decay (0.001 instead of 0.01)

For very large models (1B+ parameters), the sensitivity to learning rate is considerably higher, and even small overestimation of the learning rate can trigger training instability. Parameter-efficient methods like LoRA are often preferable for this regime, but when full fine-tuning is necessary:

  • Base learning rate: 5×10−65 \times 10^{-6} to 10−510^{-5}
  • More aggressive LLRD decay: 0.8 to 0.9
  • Longer warmup: 10-15% of training steps
  • Often better to use parameter-efficient methods like LoRA, which we'll cover in upcoming chapters

General diagnostic principles that apply across all model sizes:

  • Start with the lower end of learning rate ranges and increase if learning is too slow
  • Monitor both training loss and validation metrics; overfitting indicates the rate may be too high
  • If loss spikes early, increase warmup steps or decrease initial learning rate
  • If loss plateaus too early, try a higher learning rate or different decay schedule
  • Loss that decreases smoothly in training but stagnates on validation often indicates the decay schedule is letting the model to overfit; try reducing the warmup ratio to allow more steps at full rate, or switch to cosine decay if you are using linear
Out[23]:
Visualization
Bar chart showing recommended base learning rates decreasing from BERT-base to large LLMs.
Recommended base learning rates for different model sizes. Larger models require materially lower rates to prevent optimization instability, with the midpoint falling from around 3.5e-5 for BERT-base to 0.75e-5 for billion-parameter LLMs.
Bar chart showing recommended LLRD decay factors decreasing from BERT-base at 0.93 to large LLMs at 0.85.
Recommended layer-wise learning rate decay (LLRD) factors by model size. Larger models benefit from more aggressive decay (lower factors) to more strongly protect pre-trained lower-layer representations from being disrupted by fine-tuning.