Full Fine-tuning › Comparing Hyperparameter Configurations

Michael BrenndoerferNovember 25, 20253 min read
Language AI HandbookAll chapters
Part 34 of 67Fine-tuning Fundamentals
Chapter 250Full Fine-tuning › Comparing Hyperparameter Configurations
Reading section 8 of 11 · Comparing Hyperparameter Configurations · 3 minSection 8 of 11 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 250 of 513

Section 8 of 11 · 3 min read

Full Fine-tuning › Comparing Hyperparameter ConfigurationsOne section · Sections in this chapter8 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To illustrate the impact of hyperparameter choices, let's compare different learning rate configurations across the training process. This comparison reveals several important patterns that help build intuition for how fine-tuning behaves under different conditions.

Out[31]:
Visualization
Line plot comparing validation accuracy curves for four different learning rate settings.
Learning rate impact on fine-tuning performance across three epochs for BERT-base on sentiment classification. The optimal rate of 2x10^-5 reaches 87% accuracy by balancing adaptation speed with preservation of pre-trained knowledge. The lower rate of 1x10^-5 converges more slowly, reaching only 81% final accuracy within 3 epochs. The higher rate of 5x10^-5 shows fast initial progress but plateaus early at 84%. The excessively high rate of 1x10^-4 causes training instability with non-monotonic accuracy (rising then falling), confirming that rates above the recommended range destroy pre-trained representations.

The comparison reveals several patterns characteristic of fine-tuning:

  • Learning rates in the 2e-5 range typically perform best for BERT-style models, confirming the empirically established community recommendation
  • Lower rates (1e-5) learn more slowly but can reach good results with more epochs, making them suitable when computational budget allows for longer training runs
  • Higher rates (5e-5) show faster initial progress but can plateau earlier, showing that more aggressive updates quickly reach a local optimum that may not be globally optimal
  • Excessively high rates (1e-4) damage pre-trained representations and lead to poor, unstable performance, with non-monotonic accuracy curves that reveal the destructive effect of oversized updates

The non-monotonic behavior of the 1e-4 configuration is particularly instructive. Accuracy rises in epoch 1, as even large random updates occasionally improve performance, then falls in epoch 2 as the pre-trained representations are corrupted, then partially recovers in epoch 3 as the model attempts to find a new equilibrium. This pattern is a clear diagnostic for a learning rate that is too high: the model cannot maintain consistent progress because each update is large enough to move the parameters away from good configurations.