Fine-tuning Learning Rates › Worked Example: Fine-tuning BERT for Sentiment Analysis

Michael BrenndoerferNovember 27, 20253 min read
Language AI HandbookAll chapters
Part 34 of 67Fine-tuning Fundamentals
Chapter 252Fine-tuning Learning Rates › Worked Example: Fine-tuning BERT for Sentiment Analysis
Reading section 7 of 12 · Worked Example: Fine-tuning BERT for Sentiment Analysis · 3 minSection 7 of 12 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 252 of 513

Section 7 of 12 · 3 min read

Fine-tuning Learning Rates › Worked Example: Fine-tuning BERT for Sentiment AnalysisOne section · Sections in this chapter7 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

To make the abstract mechanics concrete, let's trace through a specific scenario: fine-tuning BERT-base-uncased for binary sentiment classification. We will compute the exact learning rates at representative steps throughout the training run and show how the combined schedule behaves in practice.

Setup: 8 epochs over a dataset of 25,000 training examples with batch size 32. This gives us approximately 25000×832=6,250\frac{25000 \times 8}{32} = 6{,}250 training steps total. We use a 10% warmup (625 steps), base learning rate η=2×10−5\eta = 2 \times 10^{-5}, LLRD decay factor ξ=0.95\xi = 0.95, and cosine decay.

Step 1: Compute the per-layer peak learning rates (end of warmup)

At the end of warmup, each layer reaches its assigned peak rate. Using the LLRD formula ηl=η⋅ξL−l\eta_l = \eta \cdot \xi^{L-l} with L=12L=12, η=2×10−5\eta = 2 \times 10^{-5}, ξ=0.95\xi = 0.95:

ηembeddings=2×10−5⋅0.9512≈1.07×10−5ηlayer 1=2×10−5⋅0.9511≈1.13×10−5ηlayer 6=2×10−5⋅0.956≈1.48×10−5ηlayer 12=2×10−5⋅0.950=2.00×10−5ηtask head=2×10−5×10=2.00×10−4\begin{aligned} \eta_{\text{embeddings}} &= 2 \times 10^{-5} \cdot 0.95^{12} \approx 1.07 \times 10^{-5} \\ \eta_{\text{layer 1}} &= 2 \times 10^{-5} \cdot 0.95^{11} \approx 1.13 \times 10^{-5} \\ \eta_{\text{layer 6}} &= 2 \times 10^{-5} \cdot 0.95^{6} \approx 1.48 \times 10^{-5} \\ \eta_{\text{layer 12}} &= 2 \times 10^{-5} \cdot 0.95^{0} = 2.00 \times 10^{-5} \\ \eta_{\text{task head}} &= 2 \times 10^{-5} \times 10 = 2.00 \times 10^{-4} \end{aligned}

Step 2: Compute learning rates at 25% of training (step 1,563)

At this point we have been past warmup for 1563−625=9381563 - 625 = 938 steps. The cosine decay progress fraction is:

p=9386250−625=9385625≈0.167p = \frac{938}{6250 - 625} = \frac{938}{5625} \approx 0.167

The cosine multiplier is:

m=0.5⋅(1+cos⁡(π⋅0.167))=0.5⋅(1+cos⁡(0.524))≈0.5⋅(1+0.866)=0.933m = 0.5 \cdot (1 + \cos(\pi \cdot 0.167)) = 0.5 \cdot (1 + \cos(0.524)) \approx 0.5 \cdot (1 + 0.866) = 0.933

The task head learning rate at this step is therefore 2.00×10−4×0.933=1.87×10−42.00 \times 10^{-4} \times 0.933 = 1.87 \times 10^{-4}. The embedding rate is 1.07×10−5×0.933=9.98×10−61.07 \times 10^{-5} \times 0.933 = 9.98 \times 10^{-6}.

Step 3: Compute learning rates at 75% of training (step 4,688)

The cosine decay progress fraction is now:

p=4688−6255625=40635625≈0.722p = \frac{4688 - 625}{5625} = \frac{4063}{5625} \approx 0.722

The cosine multiplier drops substantially:

m=0.5⋅(1+cos⁡(π⋅0.722))=0.5⋅(1+cos⁡(2.269))≈0.5⋅(1−0.637)=0.181m = 0.5 \cdot (1 + \cos(\pi \cdot 0.722)) = 0.5 \cdot (1 + \cos(2.269)) \approx 0.5 \cdot (1 - 0.637) = 0.181

The task head is now at 2.00×10−4×0.181=3.62×10−52.00 \times 10^{-4} \times 0.181 = 3.62 \times 10^{-5}. The embedding rate has dropped to 1.07×10−5×0.181=1.94×10−61.07 \times 10^{-5} \times 0.181 = 1.94 \times 10^{-6}, a tiny fraction of its peak value.

Key observation: By 75% of training, the cosine schedule has reduced all learning rates to about 18% of their peak values. The relative hierarchy between layers is preserved (the task head is always 10x the backbone top rate, which is always higher than the embedding rate), but all rates are scaling down together. This is the combined effect we want: LLRD sets the relative rates at each layer, while the decay schedule controls the global scale over time.