Fine-tuning Learning Rates: LLRD, Warmup & Decay Strategies

When you fine-tune a pre-trained language model, choosing the right learning rate is more important than when training from scratch. A learning rate that works well for pre-training can destroy months of computational effort in minutes of fine-tuning. Conversely, a learning rate that is too small might never adapt the model to your task. The challenge lies in finding the narrow band of learning rates that let real adaptation without erasing the knowledge the model has already acquired.

Think of the pre-trained model as a craftsperson who has spent years building an advanced set of skills. When you hire them for a new project, you want them to apply their expertise to the new context, not forget everything they know and start from scratch. The learning rate is the intensity of their retraining. Too intense, and they forget their foundational skills. Too gentle, and they never adapt to the new role. You need a calibrated approach that respects their existing expertise while allowing specialization.

This analogy captures something real about the mathematics. Pre-trained weights stand for a particular region of parameter space that took enormous computational resources to reach. This region is not just any local minimum; it is a high-quality one that encodes linguistic structure, semantic relationships, and world knowledge accumulated over billions of training tokens. When we fine-tune, we are not searching for a good solution from scratch. We are making careful adjustments around an already-excellent solution. The learning rate controls exactly how careful or aggressive those adjustments are.

The challenge is compounded by the heterogeneous structure of transformer models. A 12-layer BERT model is not a single monolithic entity but a stack of specialized subsystems: word embeddings that encode vocabulary semantics, early transformer layers that capture syntax and local patterns, middle layers that build richer contextual representations, and upper layers that prepare outputs for the specific pre-training task. Each of these subsystems has a different relationship to the fine-tuning objective. Some need minimal adjustment, others need substantial reconfiguration, and the task head needs to be trained almost from scratch. A single learning rate cannot optimally serve all of these needs simultaneously.

The strategies in this chapter address this heterogeneity systematically. Discriminative fine-tuning and layer-wise learning rate decay assign different rates to different layers, respecting the hierarchical structure of learned representations. Warmup schedules protect the model during the dangerous early phase of adaptation. Decay schedules guide convergence toward a stable final solution. Together, these techniques provide a way to manage the fine-tuning objective with care.

As we discussed in the previous chapter on catastrophic forgetting, fine-tuning involves a basic tension. You want the model to learn task-specific patterns, but you do not want to overwrite the rich linguistic knowledge acquired during pre-training. Learning rates sit at the heart of this tension because they directly control how much each gradient update changes the model's parameters.

Understanding why fine-tuning is qualitatively different from pre-training requires thinking carefully about the optimization objective. During pre-training, the model starts from random weights and explores the loss surface from a completely uninformed starting point. The optimizer has to make large moves to escape the random initialization and find real structure in the data. There is nothing to lose, because the starting weights encode no knowledge worth preserving. Large learning rates are not just safe at this stage; they are necessary for making rapid progress across billions of training tokens.

Fine-tuning presents a fundamentally different scenario. You are starting from a carefully optimized point that already captures real language structure. The pre-trained weights stand for a local minimum that took enormous computational resources to reach, and this minimum encodes useful knowledge about syntax and semantics as well as world knowledge. Large updates can push parameters far from this optimum, destroying useful representations that took the original training process millions of gradient steps to construct. The loss surface near a high-quality pre-trained minimum is typically much more sensitive than it is near random initialization: small perturbations in the wrong direction can cascade through the network and destabilize representations that downstream layers depend on.

During pre-training, models typically use learning rates between $10^{-4}$ and $10^{-3}$ with the Adam optimizer. These relatively large rates help the model explore the parameter space efficiently when learning from scratch. But fine-tuning requires a reduction of roughly one to two orders of magnitude, with typical values in the range of $10^{-5}$ to $5 \times 10^{-5}$ for BERT-style models. This reduction shows the different nature of the optimization problem: rather than exploring a large parameter space, we are making careful adjustments around an already-good solution.

Even this simple observation raises deeper questions. Not all parts of the model are equally sensitive to update magnitude. Not all parts need to change by the same amount. And the optimal strategy depends on characteristics of both the pre-trained model and the fine-tuning task. The strategies we explore in this chapter emerged from attempts to answer these questions systematically. Rather than treating fine-tuning as simply "training with a smaller learning rate," we have developed fine-grained approaches that consider which parts of the model need to change and by how much.

• Should all layers receive the same learning rate?
• Should the rate start small and grow, or start larger and decay?
• How do these choices interact with different model architectures?

The answers depend on the hierarchical structure of what transformers learn, which motivates our first major technique.

Discriminative fine-tuning assigns different learning rates to different parts of the model based on the intuition that not all parameters should adapt equally. This approach recognizes that a pre-trained language model is not a monolithic entity but rather a hierarchy of representations, each capturing different levels of linguistic abstraction. Think of it as applying a sliding scale of carefulness: the parts of the model that encode universal, broadly applicable knowledge are treated with more caution, while the parts that are task-specific receive more aggressive updates.

The key insight is that neural network layers do not all play the same role in the learned representation hierarchy. Deep networks develop a natural architecture of specialization that mirrors how information is progressively abstracted. In transformer models, this hierarchy is particularly well-studied because probing experiments have systematically mapped what each layer learns. Lower layers capture general linguistic patterns such as tokenization boundaries, basic syntax, and basic semantic relationships. These patterns apply broadly across virtually any language task you might encounter. Higher layers capture more task-specific features that build on these foundations, developing representations increasingly tailored to the pre-training objective. When fine-tuning for a specific task, you want to:

1. Preserve lower-layer representations: These capture transferable knowledge that applies across tasks. The basic understanding of language structure, word relationships, and syntactic patterns learned in these layers will be useful regardless of whether you are doing sentiment analysis, question answering, or named entity recognition.
2. Adapt higher-layer representations: These need to specialize for your specific task. The upper layers, which were optimized for masked language modeling or next-token prediction, need to reconfigure their representations to support your new objective.
3. Train the new task head aggressively: This is entirely task-specific and needs full adaptation. Unlike the pre-trained backbone, the task head starts from random initialization and has no prior knowledge to preserve.

This suggests a natural structure: assign progressively larger learning rates as you move up through the network. Lower layers receive small learning rates that allow only gentle adjustments, nudging the representations slightly while preserving their overall structure. Higher layers receive larger rates that let substantial adaptation to the new task. The task-specific head receives the largest rate of all, letting it to rapidly learn the mapping from pre-trained representations to task outputs. The resulting gradient of adaptation rates mirrors the gradient of specialization that the network itself embodies.

To implement discriminative fine-tuning, we need a principled way to assign learning rates across layers. The most common approach uses geometric decay, where each step down the network reduces the learning rate by a constant factor. This creates a smooth gradient of adaptation rates that respects the hierarchical nature of the model's representations.

We want to compute a learning rate for each layer that decreases as we move toward the bottom of the network. Let's denote the base learning rate as $\eta$ and define a decay factor $\xi$ (typically between 0.9 and 0.95). For a model with $L$ layers, layer $l$ receives learning rate:

where:

• $\eta_l$: the learning rate assigned to layer $l$ (the value we compute for each layer)
• $\eta$: the base learning rate assigned to the top layer (typically the highest rate in the backbone)
• $\xi$: the multiplicative decay factor (usually 0.9 to 0.95), which scales down the rate for each step down the network
• $L$: the total number of layers in the model
• $l$: the index of the current layer, where $l=L$ is the top layer, $l=1$ is the bottom transformer layer, and embeddings are treated as $l=0$

Why does this formula make sense? Notice that the exponent $L-l$ is exactly the layer's distance from the top of the network. The top layer ($l = L$) has a distance of 0, so $\xi^0=1$, receiving the full base rate $\eta$. Each subsequent layer down increases the distance by 1, compounding the decay factor $\xi$. This compounding effect means that the learning rate does not just decrease linearly as we move down the network; it decreases geometrically, with each layer receiving a fixed fraction of the rate assigned to the layer above it.

Geometric decay ensures that the relative difference between adjacent layers remains constant throughout the network. If layer 10 receives 95% of the rate of layer 11, then layer 5 also receives 95% of the rate of layer 6. This proportional relationship ensures that the adaptation dynamics remain consistent regardless of which part of the network we examine, which is an elegant property that a linear decay scheme would not give.

For a 12-layer BERT model with $\eta = 2 \times 10^{-5}$ and $\xi = 0.95$:

This creates a smooth gradient of adaptation rates throughout the network. Notice that even the bottom layer still receives a real learning rate, roughly half of the top layer's rate. This shows the philosophy that all layers should be allowed to adapt, just to different degrees. The embeddings receive the lowest rate because they encode basic vocabulary semantics that should remain maximally stable across different downstream tasks. If the embedding representations shift sharply, all subsequent layers must relearn how to interpret them, potentially cascading disruption throughout the entire network.

Layer-wise learning rate decay (LLRD) is the most common implementation of discriminative fine-tuning for transformer models. It was popularized by the ULMFiT paper and has become standard practice for fine-tuning BERT and similar models. While the underlying mathematics are identical to discriminative fine-tuning, LLRD specifically stresses the layer-by-layer structure of modern transformer architectures and has been refined through extensive empirical study across dozens of NLP benchmarks.

Think of LLRD as a zoning ordinance for your network. Historical districts near the foundation of the city (the lower layers) face strict rules about what can be changed: minor renovations only, no demolition. Commercial zones at the top of the network have much more permissive rules, letting rapid development and transformation. The zoning shows not arbitrary preference but the actual character and function of each zone in the overall system.

What makes LLRD particularly effective for transformers is that the layer-by-layer structure maps directly onto the parameter group structure required by modern optimizers. You define one parameter group per layer and assign each group its own learning rate. The AdamW optimizer, which is the standard choice for transformer fine-tuning, handles per-group learning rates natively, so the implementation adds very little complexity over a single-learning-rate setup. The engineering cost is low, but the empirical benefits, particularly on smaller datasets, can be substantial.

LLRD works because transformer representations are hierarchical. Varying learning rates by layer is based on research into what each layer learns during pre-training. Research on probing tasks, where simple classifiers are trained on frozen layer representations, has systematically mapped out this hierarchy with remarkable precision:

• Embedding layers capture lexical semantics and basic word relationships. At this level, the model encodes what words mean in isolation and how they relate to other words with similar meanings. These representations are highly general and transfer well to virtually any downstream task. Damaging them would undermine the entire network.
• Lower transformer layers (1-4 in BERT) capture syntactic structure and local dependencies. These layers develop sensitivity to part-of-speech patterns, phrase boundaries, and local word order. They learn to distinguish nouns from verbs, identify subject-verb relationships, and recognize basic grammatical patterns. These structural properties are shared across all syntactically regular language tasks.
• Middle layers (5-8) develop richer semantic representations that integrate local syntactic information with broader contextual meaning. These layers begin to capture coreference, semantic roles, and the flow of meaning across sentences. They are more specialized than lower layers but still broadly useful.
• Upper layers (9-12) capture task-relevant features that are more specific to the pre-training objective. For a model trained with masked language modeling, these layers specialize in predicting missing words given context. This specialization is useful but also means these representations are most tightly coupled to the pre-training task and farthest from what any given fine-tuning task might need.

When fine-tuning for a downstream task, the upper layers typically need the most adaptation because they were optimized for the pre-training task rather than your specific objective. Lower layers, which capture more universal linguistic properties that any NLP task would benefit from, need less adjustment. Aggressive updates to lower layers risk disrupting the carefully learned syntactic and semantic foundations that the entire model relies upon.

For transformer models, we typically define parameter groups corresponding to:

1. Embedding layer: Token embeddings and position embeddings
2. Encoder/decoder layers: Each transformer block as a separate group
3. Task head: The newly added classification or regression layer

This visualization shows the learning rate hierarchy. The embeddings and lower layers remain relatively stable to preserve basic language understanding, while the learning rate increases exponentially toward the task head to facilitate rapid adaptation to the new problem. The smooth progression ensures that no single layer boundary experiences a large discontinuity in adaptation rates, which could create optimization instabilities where one layer is changing rapidly relative to its neighbors.

The decay factor $\xi$ controls how aggressively learning rates differ between layers, and choosing the right value requires balancing two competing concerns. A decay factor too close to 1.0 gives insufficient differentiation between layers, losing the benefits of discriminative fine-tuning. A decay factor too small may prevent lower layers from adapting at all, which can be problematic if your task requires the model to learn new low-level patterns not well-represented in the pre-training data. The optimal value depends on your task and how much the pre-trained representations need to change:

• $\xi = 1.0$: No decay; all layers receive the same learning rate (standard fine-tuning). This is equivalent to ignoring the hierarchical structure of the network and may be appropriate when you have abundant data and the task differs substantially from pre-training.
• $\xi = 0.95$: Gentle decay; good default for most NLP tasks. This creates roughly a 2:1 ratio between the top and bottom layer learning rates. This gives differentiation while still allowing all layers to adapt.
• $\xi = 0.9$: Moderate decay; useful when lower layers should change minimally. This creates approximately a 3:1 ratio and is appropriate when you have limited data or when the pre-training domain closely matches your target task.
• $\xi = 0.8$: Aggressive decay; for tasks very similar to pre-training. This can create ratios of 5:1 or greater and in effect "freezes" the lower layers by giving them very small learning rates, putting all adaptation in the upper layers.

Empirically, values between 0.9 and 0.95 work well for most transformer fine-tuning scenarios. When in doubt, 0.95 gives a safe starting point that rarely hurts performance and often helps. For domain adaptation tasks where the target text differs substantially from the pre-training corpus, a slightly larger factor (closer to 1.0) may be warranted because even the lower layers need to adjust to the new vocabulary distribution and stylistic patterns.

Warmup refers to starting training with a very small learning rate and gradually increasing it to the target value over a specified number of steps. This technique is especially important for fine-tuning. While warmup was originally motivated by considerations specific to the Adam optimizer and very deep networks, its benefits during fine-tuning arise from slightly different concerns: the unique vulnerability of pre-trained representations during the earliest phase of adaptation.

Think of warmup as the first few minutes of a workout. Jumping straight into maximum-intensity exercise risks injury precisely because the body is not yet prepared for peak exertion. Similarly, starting fine-tuning at the full target learning rate risks "injuring" the pre-trained representations before the optimizer has had a chance to build reliable gradient estimates. Warmup gives a controlled ramp-up period where the optimizer calibrates itself before making full-size updates.

The warmup phase serves several distinct but complementary purposes that interact to stabilize the beginning of fine-tuning. Understanding each of these purposes helps you reason about how long warmup should last and what can go wrong when it is omitted. In practice, models fine-tuned without warmup often show erratic early loss behavior, sometimes followed by eventual recovery, but frequently exhibiting worse final performance than runs that included warmup from the start.

At the start of fine-tuning, several factors make large learning rates dangerous:

1. Task head initialization: The newly added classification head is randomly initialized while the pre-trained backbone is well-optimized. This creates an asymmetry in the network: the backbone produces real representations, but the randomly initialized head turns these into in effect random predictions. The gradients flowing back from this random head can be large and poorly directed. Large initial updates from the random head can destabilize the entire network, propagating noise deep into the pre-trained layers. Warmup allows the task head to "catch up" to the backbone before the learning rate becomes large enough to cause damage.

2. Gradient variance: Early mini-batches may not be representative of the full dataset, leading to noisy gradient estimates. The first few batches might happen to contain unusual examples or reflect sampling artifacts rather than the true data distribution. Small learning rates during warmup prevent over-reacting to this noise, giving the optimizer time to aggregate gradient information from more representative samples.

3. Adam statistics: As we discussed in the Adam optimizer chapter, Adam maintains running estimates of gradient moments, specifically the first moment (mean) and second moment (variance) of the gradients. These estimates are unreliable early in training because they are initialized to zero and require many updates to become accurate. The bias correction terms in Adam help, but they do not fully compensate for the lack of historical information. Warmup gives these moment estimates time to stabilize before making large parameter updates that rely on their accuracy.

The interaction between these three factors is multiplicative in the worst case. A poorly calibrated task head generating large, noisy gradients, evaluated against a dataset subset that is not representative, feeding into an Adam optimizer with unreliable moment estimates, could easily result in a catastrophic early update that erases key pre-trained representations. Warmup breaks this chain by so that no single factor has time to fully develop before the update magnitude is brought under control.

The most common warmup schedule increases the learning rate linearly from 0 to the target value over the warmup period. This approach is simple to implement, easy to understand, and effective in practice.

We want to compute a learning rate that starts at zero on the first step and reaches the target value exactly at the end of the warmup period. At step $t$ during warmup, the learning rate is:

where:

• $\eta_t$: the learning rate at step $t$ (what we compute and apply to the optimizer)
• $t$: the current training step, starting at 0
• $T_{\text{warmup}}$: the total number of warmup steps (defining the duration of the ramp-up phase)
• $\eta_{\text{target}}$: the target learning rate to reach at the end of warmup (the peak rate)

Why does this formula make sense? The ratio $\frac{t}{T_{\text{warmup}}}$ is the fraction of the warmup phase completed, which grows linearly from 0 to 1 as $t$ increases from 0 to $T_{\text{warmup}}$. Multiplying this fraction by $\eta_{\text{target}}$ scales the learning rate linearly from 0 up to the target value. The linear relationship means that the learning rate increases by a constant amount at each step. This gives a predictable and smooth ramp-up that is easy to reason about and debug.

For fine-tuning, warmup typically spans 5-10% of total training steps. For a fine-tuning run of 10,000 steps, you might use 500-1,000 warmup steps. This duration allows Adam's moment estimates to stabilize and the task head to begin learning patterns without wasting training time in an overly cautious exploration period.

A more aggressive warmup strategy combines warmup with gradual unfreezing. Instead of training all layers from the start with small learning rates, you begin by training only the task head, then progressively unfreeze layers from top to bottom as training proceeds:

1. Epoch 1: Train only the task head
2. Epoch 2: Unfreeze top transformer layers
3. Epoch 3: Unfreeze middle layers
4. Epoch 4+: Train all layers

This approach, introduced in ULMFiT, gives an extreme form of "warmup" where lower layers literally receive zero gradient until later in training. The philosophy is that by the time lower layers are unfrozen, the task head and upper layers have already adapted substantially. This gives more stable and informative gradients. The gradients that flow into the lower layers at the point of unfreezing are no longer the confused, high-variance signals of a randomly initialized network, but the more structured, task-relevant signals of a partially adapted one.

While effective, gradual unfreezing requires more epochs and careful tuning of the unfreezing schedule. It has fallen somewhat out of favor for transformer fine-tuning because LLRD reaches similar goals with less complexity and in fewer training epochs. However, it remains a useful technique when maximum caution is warranted, particularly in low-data regimes where even a brief period of high-variance gradient updates could cause irreversible damage to the pre-trained representations. If you find that standard LLRD with warmup still results in unstable early training, gradual unfreezing is worth adding to your toolkit.

After warmup completes, learning rate decay schedules reduce the rate throughout training. This helps the model converge to a stable minimum rather than oscillating around it. The intuition is that large learning rates are useful early in training when the model needs to make significant adjustments, but become counterproductive later when the model is close to a good solution and only needs fine adjustments.

Think of decay as the difference between making broad brushstrokes and fine detail work when painting. Early in the process, you want big, confident strokes that establish the overall composition. As the painting develops, you switch to finer tools for precise work. A painter who keeps using the broad brush throughout the entire process will never reach the detail that the final piece requires. Similarly, a constant learning rate throughout fine-tuning tends to produce models that have adapted well on average but oscillate around the optimal parameters rather than converging to them precisely.

The choice of decay schedule shapes the final convergence and the entire trajectory of learning. Different schedules allocate the "learning budget" differently across the training run: some front-load it (maintaining higher rates early), others distribute it more uniformly, and others back-load it (maintaining higher rates in the middle of training). These differences can have real effects on final performance, particularly for tasks where the model needs to make large initial adjustments to the fine-tuning task before settling into precise calibration.

The simplest decay schedule reduces the learning rate linearly to zero (or some minimum value) by the end of training. This approach gives a steady, predictable reduction in learning rate that is easy to reason about and implement.

We want to compute a learning rate that starts at the target value immediately after warmup ends and decreases uniformly to zero at the final training step. After warmup ends at step $T_{\text{warmup}}$, the learning rate at step $t$ is:

where:

• $\eta_t$: the learning rate at step $t$
• $\eta_{\text{target}}$: the learning rate at the start of decay (equal to the peak warmup value)
• $t$: the current training step
• $T_{\text{warmup}}$: the step number where decay begins (end of warmup)
• $T_{\text{total}}$: the total number of training steps

Why does this formula make sense? The fraction $\frac{t - T_{\text{warmup}}}{T_{\text{total}} - T_{\text{warmup}}}$ is the progress through the decay phase, ranging from 0 at the start of decay to 1 at the final step. Subtracting this from 1 yields the remaining proportion of the learning rate. This creates a steady linear decline from $\eta_{\text{target}}$ down to 0. At each step during the decay phase, the learning rate decreases by exactly the same absolute amount. This keeps uniform reduction throughout training.

This is the default schedule for most BERT fine-tuning recipes because it strikes a reasonable balance between simplicity and effectiveness. The linear nature makes it trivial to predict the learning rate at any step, which is helpful for debugging and monitoring training progress.

Cosine decay gives a smoother transition that decays slowly at first, accelerates in the middle, and slows again near the end. This schedule has become increasingly popular because it maintains higher learning rates for longer during the middle of training, giving the optimizer more opportunity to explore the loss surface before settling into a final minimum.

We want to compute a smoothly decaying learning rate using the cosine function's natural properties. After warmup, the learning rate at step $t$ is:

where:

• $\eta_t$: the learning rate at step $t$
• $\eta_{\text{min}}$: the minimum learning rate to reach at the end of training (often 0)
• $\eta_{\text{target}}$: the target learning rate at the start of decay (peak warmup value)
• $t$: the current training step
• $T_{\text{warmup}}$: the step number where decay begins
• $T_{\text{total}}$: the total number of training steps

Why does this formula produce an S-shaped decay? We can trace the mathematical transformations step by step to see the mechanism clearly:

The cosine transform is what creates the S-curve behavior. The derivative of $\cos(\theta)$ is $-\sin(\theta)$, which equals zero at $\theta = 0$ and $\theta = \pi$ (the endpoints) but reaches its maximum magnitude at $\theta = \pi/2$ (the midpoint). This means the cosine schedule changes slowly at the beginning and end, and rapidly in the middle, which is exactly the behavior we want: the model maintains higher learning rates early to explore broadly, transitions through a rapid reduction phase in the middle, and then approaches its final value gently without overshooting.

While all three schedules eventually reach zero, they allocate learning rate differently throughout training. Cosine decay maintains higher learning rates for longer in the middle of training, potentially letting the model to escape local minima more effectively than linear decay. Polynomial decay with power 2 drops quickly at first and then more slowly, front-loading the high learning rate period. The choice between these schedules is a trade-off between exploration and convergence speed. Linear decay is the most predictable and easiest to reason about; cosine decay is often slightly better in practice due to its gentler terminal convergence; polynomial decay gives the most control through its exponent parameter.

The choice of decay schedule depends on your training dynamics and the specific characteristics of your task:

• Linear decay: Simple and effective; good default for short fine-tuning runs. The predictable reduction makes it easy to reason about training dynamics and diagnose issues such as learning rate being too high or too low.
• Cosine decay: Often slightly better performance; allows more learning early and late in training. The gentler decay at both ends can help the model both explore better initially and converge more precisely at the end. It is the recommended default when you have sufficient compute for even a modest hyperparameter search.
• Polynomial decay: Useful when you want to front-load learning (higher powers) or extend it (lower powers). The exponent parameter gives direct control over the shape of the decay curve, making this a flexible option for tasks with unusual training dynamics.

Empirically, cosine and linear decay perform similarly for most fine-tuning tasks, with cosine sometimes giving a small but consistent improvement. The difference is often smaller than the effect of other hyperparameters like the base learning rate or batch size, so either choice is reasonable for most applications. When in doubt, cosine decay is the better default because its smooth behavior near both endpoints reduces the risk of numerical instability.

To make the abstract mechanics concrete, let's trace through a specific scenario: fine-tuning BERT-base-uncased for binary sentiment classification. We will compute the exact learning rates at representative steps throughout the training run and show how the combined schedule behaves in practice.

Setup: 8 epochs over a dataset of 25,000 training examples with batch size 32. This gives us approximately $\frac{25000 \times 8}{32} = 6{,}250$ training steps total. We use a 10% warmup (625 steps), base learning rate $\eta = 2 \times 10^{-5}$, LLRD decay factor $\xi = 0.95$, and cosine decay.

Step 1: Compute the per-layer peak learning rates (end of warmup)

At the end of warmup, each layer reaches its assigned peak rate. Using the LLRD formula $\eta_l = \eta \cdot \xi^{L-l}$ with $L=12$, $\eta = 2 \times 10^{-5}$, $\xi = 0.95$:

Step 2: Compute learning rates at 25% of training (step 1,563)

At this point we have been past warmup for $1563 - 625 = 938$ steps. The cosine decay progress fraction is:

The cosine multiplier is:

The task head learning rate at this step is therefore $2.00 \times 10^{-4} \times 0.933 = 1.87 \times 10^{-4}$. The embedding rate is $1.07 \times 10^{-5} \times 0.933 = 9.98 \times 10^{-6}$.

Step 3: Compute learning rates at 75% of training (step 4,688)

The cosine decay progress fraction is now:

The cosine multiplier drops substantially:

The task head is now at $2.00 \times 10^{-4} \times 0.181 = 3.62 \times 10^{-5}$. The embedding rate has dropped to $1.07 \times 10^{-5} \times 0.181 = 1.94 \times 10^{-6}$, a tiny fraction of its peak value.

Key observation: By 75% of training, the cosine schedule has reduced all learning rates to about 18% of their peak values. The relative hierarchy between layers is preserved (the task head is always 10x the backbone top rate, which is always higher than the embedding rate), but all rates are scaling down together. This is the combined effect we want: LLRD sets the relative rates at each layer, while the decay schedule controls the global scale over time.

Let's combine these techniques into a complete fine-tuning setup. We will create a training configuration with layer-wise learning rates and a warmup-plus-cosine schedule. This implementation shows how the individual components work together in practice rather than as a collection of independent tricks.

Here is how these components integrate into a complete fine-tuning loop. Notice how the optimizer step, gradient clipping, and scheduler step must occur in the correct order: compute gradients, clip them, apply the optimizer update, then advance the scheduler. Reversing any of these steps would break the intended behavior.

These learning rate strategies have been refined through extensive empirical study across many NLP benchmarks, and certain practical patterns have emerged as reliable defaults. Understanding these defaults and why they differ across model sizes helps you adapt them intelligently to novel scenarios.

For BERT-base and similar models (110M parameters), the relatively modest size means that even aggressive learning rates rarely cause catastrophic failure; the model has enough capacity to recover from moderate perturbations. Recommended settings:

• Base learning rate: $2 \times 10^{-5}$ to $5 \times 10^{-5}$
• LLRD decay: 0.9 to 0.95
• Warmup: 6-10% of training steps
• Task head learning rate: 10x base rate

For larger models (GPT-2, RoBERTa-large, 340M+ parameters), the increased depth means that gradient signals must travel through more layers, amplifying any instabilities in the optimization dynamics. More conservative settings are needed:

• Base learning rate: $10^{-5}$ to $2 \times 10^{-5}$ (lower due to more layers)
• LLRD decay: 0.85 to 0.95
• Warmup: 10% of training steps
• Consider lower weight decay (0.001 instead of 0.01)

For very large models (1B+ parameters), the sensitivity to learning rate is considerably higher, and even small overestimation of the learning rate can trigger training instability. Parameter-efficient methods like LoRA are often preferable for this regime, but when full fine-tuning is necessary:

• Base learning rate: $5 \times 10^{-6}$ to $10^{-5}$
• More aggressive LLRD decay: 0.8 to 0.9
• Longer warmup: 10-15% of training steps
• Often better to use parameter-efficient methods like LoRA, which we'll cover in upcoming chapters

General diagnostic principles that apply across all model sizes:

• Start with the lower end of learning rate ranges and increase if learning is too slow
• Monitor both training loss and validation metrics; overfitting indicates the rate may be too high
• If loss spikes early, increase warmup steps or decrease initial learning rate
• If loss plateaus too early, try a higher learning rate or different decay schedule
• Loss that decreases smoothly in training but stagnates on validation often indicates the decay schedule is letting the model to overfit; try reducing the warmup ratio to allow more steps at full rate, or switch to cosine decay if you are using linear

These learning rate strategies have materially improved fine-tuning results across many NLP tasks, but they come with important caveats that are easy to overlook when applying them as recipes without understanding their assumptions.

The effectiveness of layer-wise learning rate decay depends fundamentally on the assumption that lower layers capture more general, transferable features while upper layers capture more task-specific ones. This assumption holds strongly for models pre-trained with masked language modeling or causal language modeling on diverse text corpora. But it may not apply universally. For tasks that differ substantially from the pre-training distribution, such as fine-tuning a general English model for a specialized scientific domain with unusual notation and vocabulary, the lower layers may need as much adaptation as the upper ones. In domain adaptation scenarios, practitioners have sometimes found that uniform learning rates perform equally well or better than LLRD, suggesting that the hierarchy of "what needs to change" can invert for out-of-distribution tasks. The key insight is that LLRD is not a universal law but a heuristic grounded in a specific empirical pattern about what transformers learn, and that heuristic can fail when the conditions change.

The interaction between these techniques and other hyperparameters creates a complex, high-dimensional optimization problem. A learning rate that works perfectly with dropout rate 0.1 might be suboptimal with dropout rate 0.3. A warmup ratio of 10% that works well with batch size 32 might be too short with batch size 8, where each step covers fewer examples and gradient estimates are noisier. The LLRD decay factor that reaches the best results with 3 fine-tuning epochs might not be optimal for 10 epochs, because more epochs mean more opportunities for lower-layer drift that the decay factor might need to compensate for. This interdependence means that a complete hyperparameter search, while expensive, often yields better results than following any single recipe. A grid search over the base learning rate and decay factor, with a warmup ratio fixed at 10%, is a common approach when compute allows.

The computational overhead of the combined schedule is minimal: it adds negligible time to the forward-backward pass. But the engineering complexity is not trivial for practitioners new to the field. Maintaining separate learning rates for dozens of parameter groups, coordinating warmup schedules across distributed training setups, and debugging issues where the wrong parameter group received the wrong rate all add development time and potential for subtle bugs. For rapid prototyping or initial experiments, uniform learning rates with basic warmup often give 90% of the benefit with much less engineering overhead. The Hugging Face Trainer class and similar high-level abstractions have helped by implementing these strategies as defaults, reducing the barrier to using them correctly.

These techniques also do not address all fine-tuning challenges. Catastrophic forgetting can still occur even with optimal learning rates, particularly on small datasets or with many training epochs. Methods like learning rate manipulation cannot prevent the model from overfitting if the fine-tuning dataset is too small relative to the number of trainable parameters. As we explore in upcoming chapters on parameter-efficient fine-tuning, methods like LoRA offer complementary approaches that limit the number of parameters updated rather than just controlling how fast each parameter changes. In many practical scenarios, combining LLRD and warmup with LoRA or similar techniques gives the best of both worlds: carefully controlled adaptation dynamics applied to a small, targeted set of parameters.

Finally, it is worth acknowledging that these techniques were largely developed and validated on BERT-style encoder models fine-tuned for classification tasks. Their applicability to decoder-only language models fine-tuned with instruction tuning or RLHF, to multi-modal models, or to architectures with materially different layer structures may require re-investigation. The field is actively studying these questions, and the recommendations in this chapter should be treated as strong empirical starting points rather than definitive theoretical conclusions.

Fine-tuning learning rates require careful consideration that goes beyond simply selecting a single value. The strategies covered in this chapter address different aspects of the fine-tuning challenge, and together they form a principled framework for protecting pre-trained knowledge while letting real task-specific adaptation.

Layer-wise learning rate decay assigns different learning rates to different layers based on the principle that lower layers capture general features (requiring less adaptation) while upper layers capture task-specific features (requiring more adaptation). Using a decay factor between 0.9 and 0.95 across layers typically gives a good balance between differentiation and flexibility, with the geometric decay formula $\eta_l = \eta \cdot \xi^{L-l}$ so that the ratio between adjacent layers remains constant throughout the network.

Learning rate warmup gradually increases the learning rate from zero during the first 5-10% of training. This stabilizes optimization by letting Adam's moment estimates to accumulate, preventing the randomly initialized task head from sending large, noisy gradients into the pre-trained backbone, and giving early mini-batch gradient estimates time to become more representative of the full data distribution.

Learning rate decay reduces the rate throughout training after warmup, helping the model converge to a stable minimum rather than oscillating around it. Cosine decay is often the preferred choice because its smooth behavior near both the start and end of the decay phase reduces the risk of instability during transitions. Linear decay gives a simpler alternative that works equally well in most cases.

Combining these techniques creates a reliable fine-tuning setup: warmup followed by decay sets the global learning rate trajectory, while LLRD distributes that trajectory differentially across the network hierarchy. The task head typically receives 10x the base learning rate since it starts from random initialization and needs aggressive training from the outset. Understanding why each component works, not just how to configure it, allows you to adapt these strategies intelligently when facing unusual tasks, architectures, or data conditions that fall outside the standard BERT fine-tuning regime.

These strategies stand for one dimension of efficient fine-tuning. In the next chapter, we will examine data efficiency: how to get the most out of limited labeled data when adapting pre-trained models to new tasks.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about fine-tuning learning rate strategies.