Fine-tuning Learning Rates › Discriminative Fine-tuning

Discriminative fine-tuning assigns different learning rates to different parts of the model based on the intuition that not all parameters should adapt equally. This approach recognizes that a pre-trained language model is not a monolithic entity but rather a hierarchy of representations, each capturing different levels of linguistic abstraction. Think of it as applying a sliding scale of carefulness: the parts of the model that encode universal, broadly applicable knowledge are treated with more caution, while the parts that are task-specific receive more aggressive updates.

The key insight is that neural network layers do not all play the same role in the learned representation hierarchy. Deep networks develop a natural architecture of specialization that mirrors how information is progressively abstracted. In transformer models, this hierarchy is particularly well-studied because probing experiments have systematically mapped what each layer learns. Lower layers capture general linguistic patterns such as tokenization boundaries, basic syntax, and basic semantic relationships. These patterns apply broadly across virtually any language task you might encounter. Higher layers capture more task-specific features that build on these foundations, developing representations increasingly tailored to the pre-training objective. When fine-tuning for a specific task, you want to:

1. Preserve lower-layer representations: These capture transferable knowledge that applies across tasks. The basic understanding of language structure, word relationships, and syntactic patterns learned in these layers will be useful regardless of whether you are doing sentiment analysis, question answering, or named entity recognition.
2. Adapt higher-layer representations: These need to specialize for your specific task. The upper layers, which were optimized for masked language modeling or next-token prediction, need to reconfigure their representations to support your new objective.
3. Train the new task head aggressively: This is entirely task-specific and needs full adaptation. Unlike the pre-trained backbone, the task head starts from random initialization and has no prior knowledge to preserve.

This suggests a natural structure: assign progressively larger learning rates as you move up through the network. Lower layers receive small learning rates that allow only gentle adjustments, nudging the representations slightly while preserving their overall structure. Higher layers receive larger rates that let substantial adaptation to the new task. The task-specific head receives the largest rate of all, letting it to rapidly learn the mapping from pre-trained representations to task outputs. The resulting gradient of adaptation rates mirrors the gradient of specialization that the network itself embodies.

To implement discriminative fine-tuning, we need a principled way to assign learning rates across layers. The most common approach uses geometric decay, where each step down the network reduces the learning rate by a constant factor. This creates a smooth gradient of adaptation rates that respects the hierarchical nature of the model's representations.

We want to compute a learning rate for each layer that decreases as we move toward the bottom of the network. Let's denote the base learning rate as $\eta$ and define a decay factor $\xi$ (typically between 0.9 and 0.95). For a model with $L$ layers, layer $l$ receives learning rate:

where:

• $\eta_l$: the learning rate assigned to layer $l$ (the value we compute for each layer)
• $\eta$: the base learning rate assigned to the top layer (typically the highest rate in the backbone)
• $\xi$: the multiplicative decay factor (usually 0.9 to 0.95), which scales down the rate for each step down the network
• $L$: the total number of layers in the model
• $l$: the index of the current layer, where $l=L$ is the top layer, $l=1$ is the bottom transformer layer, and embeddings are treated as $l=0$

Why does this formula make sense? Notice that the exponent $L-l$ is exactly the layer's distance from the top of the network. The top layer ($l = L$) has a distance of 0, so $\xi^0=1$, receiving the full base rate $\eta$. Each subsequent layer down increases the distance by 1, compounding the decay factor $\xi$. This compounding effect means that the learning rate does not just decrease linearly as we move down the network; it decreases geometrically, with each layer receiving a fixed fraction of the rate assigned to the layer above it.

Geometric decay ensures that the relative difference between adjacent layers remains constant throughout the network. If layer 10 receives 95% of the rate of layer 11, then layer 5 also receives 95% of the rate of layer 6. This proportional relationship ensures that the adaptation dynamics remain consistent regardless of which part of the network we examine, which is an elegant property that a linear decay scheme would not give.

For a 12-layer BERT model with $\eta = 2 \times 10^{-5}$ and $\xi = 0.95$:

This creates a smooth gradient of adaptation rates throughout the network. Notice that even the bottom layer still receives a real learning rate, roughly half of the top layer's rate. This shows the philosophy that all layers should be allowed to adapt, just to different degrees. The embeddings receive the lowest rate because they encode basic vocabulary semantics that should remain maximally stable across different downstream tasks. If the embedding representations shift sharply, all subsequent layers must relearn how to interpret them, potentially cascading disruption throughout the entire network.