Full Fine-tuning › Key Hyperparameters

Fine-tuning success depends on correctly setting several interconnected hyperparameters. Unlike pre-training, where large compute budgets allow for extensive hyperparameter searches, fine-tuning often requires getting these settings right with limited experimentation. Understanding the principles behind each hyperparameter helps practitioners make informed choices rather than relying solely on trial and error. The good news is that the community has converged on reliable default ranges that work well for most scenarios, so the search space is smaller than it might appear.

Think of the hyperparameter configuration as tuning a set of interdependent dials on a complex instrument. Each dial affects the others: the learning rate interacts with batch size, the warmup ratio interacts with the learning rate schedule, and the number of epochs interacts with regularization strength. The challenge is that these interactions are not always linear or predictable. A learning rate that works perfectly with one batch size may fail completely with another. Understanding the principles behind each interaction empowers you to reason about configurations rather than relying purely on grid search.

The key insight from the research literature is that most failures in fine-tuning trace back to one of three root causes: the learning rate is too high (destroying pre-trained representations), the number of epochs is too large (causing overfitting), or the warmup period is absent or too short (causing early instability). Mastering these three controls goes a long way toward reliable fine-tuning performance.

The learning rate is the most necessary hyperparameter in fine-tuning. It controls how much the weights change in response to computed gradients. A learning rate that is too high will destroy pre-trained representations, causing the model to forget what it learned during pre-training. A learning rate that is too low will result in insufficient adaptation to the new task. Finding the right balance requires understanding the unique constraints of fine-tuning compared to training from scratch.

For transformer models, fine-tuning learning rates typically fall in the range:

$$\text{lr}_{\text{fine-tune}} \in [10^{-5}, 5 \times 10^{-5}]$$

where:

• $\text{lr}_{\text{fine-tune}}$: the learning rate used during fine-tuning
• The range $[10^{-5}, 5 \times 10^{-5}]$: typical values that balance adaptation speed with preservation of pre-trained knowledge. The lower bound $10^{-5}$ is the minimum learning rate that still allows real adaptation, and the upper bound $5 \times 10^{-5}$ is the maximum rate before risking destruction of pre-trained representations

This narrow range shows a basic trade-off. Rates that are too high destroy pre-trained representations, while rates that are too low fail to adapt the model to new task patterns. The exponential notation ($10^{-5}$ means 0.00001) stresses how materially these rates differ from standard neural network training, where learning rates of 0.001 to 0.1 are common. The difference spans two to four orders of magnitude, showing that fine-tuning operates in a fundamentally different regime than training from random initialization.

The intuition behind this narrow range is that pre-trained models have already learned useful representations. We want to gently nudge these representations toward our specific task, not overwrite them entirely. Think of fine-tuning like adjusting a musical instrument: a violin that is slightly out of tune needs small, careful adjustments. Large, aggressive changes would destroy its calibration entirely. Similarly, a pre-trained model has found a good region of parameter space, and we want to explore nearby configurations rather than jumping to distant, potentially inferior locations. The mathematical relationship can be expressed as follows:

$$\text{lr}_{\text{fine-tune}} \approx \frac{\text{lr}_{\text{pre-train}}}{k}$$

where:

• $\text{lr}_{\text{pre-train}}$: learning rate during pre-training (typically $1 \times 10^{-4}$ to $1 \times 10^{-3}$)
• $\text{lr}_{\text{fine-tune}}$: learning rate during fine-tuning (typically $1 \times 10^{-5}$ to $5 \times 10^{-5}$)
• $k$: the reduction factor, typically ranging from 10 to 100, representing how much we scale down the pre-training rate
• The division by $k$: scales down the pre-training rate to preserve learned representations while still letting adaptation
• The approximation symbol $\approx$: acknowledges that this is a practical guideline rather than an exact formula, since optimal rates vary by task and dataset

This relationship captures the core principle of fine-tuning: we start from good parameters and make small adjustments. The reduction factor $k$ quantifies "how small" those adjustments should be, balancing preservation of pre-trained knowledge versus adaptation to the new task. The formula gives a principled way to derive a starting learning rate when you know the pre-training configuration, and it explains why fine-tuning rates fall into such a specific, narrow range.

The reduction factor $k$ typically ranges from 10 to 100. Fine-tuning learning rates are one to two orders of magnitude smaller than pre-training rates. This large reduction is necessary because pre-trained parameters have already converged to representations that capture general linguistic knowledge. These representations are not perfect for any specific task, but they give an excellent foundation. Large updates would destroy this foundation, requiring the model to rebuild representations it already possesses. The analogy is to surgery: a scalpel, not a sledgehammer, is the right tool when you are making precise adjustments to something that is mostly already working correctly.

Fine-tuning typically requires only 2-5 epochs over the training data. This stands in stark contrast to training from scratch, which might require hundreds of epochs. The pre-trained model starts from an excellent initialization, so it needs only minor adjustments to perform well on the new task. The model has already learned how to process language; it just needs to learn how to apply that knowledge to your specific problem. Think of it like a musician learning a new piece: an experienced concert pianist can learn a new piece in a few hours, while a beginner might need months of practice to reach the same level.

Training for too many epochs leads to overfitting, where the model memorizes the training data rather than learning generalizable patterns. Because fine-tuning datasets are typically much smaller than pre-training corpora, this risk is particularly acute. A pre-training corpus might contain billions of tokens, while a fine-tuning dataset might contain only thousands or tens of thousands of examples. This size disparity means the model can easily memorize the fine-tuning data, losing its ability to generalize to new examples.

The key diagnostic for choosing the right number of epochs is the validation loss curve. If validation loss is still decreasing at the end of your training, you may benefit from additional epochs. If validation loss starts increasing while training loss continues to decrease, you have begun overfitting and should stop earlier. Many practitioners use early stopping: they monitor validation performance and halt training when it stops improving, regardless of the preset number of epochs. The best checkpoint is then the one with the lowest validation loss, not necessarily the last one.

The right number of epochs also depends on dataset size. Very small datasets (a few hundred examples) may benefit from more epochs, up to 10 or even 20, because each epoch gives limited learning signal. Larger datasets (tens of thousands of examples) typically require fewer epochs, since each epoch covers more diverse examples. When in doubt, err on the side of fewer epochs and use validation loss to guide your decision.

Batch size affects both training dynamics and computational requirements. Larger batches give more stable gradient estimates but require more memory. In fine-tuning, common batch sizes range from 16 to 32 for models like BERT on consumer GPUs. These moderate batch sizes balance the need for reliable gradient estimates against memory constraints and training speed.

The interaction between batch size and learning rate is important: larger batches generally benefit from higher learning rates, a relationship known as the linear scaling rule. If you double the batch size, you can approximately double the learning rate to maintain similar training dynamics. This relationship arises because larger batches produce more stable gradient estimates, letting the optimizer to take larger steps without risking divergence. Conversely, if you are forced to use a smaller batch size due to memory constraints, you should also reduce the learning rate proportionally to avoid training instability.

Batch size also affects the nature of the optimization trajectory. Smaller batches introduce more stochasticity into gradient estimates, which can act as a form of regularization by preventing the model from settling into sharp minima. Research by Keskar et al. (2016) showed that large-batch training tends to converge to sharp minima with poor generalization, while small-batch training tends to converge to flatter minima with better generalization. For fine-tuning tasks where overfitting is a concern, using moderately small batches (16-32 rather than 256-512) can improve test performance even if it slows training.

Warmup gradually increases the learning rate from zero to its target value over the first portion of training. This prevents the randomly initialized classification head from causing large, destabilizing gradient updates at the start of training. The newly added classification head produces in effect random outputs initially, leading to large loss values and correspondingly large gradients. If applied at the full learning rate, these gradients could corrupt the carefully learned backbone representations.

A typical warmup ratio is 0.06 to 0.1, meaning the learning rate ramps up linearly for the first 6-10% of total training steps. This gradual increase gives the classification head time to develop basic competence before the full learning rate is applied to the backbone. The warmup schedule can be expressed mathematically as follows:

$$\text{lr}(t) = \begin{cases} \text{lr}_{\text{max}} \times \frac{t}{T_{\text{warmup}}} & \text{if } t < T_{\text{warmup}} \\ \text{lr}_{\text{max}} & \text{if } t \geq T_{\text{warmup}} \end{cases}$$

where:

• $\text{lr}(t)$: the learning rate at training step $t$
• $\text{lr}_{\text{max}}$: the target maximum learning rate (e.g., $2 \times 10^{-5}$)
• $t$: the current training step
• $T_{\text{warmup}}$: the number of warmup steps
• The ratio $\frac{t}{T_{\text{warmup}}}$: the linear interpolation factor, starting at 0 and reaching 1 at the end of warmup

This piecewise function defines two phases: during warmup ($t < T_{\text{warmup}}$), the learning rate increases linearly from 0 to $\text{lr}_{\text{max}}$, and after warmup ($t \geq T_{\text{warmup}}$), the learning rate stays constant at $\text{lr}_{\text{max}}$ or begins following a decay schedule if configured. The piecewise structure shows the distinct needs of early versus later training: caution initially, then full engagement once the model has stabilized.

The function ensures a smooth ramp-up. At $t = 0$:

$$\text{lr}(0) = \text{lr}_{\text{max}} \times \frac{0}{T_{\text{warmup}}} = 0$$

At $t = T_{\text{warmup}}$:

$$\begin{aligned} \text{lr}(T_{\text{warmup}}) &= \text{lr}_{\text{max}} \times \frac{T_{\text{warmup}}}{T_{\text{warmup}}} && \text{(substitute)} \\ &= \text{lr}_{\text{max}} \times 1 && \text{(simplify fraction)} \\ &= \text{lr}_{\text{max}} && \text{(multiply)} \end{aligned}$$

The linear interpolation prevents sudden jumps in the learning rate, which is necessary because the randomly initialized classification head produces large, unstable gradients at the start of training. By ramping up gradually, we give the head time to learn reasonable outputs before applying full learning rate updates to the pre-trained backbone. The head moves from random predictions toward sensible outputs over the warmup period, and by the time the full learning rate engages, its gradients reflect real learning signals rather than noise from random initialization.

Warmup matters because without this gradual ramp-up, the random classification head would compute meaningless losses on the first few batches, generating enormous gradients that could corrupt the carefully pre-trained backbone weights. Warmup gives the head several hundred steps to develop basic competence before we fully engage the learning rate. The technique turns what could be a destructive start into a controlled initialization process. Practically, the first few steps with near-zero learning rate act as a "scouting" phase where the network observes the data and makes tiny adjustments before committing to larger updates.

Weight decay (L2 regularization) prevents parameters from growing too large and encourages simpler models. For fine-tuning, values around 0.01 are common. As we discussed in the AdamW chapter, decoupled weight decay handles this more correctly than traditional L2 regularization in adaptive optimizers. Without weight decay, parameters can drift to very large values during fine-tuning, particularly for the newly initialized task head, leading to overconfident predictions and poor generalization.

The selection of which parameters to apply weight decay to is a subtle but important detail. Following the original BERT fine-tuning recipe, weight decay should be applied to most parameters but excluded for biases and layer normalization weights. Biases are typically small in magnitude and have little impact on generalization, while layer normalization weights are scaling parameters that should be free to adjust without regularization pressure. Applying weight decay to these parameters can prevent the normalization layers from correctly scaling activations, degrading model performance.