Full Fine-tuning: Hyperparameters & Learning Rate Schedules

In the previous chapter, we established that transfer learning lets us use knowledge from pre-trained models rather than train from scratch. Full fine-tuning is the most direct approach to transfer learning: we take a pre-trained model, update all of its parameters on task-specific data, and produce a specialized model. This chapter covers the mechanics of full fine-tuning, including the hyperparameters that govern the process and the subtle decisions that determine success or failure. We'll explain each step and give practical guidance for configuring your own fine-tuning runs.

Full fine-tuning updates every parameter in the model during training on your downstream task. For a model like BERT-base with 110 million parameters, this means computing gradients for and updating all 110 million weights. This approach maximizes the model's capacity to adapt to new tasks, but it requires significant computational and memory resources. Understanding how to configure the fine-tuning process correctly is needed for achieving good results, and the decisions you make here have real consequences: the difference between a well-configured fine-tuning run and a poorly configured one can mean the difference between 87% accuracy and 63% accuracy on the same dataset.

Think of fine-tuning as adjusting a highly skilled professional to a new job. Imagine hiring an experienced software engineer trained on a broad range of systems and codebases. That engineer already knows how to read documentation, debug problems, reason about data structures, and communicate technical concepts. When you hire them for your specific product, you don't re-teach those foundational skills. Instead, you help them apply those skills to your particular codebase, your team's conventions, and your product's domain. The adjustment takes weeks, not years, because the fundamentals are already there. Full fine-tuning works the same way: the model already knows how language works, how to extract meaning from context, and how to produce relevant representations. You are guiding that general competence toward your specific task.

The distinction between fine-tuning and training from scratch is qualitative as well as quantitative. A model trained from scratch on your small dataset must learn everything: vocabulary, syntax, semantics, world knowledge, and task-specific patterns, all from a few thousand examples. A fine-tuned model already possesses the first four, needing only to acquire the fifth. This specialization lets fine-tuned models to reach remarkable performance with surprisingly little task-specific data, and it is the foundation of how modern NLP works in practice.

The key insight is that the quality of your fine-tuning configuration matters far more than the size of your dataset. A BERT model fine-tuned with excellent hyperparameters on 5,000 examples often outperforms the same model fine-tuned poorly on 50,000 examples. Understanding the machinery of fine-tuning gives you the power to extract maximum performance from whatever resources you have available.

This chapter covers the complete fine-tuning pipeline: loading and configuring the model, preparing the data, selecting hyperparameters, understanding learning rate schedules mathematically, and diagnosing common failure modes. We walk through a concrete worked example with specific numbers to build intuition, and we examine the practical limitations that motivate the parameter-efficient alternatives we will explore in later chapters.

Full fine-tuning follows a structured procedure that turns a general-purpose pre-trained model into a task-specific one. While conceptually simple, each step involves decisions that affect the final model's performance, representing a careful balance between preserving pre-trained knowledge and adapting the model to excel at a specific downstream task. The procedure has four main phases: loading the pre-trained model and attaching a task head, preparing and tokenizing the data, configuring the training hyperparameters, and running the optimization loop while monitoring for convergence and overfitting.

Think of full fine-tuning as adapting a working system for a new purpose. The pre-trained model already has useful internal structure, and the task is to adjust the components needed for a particular output. You preserve pre-trained representations while adapting task-specific outputs. Fine-tuning therefore requires different hyperparameter choices than training from scratch.

The key insight is that the pre-trained backbone and the newly added task head have fundamentally different characteristics at the start of fine-tuning. The backbone contains carefully learned representations developed over billions of training examples. The task head is randomly initialized and has never processed a single input. These two components need to be synchronized during training, which shapes almost every hyperparameter choice in fine-tuning.

The process begins by loading a pre-trained checkpoint. This checkpoint contains the learned weights from pre-training on large corpora. For encoder models like BERT, these weights encode rich contextual representations that capture syntax and semantics, including pragmatic patterns and some factual world knowledge. For decoder models like GPT, they capture language generation capabilities that let fluent text production across diverse domains. The checkpoint is months of compute time and billions of training examples. All of this is distilled into a set of parameter values that serve as our starting point.

The magnitude of what a checkpoint contains is easy to underestimate. When BERT was pre-trained, it processed 3.3 billion words from BooksCorpus and English Wikipedia. It learned to predict masked tokens by developing internal representations that capture long-range dependencies, coreference relationships, syntactic structures, and semantic similarities. These representations are not stored as explicit rules or lookup tables but as patterns distributed across 110 million parameters. Loading the checkpoint loads all of that accumulated knowledge, instantly.

The key parameters for loading the model are:

• model_name: The identifier for the pre-trained model on Hugging Face Hub.
• num_labels: Number of output classes for the classification task.

When loading a model for a downstream task, the library adds a task-specific head on top of the pre-trained backbone. This head converts the model's internal representations into task-specific outputs, such as class probabilities for classification. For sequence classification, this is typically a linear layer that maps the final hidden states to class logits. The architecture follows a simple principle: the pre-trained model produces rich, contextual representations, and the task head turns these representations into the specific format required by your task.

Key concepts:

• task-specific head: a small neural network layer added on top of the pre-trained model to produce outputs for the target task
• final hidden states: the contextual representations produced by the pre-trained backbone for each token
• class logits: unnormalized scores for each class that will be converted to probabilities via softmax

The task-specific head acts as a bridge between the model's rich internal representations and the specific output format required by the task. For classification, it reduces the high-dimensional hidden states to a simple score per class. This reduction is mathematically straightforward: a linear transformation projects from the hidden dimension (768 for BERT-base) to the number of classes (2 for binary classification), but the simplicity of this mapping belies the complexity of the representations it receives.

The backbone parameters are initialized from pre-training, while the classification head is randomly initialized. This asymmetry changes the interpretation for training. The backbone starts with excellent representations, while the head must learn from scratch to interpret them correctly. The head is like a new translator who has just been handed a beautifully written document in a language they don't yet know how to interpret. The document is full of meaning; the translator just needs to learn the mapping between the representations and the output format required.

Fine-tuning requires task-specific labeled data. The data must be tokenized using the same tokenizer that was used during pre-training to ensure consistency in how text is represented. This consistency is important because the model learned to interpret specific token patterns during pre-training, and using a different tokenization scheme would create a mismatch between the input format and the model's learned expectations. The model expects its inputs to look exactly like the inputs it saw during pre-training, down to the specific subword boundaries and special tokens.

Think of tokenizer consistency like communicating with someone in a particular dialect. If you learned to speak with someone using British English, then suddenly switched to a completely different dialect, your communication partner would struggle even if both dialects convey the same underlying meaning. The model learned patterns over a specific tokenization scheme, and those patterns are now embedded in its weights. Switching tokenizers would be like speaking to it in an unfamiliar dialect that uses the same words but arranges them differently.

The tokenization process also introduces several structural elements that the model expects to see. For BERT, every sequence begins with a [CLS] token, ends with a [SEP] token, and may contain a second [SEP] token for paired-sequence tasks. These special tokens are not present in the raw text but are necessary for the model to function correctly: the [CLS] token's final hidden state is what BERT uses as the sequence-level representation for classification tasks.

The Rotten Tomatoes dataset gives thousands of movie reviews divided into a training set and separate validation and test sets. The training set size determines how many parameter updates will occur per epoch, while the validation and test sets allow us to monitor generalization and measure final performance on unseen data. The tokenization process converts text into the input format expected by the model. This format includes input IDs, attention masks, and optionally token type IDs. Using max_length=128 truncates longer sequences and pads shorter ones to a fixed length, letting efficient batched processing.

The key parameters for dataset preparation are:

• padding: Strategy for padding sequences. "max_length" pads all sequences to a fixed length.
• truncation: Whether to truncate sequences longer than max_length.
• max_length: Maximum sequence length, chosen to balance computational efficiency with information retention.

Key tokenization concepts:

• input IDs: integer tokens representing each word or subword piece in the vocabulary, mapping text to the numerical format required by the model
• attention masks: binary values showing which tokens are real content (1) versus padding (0), so the model ignores padding when computing attention
• token type IDs: optional integers distinguishing different segments in paired-sequence tasks, such as separating question from context

The attention mask is important because it prevents padded positions from influencing the model's computations. This keeps only real content contributes to the final representations. Without this mechanism, padding tokens would distort the attention patterns and corrupt the contextual representations that make transformers so effective. When computing self-attention, the model would otherwise treat padding tokens as real content. This allows them to influence the contextual representations of real tokens. The attention mask zeros out these contributions, effectively rendering padding tokens invisible to the attention mechanism.

Choosing max_length involves a practical tradeoff. Longer sequences capture more context but require more memory, limiting batch size. For sentiment classification of movie reviews, most reviews fit comfortably within 128 tokens. For longer documents, such as legal contracts or scientific papers, you would need larger max lengths, which come at the cost of smaller batch sizes or more GPU memory. This tradeoff between context length and computational cost recurs throughout transformer-based NLP.

Fine-tuning hyperparameters differ substantially from pre-training settings. Pre-training uses large batch sizes, high learning rates, and millions of steps. Fine-tuning uses smaller batch sizes, much lower learning rates, and only a few epochs. This approach prevents the pre-trained knowledge from being overwritten too quickly, preserving the model's existing capabilities while letting task-specific adaptation. The philosophy underlying these choices shows a basic insight: we are not learning from scratch, but rather making targeted adjustments to an already capable model.

The configuration choices for fine-tuning encode decades of accumulated empirical wisdom. The specific values, such as learning rates in the $10^{-5}$ range and warmup ratios of 0.06 to 0.1, were not derived from first principles but were discovered through extensive experimentation. The original BERT paper by Devlin et al. (2018) ran systematic experiments across multiple fine-tuning tasks and reported these ranges as reliably effective. Subsequent research has largely confirmed these findings, with some task-specific adjustments for very large or very small datasets.

These settings stand for a reasonable starting point for fine-tuning BERT on a classification task. The learning rate of $2 \times 10^{-5}$ is roughly 100 times smaller than typical pre-training learning rates. The warmup ratio of 0.1 means the learning rate will increase linearly for the first 10% of training steps before following the decay schedule. Each of these choices shows accumulated wisdom from thousands of fine-tuning experiments across the research community.

The fp16=True setting lets 16-bit floating-point precision during training, which roughly halves the memory requirements compared to 32-bit precision. This allows larger batch sizes on the same hardware. Modern GPUs have special hardware support for 16-bit operations, making them faster than 32-bit computations for many matrix operations. The tradeoff is a small risk of numerical instability due to reduced precision, which is why gradient scaling is applied automatically when fp16 training is enabled.

With the model loaded and the data prepared under the chosen configuration, training proceeds through the standard optimization loop. The Trainer API handles batching, gradient computation, parameter updates, and evaluation. Behind this convenient abstraction lies a advanced process that coordinates data loading, forward passes through the model, loss computation, backward passes for gradient calculation, and parameter updates according to the optimizer's rules.

Each training step involves the following sequence of operations. First, a batch of tokenized sequences is loaded from the training dataset. Second, the model processes the batch in a forward pass, computing representations at each layer until the classification head produces logits. Third, the cross-entropy loss between the logits and the true labels is computed. Fourth, gradients are computed via backpropagation, flowing from the loss through the classification head and then through every layer of the backbone. Fifth, the optimizer uses these gradients to update all parameters. This cycle repeats for every batch in every epoch.

The final training loss indicates successful convergence. This decreasing loss shows that the model has adapted its representations to the sentiment classification task, with both the pre-trained backbone and classification head updating to minimize prediction errors on movie reviews. During training, all parameters receive gradient updates. The pre-trained backbone parameters are nudged toward representations that better serve the classification task, while the randomly initialized classification head learns to map these representations to class predictions. The interplay between backbone adaptation and head learning creates a collaborative optimization process where both components adjust to work together effectively.

The convergence trajectory of the loss also tells you whether your hyperparameters are well-chosen. A steadily decreasing loss that stabilizes near the end of training indicates good hyperparameters. A loss that oscillates without converging often indicates a learning rate that is too high. A loss that decreases very slowly may indicate a learning rate that is too low or a batch size that is too small. Reading the loss curve is one of the most informative diagnostic tools available to you during fine-tuning.

Fine-tuning success depends on correctly setting several interconnected hyperparameters. Unlike pre-training, where large compute budgets allow for extensive hyperparameter searches, fine-tuning often requires getting these settings right with limited experimentation. Understanding the principles behind each hyperparameter helps practitioners make informed choices rather than relying solely on trial and error. The good news is that the community has converged on reliable default ranges that work well for most scenarios, so the search space is smaller than it might appear.

Think of the hyperparameter configuration as tuning a set of interdependent dials on a complex instrument. Each dial affects the others: the learning rate interacts with batch size, the warmup ratio interacts with the learning rate schedule, and the number of epochs interacts with regularization strength. The challenge is that these interactions are not always linear or predictable. A learning rate that works perfectly with one batch size may fail completely with another. Understanding the principles behind each interaction empowers you to reason about configurations rather than relying purely on grid search.

The key insight from the research literature is that most failures in fine-tuning trace back to one of three root causes: the learning rate is too high (destroying pre-trained representations), the number of epochs is too large (causing overfitting), or the warmup period is absent or too short (causing early instability). Mastering these three controls goes a long way toward reliable fine-tuning performance.

The learning rate is the most necessary hyperparameter in fine-tuning. It controls how much the weights change in response to computed gradients. A learning rate that is too high will destroy pre-trained representations, causing the model to forget what it learned during pre-training. A learning rate that is too low will result in insufficient adaptation to the new task. Finding the right balance requires understanding the unique constraints of fine-tuning compared to training from scratch.

For transformer models, fine-tuning learning rates typically fall in the range:

where:

• $\text{lr}_{\text{fine-tune}}$: the learning rate used during fine-tuning
• The range $[10^{-5}, 5 \times 10^{-5}]$: typical values that balance adaptation speed with preservation of pre-trained knowledge. The lower bound $10^{-5}$ is the minimum learning rate that still allows real adaptation, and the upper bound $5 \times 10^{-5}$ is the maximum rate before risking destruction of pre-trained representations

This narrow range shows a basic trade-off. Rates that are too high destroy pre-trained representations, while rates that are too low fail to adapt the model to new task patterns. The exponential notation ($10^{-5}$ means 0.00001) stresses how materially these rates differ from standard neural network training, where learning rates of 0.001 to 0.1 are common. The difference spans two to four orders of magnitude, showing that fine-tuning operates in a fundamentally different regime than training from random initialization.

The intuition behind this narrow range is that pre-trained models have already learned useful representations. We want to gently nudge these representations toward our specific task, not overwrite them entirely. Think of fine-tuning like adjusting a musical instrument: a violin that is slightly out of tune needs small, careful adjustments. Large, aggressive changes would destroy its calibration entirely. Similarly, a pre-trained model has found a good region of parameter space, and we want to explore nearby configurations rather than jumping to distant, potentially inferior locations. The mathematical relationship can be expressed as follows:

where:

• $\text{lr}_{\text{pre-train}}$: learning rate during pre-training (typically $1 \times 10^{-4}$ to $1 \times 10^{-3}$)
• $\text{lr}_{\text{fine-tune}}$: learning rate during fine-tuning (typically $1 \times 10^{-5}$ to $5 \times 10^{-5}$)
• $k$: the reduction factor, typically ranging from 10 to 100, representing how much we scale down the pre-training rate
• The division by $k$: scales down the pre-training rate to preserve learned representations while still letting adaptation
• The approximation symbol $\approx$: acknowledges that this is a practical guideline rather than an exact formula, since optimal rates vary by task and dataset

This relationship captures the core principle of fine-tuning: we start from good parameters and make small adjustments. The reduction factor $k$ quantifies "how small" those adjustments should be, balancing preservation of pre-trained knowledge versus adaptation to the new task. The formula gives a principled way to derive a starting learning rate when you know the pre-training configuration, and it explains why fine-tuning rates fall into such a specific, narrow range.

The reduction factor $k$ typically ranges from 10 to 100. Fine-tuning learning rates are one to two orders of magnitude smaller than pre-training rates. This large reduction is necessary because pre-trained parameters have already converged to representations that capture general linguistic knowledge. These representations are not perfect for any specific task, but they give an excellent foundation. Large updates would destroy this foundation, requiring the model to rebuild representations it already possesses. The analogy is to surgery: a scalpel, not a sledgehammer, is the right tool when you are making precise adjustments to something that is mostly already working correctly.

Fine-tuning typically requires only 2-5 epochs over the training data. This stands in stark contrast to training from scratch, which might require hundreds of epochs. The pre-trained model starts from an excellent initialization, so it needs only minor adjustments to perform well on the new task. The model has already learned how to process language; it just needs to learn how to apply that knowledge to your specific problem. Think of it like a musician learning a new piece: an experienced concert pianist can learn a new piece in a few hours, while a beginner might need months of practice to reach the same level.

Training for too many epochs leads to overfitting, where the model memorizes the training data rather than learning generalizable patterns. Because fine-tuning datasets are typically much smaller than pre-training corpora, this risk is particularly acute. A pre-training corpus might contain billions of tokens, while a fine-tuning dataset might contain only thousands or tens of thousands of examples. This size disparity means the model can easily memorize the fine-tuning data, losing its ability to generalize to new examples.

The key diagnostic for choosing the right number of epochs is the validation loss curve. If validation loss is still decreasing at the end of your training, you may benefit from additional epochs. If validation loss starts increasing while training loss continues to decrease, you have begun overfitting and should stop earlier. Many practitioners use early stopping: they monitor validation performance and halt training when it stops improving, regardless of the preset number of epochs. The best checkpoint is then the one with the lowest validation loss, not necessarily the last one.

The right number of epochs also depends on dataset size. Very small datasets (a few hundred examples) may benefit from more epochs, up to 10 or even 20, because each epoch gives limited learning signal. Larger datasets (tens of thousands of examples) typically require fewer epochs, since each epoch covers more diverse examples. When in doubt, err on the side of fewer epochs and use validation loss to guide your decision.

Batch size affects both training dynamics and computational requirements. Larger batches give more stable gradient estimates but require more memory. In fine-tuning, common batch sizes range from 16 to 32 for models like BERT on consumer GPUs. These moderate batch sizes balance the need for reliable gradient estimates against memory constraints and training speed.

The interaction between batch size and learning rate is important: larger batches generally benefit from higher learning rates, a relationship known as the linear scaling rule. If you double the batch size, you can approximately double the learning rate to maintain similar training dynamics. This relationship arises because larger batches produce more stable gradient estimates, letting the optimizer to take larger steps without risking divergence. Conversely, if you are forced to use a smaller batch size due to memory constraints, you should also reduce the learning rate proportionally to avoid training instability.

Batch size also affects the nature of the optimization trajectory. Smaller batches introduce more stochasticity into gradient estimates, which can act as a form of regularization by preventing the model from settling into sharp minima. Research by Keskar et al. (2016) showed that large-batch training tends to converge to sharp minima with poor generalization, while small-batch training tends to converge to flatter minima with better generalization. For fine-tuning tasks where overfitting is a concern, using moderately small batches (16-32 rather than 256-512) can improve test performance even if it slows training.

Warmup gradually increases the learning rate from zero to its target value over the first portion of training. This prevents the randomly initialized classification head from causing large, destabilizing gradient updates at the start of training. The newly added classification head produces in effect random outputs initially, leading to large loss values and correspondingly large gradients. If applied at the full learning rate, these gradients could corrupt the carefully learned backbone representations.

A typical warmup ratio is 0.06 to 0.1, meaning the learning rate ramps up linearly for the first 6-10% of total training steps. This gradual increase gives the classification head time to develop basic competence before the full learning rate is applied to the backbone. The warmup schedule can be expressed mathematically as follows:

where:

• $\text{lr}(t)$: the learning rate at training step $t$
• $\text{lr}_{\text{max}}$: the target maximum learning rate (e.g., $2 \times 10^{-5}$)
• $t$: the current training step
• $T_{\text{warmup}}$: the number of warmup steps
• The ratio $\frac{t}{T_{\text{warmup}}}$: the linear interpolation factor, starting at 0 and reaching 1 at the end of warmup

This piecewise function defines two phases: during warmup ($t < T_{\text{warmup}}$), the learning rate increases linearly from 0 to $\text{lr}_{\text{max}}$, and after warmup ($t \geq T_{\text{warmup}}$), the learning rate stays constant at $\text{lr}_{\text{max}}$ or begins following a decay schedule if configured. The piecewise structure shows the distinct needs of early versus later training: caution initially, then full engagement once the model has stabilized.

The function ensures a smooth ramp-up. At $t = 0$:

At $t = T_{\text{warmup}}$:

The linear interpolation prevents sudden jumps in the learning rate, which is necessary because the randomly initialized classification head produces large, unstable gradients at the start of training. By ramping up gradually, we give the head time to learn reasonable outputs before applying full learning rate updates to the pre-trained backbone. The head moves from random predictions toward sensible outputs over the warmup period, and by the time the full learning rate engages, its gradients reflect real learning signals rather than noise from random initialization.

Warmup matters because without this gradual ramp-up, the random classification head would compute meaningless losses on the first few batches, generating enormous gradients that could corrupt the carefully pre-trained backbone weights. Warmup gives the head several hundred steps to develop basic competence before we fully engage the learning rate. The technique turns what could be a destructive start into a controlled initialization process. Practically, the first few steps with near-zero learning rate act as a "scouting" phase where the network observes the data and makes tiny adjustments before committing to larger updates.

Weight decay (L2 regularization) prevents parameters from growing too large and encourages simpler models. For fine-tuning, values around 0.01 are common. As we discussed in the AdamW chapter, decoupled weight decay handles this more correctly than traditional L2 regularization in adaptive optimizers. Without weight decay, parameters can drift to very large values during fine-tuning, particularly for the newly initialized task head, leading to overconfident predictions and poor generalization.

The selection of which parameters to apply weight decay to is a subtle but important detail. Following the original BERT fine-tuning recipe, weight decay should be applied to most parameters but excluded for biases and layer normalization weights. Biases are typically small in magnitude and have little impact on generalization, while layer normalization weights are scaling parameters that should be free to adjust without regularization pressure. Applying weight decay to these parameters can prevent the normalization layers from correctly scaling activations, degrading model performance.

Choosing the right learning rate requires balancing competing concerns. We want enough change to adapt to the new task, but not so much that we destroy pre-trained knowledge. This balancing act is at the heart of successful fine-tuning. Understanding the available tools helps practitioners address this challenge effectively. The community has developed systematic methods for finding good learning rates without exhaustive search.

The search for a good learning rate is constrained by the nature of the problem. Too low, and the model makes negligible progress within the allotted epochs. Too high, and the pre-trained representations are corrupted early in training, making recovery impossible. The optimal learning rate sits in a relatively narrow band between these failure modes, and the goal of the strategies we discuss is to find that band efficiently.

The learning rate typically follows a schedule that varies it throughout training. The most common schedules for fine-tuning are designed to start with more aggressive learning (once warmup completes) and gradually become more conservative as training progresses. This pattern shows the intuition that early training benefits from larger updates to establish good task-specific representations, while later training benefits from smaller updates to fine-tune details without overshooting optimal values.

Linear decay starts at the target learning rate after warmup and decreases linearly to zero by the end of training. This straightforward approach gives a predictable reduction in learning rate that many practitioners find effective. After the warmup phase, the schedule follows:

where:

• $\text{lr}(t)$: the learning rate at training step $t$ (after warmup)
• $\text{lr}_{\text{max}}$: the maximum learning rate reached after warmup
• $t$: the current training step (where $t \geq T_{\text{warmup}}$)
• $T_{\text{warmup}}$: the number of warmup steps
• $T_{\text{total}}$: the total number of training steps
• The term $\left(1 - \frac{t - T_{\text{warmup}}}{T_{\text{total}} - T_{\text{warmup}}}\right)$: the linear decay factor, starting at 1 and reaching 0 at the end of training
• The numerator $t - T_{\text{warmup}}$: measures how many steps have elapsed since warmup ended
• The denominator $T_{\text{total}} - T_{\text{warmup}}$: measures the total number of training steps after warmup
• Their ratio: is the fraction of post-warmup training completed, growing from 0 to 1

To understand how this formula works, let's examine the decay factor at key points during training. This analysis reveals how the schedule transitions smoothly from full learning rate to zero.

At the start of decay ($t = T_{\text{warmup}}$):

This gives the full learning rate: $\text{lr}(T_{\text{warmup}}) = \text{lr}_{\text{max}} \times 1 = \text{lr}_{\text{max}}$.

At the end of training ($t = T_{\text{total}}$):

This reduces the learning rate to zero: $\text{lr}(T_{\text{total}}) = \text{lr}_{\text{max}} \times 0 = 0$.

The intuition is that as we approach the end of training, we want smaller updates to avoid overshooting the optimal parameters. At intermediate points, the decay factor falls between 0 and 1, creating a smooth linear transition from the maximum learning rate to zero. Starting with larger updates allows faster initial progress, while ending with tiny updates lets fine refinement. This linear decrease gives a gentle transition that helps the model converge smoothly, which makes it the default choice for BERT-style fine-tuning because it works well across many tasks. The simplicity of linear decay also makes it easy to reason about: halfway through training, the learning rate will be at half its maximum value.

Cosine decay decreases the learning rate following a cosine curve with a smoother transition. It decreases slowly at first, more rapidly in the middle, and slowly again at the end. This shape shows the mathematical properties of the cosine function and gives a more gradual transition than linear decay. After warmup, the schedule follows:

where:

• $\text{lr}(t)$: the learning rate at training step $t$ (after warmup)
• $\text{lr}_{\text{max}}$: the maximum learning rate reached after warmup
• $t$: the current training step (where $t \geq T_{\text{warmup}}$)
• $T_{\text{warmup}}$: the number of warmup steps
• $T_{\text{total}}$: the total number of training steps
• $\cos(\cdot)$: the cosine function, which oscillates between -1 and 1, creating the smooth decay curve when combined with the other terms
• The argument $\pi \times \frac{t - T_{\text{warmup}}}{T_{\text{total}} - T_{\text{warmup}}}$: maps training progress linearly from 0 to $\pi$
• The cosine of this argument: decreases from $\cos(0) = 1$ to $\cos(\pi) = -1$ as training progresses
• The addition of 1: shifts the range from $[-1, 1]$ to $[0, 2]$
• The factor 0.5: scales the range from $[0, 2]$ to $[0, 1]$, giving the final decay multiplier

Let's verify the formula at key points to confirm it behaves as expected. Understanding these boundary conditions helps build intuition for how the schedule operates throughout training.

At the start of decay ($t = T_{\text{warmup}}$):

At the end of training ($t = T_{\text{total}}$):

The cosine curve creates a smooth S-shaped transition: it starts with slow decay (letting aggressive early learning), accelerates in the middle (transitioning from exploration to refinement), and then slows again near zero (letting precise convergence). This gentler schedule can be particularly beneficial when fine-tuning requires careful preservation of pre-trained representations. The characteristic shape of the cosine curve means the model spends more time at higher learning rates (good for making progress) and more time at very low learning rates (good for final convergence) compared to linear decay.

Constant keeps the learning rate fixed throughout training. This is simpler but often underperforms scheduled approaches. Without decay, the model continues making relatively large updates even as it approaches convergence, which can cause oscillation around the optimum rather than settling into it. However, constant learning rates can work well for very short training runs or when combined with other regularization techniques. Some modern training recipes for large language models use constant learning rates with an aggressive final decay, which can reach strong results while being simpler to configure.

While default values like $2 \times 10^{-5}$ work well for many BERT fine-tuning scenarios, optimal learning rates vary with the model and task as well as the dataset size. Several strategies can help identify good values, each giving different trade-offs between thoroughness and computational cost.

Grid search evaluates a small set of candidate learning rates, typically varying by factors of 2-3 (e.g., 1e-5, 2e-5, 5e-5). With small fine-tuning datasets, this remains computationally feasible. The approach is systematic and gives clear comparisons across candidates. A grid search over 3-5 candidate learning rates requires 3-5x the compute of a single run, but the improvement from finding the right learning rate can more than compensate for this cost.

Learning rate finder gradually increases the learning rate during a short training run, plotting loss against learning rate. The optimal learning rate is often just before the loss begins to increase sharply. This technique gives quick insights without full training runs. The method exploits the fact that loss should decrease as learning rate increases from very small values, but will eventually start increasing when the learning rate becomes too large. The "sweet spot" is typically just before the loss begins to rise steeply, showing the onset of instability.

Model-specific heuristics use community knowledge. BERT models typically work well with 2e-5 to 3e-5. Larger models like RoBERTa-large often benefit from slightly lower rates. GPT-style decoder models may need different ranges depending on the task. These heuristics encode the collective experience of thousands of practitioners and give reliable starting points. When starting a new fine-tuning project, these community-established defaults are your best first guess.

To build concrete intuition for how these hyperparameters interact, let's walk through a complete configuration calculation from scratch. Suppose we want to fine-tune BERT-base on a sentiment classification dataset with 8,000 training examples, using a batch size of 16 and planning to train for 3 epochs.

Step 1: Calculate total training steps. Each epoch processes all 8,000 training examples in batches of 16.

With 3 epochs, the total number of training steps is:

Step 2: Calculate warmup steps. With a warmup ratio of 0.1, we devote 10% of training to warmup:

So the learning rate increases linearly over the first 150 steps, then decays over the remaining 1,350 steps.

Step 3: Verify the warmup-to-training ratio. The warmup covers 150 out of 500 steps in the first epoch, meaning the learning rate is still ramping up for the first 30% of epoch 1. This is normal and expected. By step 150 (30% into epoch 1), the learning rate reaches its maximum. It then decays linearly over the remaining 1,350 steps.

Step 4: Calculate the learning rate at each step. Using $\text{lr}_{\text{max}} = 2 \times 10^{-5}$ and the linear decay formula, the learning rate at step 750 (halfway through training, midpoint of the decay phase) is:

At the midpoint of the decay phase, the learning rate has decreased to approximately $1.11 \times 10^{-5}$, which is about 55.6% of its maximum value. This is expected from a linear schedule: at the midpoint, you should have approximately half the maximum learning rate remaining.

Step 5: Calculate effective batch size with gradient accumulation. If GPU memory allows only 8 samples per forward pass but we want an effective batch size of 32, we need gradient accumulation:

With gradient accumulation of 4, the number of optimizer steps per epoch becomes:

This means the total number of parameter updates across all 3 epochs is $125 \times 3 = 375$, rather than the 1,500 forward/backward passes. The gradient quality is equivalent to using a batch size of 32, but you need only 8 samples in GPU memory at any time.

Step 6: Verify the linear scaling rule for learning rate. Our baseline learning rate of $2 \times 10^{-5}$ was validated for batch size 16. If we switch to effective batch size 32, the linear scaling rule suggests:

However, since we are using gradient accumulation to simulate batch size 32 rather than true batch size 32, the effective gradient dynamics are slightly different. In practice, gradient accumulation with 4 steps of batch size 8 is mathematically equivalent to true batch size 32, so the learning rate scaling applies equally. The final configuration for this scenario would be: batch size 8, gradient accumulation 4, effective batch size 32, learning rate $4 \times 10^{-5}$, warmup steps 150, total optimizer steps 375.

Batch size interacts with training in multiple ways, affecting memory usage, gradient quality, and convergence behavior. Understanding these interactions helps practitioners make informed choices when configuring their training runs. The batch size decision is rarely made in isolation: it must be considered alongside learning rate, gradient accumulation, and available hardware.

Think of batch size as the number of voters you poll before making a policy decision. Polling more voters gives you a more accurate estimate of public opinion, but it takes more time and resources. If you poll only 8 people (small batch), your estimate will be noisy and might lead you in the wrong direction on any given decision. If you poll 256 people (large batch), your estimate is reliable but each poll is expensive. The right number depends on how much you can afford per decision and how much noise you can tolerate. Importantly, with noisier estimates, you should be more conservative in how sharply you act on each one, which is why smaller batches pair with smaller learning rates.

The primary constraint on batch size is GPU memory. Each sample in a batch requires memory for input tensors (token IDs, attention masks), intermediate activations (stored for backward pass), gradients for all parameters, and optimizer states (momentum buffers for Adam). For a model like BERT-base with sequences of length 512, batch sizes of 8-16 are common on consumer GPUs with 12-16GB of memory.

The memory requirements break down into several components:

• Input tensors: the tokenized representation of each text sample, including token IDs and attention masks
• Intermediate activations: the outputs of each layer during the forward pass, cached for gradient computation during the backward pass
• Gradients: the partial derivatives of the loss with respect to each parameter, computed during backpropagation
• Optimizer states: additional buffers that adaptive optimizers like Adam maintain, specifically momentum and variance estimates for each parameter

The optimizer state memory deserves special attention. Adam and its variants maintain two additional state vectors per parameter: a running mean of gradients (first moment) and a running mean of squared gradients (second moment). For a model with $P$ parameters stored in 32-bit precision, the parameter storage alone requires $4P$ bytes. The optimizer states add another $8P$ bytes (4 bytes each for first and second moments), bringing the total memory for parameters plus optimizer states to $12P$ bytes. For BERT-base with 110 million parameters, this amounts to approximately 1.3 GB just for parameters and optimizer states, before counting activations and inputs.

Larger models or longer sequences require smaller batches or memory optimization techniques like gradient checkpointing, which trades computation for memory by recomputing activations during the backward pass rather than storing them. Gradient checkpointing can reduce memory requirements materially at the cost of approximately 30-40% more computation during the backward pass.

Smaller batches produce noisier gradient estimates. Each gradient is computed from fewer examples, which makes it a less accurate estimate of the true gradient over the entire dataset. This noise has both advantages and disadvantages that practitioners should understand.

Advantages of noise: Gradient noise can help escape local minima and may give implicit regularization, preventing overfitting. Some research suggests noisy gradients improve generalization by preventing the model from settling into sharp minima that may not transfer well to test data. For fine-tuning tasks with small datasets, the implicit regularization from noisy gradients can be beneficial, helping the model avoid memorizing the training set.

Disadvantages of noise: Excessive noise makes training unstable and can prevent convergence. Very small batches may require learning rate reductions that slow training sharply. The model may oscillate instead of making consistent progress toward the optimum. When the gradient noise is larger than the gradient signal, the optimizer is in effect making random steps, which is inefficient and potentially harmful.

The appropriate level of gradient noise depends on your task and dataset size. For very small datasets (a few hundred examples), using small batches gives more gradient noise, which can act as regularization. For larger datasets where the main concern is training efficiency rather than overfitting, larger batches and higher learning rates are preferable.

When memory constraints prevent using the desired batch size, gradient accumulation gives a workaround. Instead of updating weights after each batch, we accumulate gradients over multiple batches, then update. This technique effectively decouples the memory requirements (determined by the batch that fits in GPU memory) from the effective batch size (determined by how many batches we accumulate before updating).

If you want an effective batch size of 32 but can only fit 8 samples in memory, you accumulate gradients over 4 batches. Gradient accumulation allows us to simulate larger batch sizes by computing multiple forward and backward passes before performing a single parameter update. The relationship is expressed by the following formula:

where:

• $B_{\text{effective}}$: the effective batch size used for parameter updates (32 in this example)
• $B_{\text{device}}$: the batch size that fits in device memory (8 in this example)
• $N_{\text{accum}}$: the number of gradient accumulation steps (4 in this example)
• The multiplication: combines multiple small batches into one large gradient update, trading time for memory

This formula shows that the effective batch size equals the device batch size multiplied by the number of accumulation steps. The key insight is that gradient accumulation computes $N_{\text{accum}}$ forward and backward passes, summing the gradients before performing a single parameter update. This produces mathematically equivalent results to processing all $B_{\text{effective}}$ samples simultaneously, but uses only $B_{\text{device}}$ worth of memory at any given time. The trade-off is computational: we perform more forward and backward passes, but each pass uses less memory.

Applying this formula to our example shows how the pieces fit together:

Larger batches benefit from higher learning rates. The intuition is straightforward: with more samples, the gradient estimate is more reliable, so we can take larger steps confidently. When we compute a gradient from 32 samples instead of 16, the estimate is closer to the true gradient over the full dataset, and we can trust it more. The linear scaling rule suggests that learning rate should scale proportionally with batch size according to the following relationship:

where:

• $\text{lr}_{\text{new}}$: the adjusted learning rate for the new batch size
• $\text{lr}_{\text{base}}$: the baseline learning rate that was validated with the baseline batch size
• $B_{\text{new}}$: the new batch size you want to use
• $B_{\text{base}}$: the baseline batch size used during validation
• The ratio $\frac{B_{\text{new}}}{B_{\text{base}}}$: the scaling factor by which the batch size changes

This formula adjusts the learning rate proportionally to changes in batch size. The linear scaling rule works because doubling the batch size doubles the number of samples adding to each gradient, making the gradient estimate twice as stable. This stability allows us to take steps that are twice as large (double the learning rate) while maintaining similar training dynamics. The rule preserves the expected magnitude of parameter updates per training sample, keeping the optimization trajectory similar despite different batch sizes.

However, this rule has limits. Very large batches (thousands of samples) often fail to benefit from proportionally scaled learning rates. The relationship becomes sublinear, and other techniques like LAMB (Layer-wise Adaptive Moments) become necessary for successful training. The breakdown occurs because extremely large batches can converge to sharp minima that generalize poorly, and at some point the additional samples give diminishing marginal benefit to gradient quality. Practitioners using batch sizes above 256 for fine-tuning should validate that the linear scaling rule still holds for their specific configuration.

For fine-tuning with typical batch sizes (16-64), the linear scaling rule gives reasonable guidance. If you've validated that a learning rate of 2e-5 works well with batch size 16, then batch size 32 might benefit from 4e-5. Let's verify this by applying the linear scaling rule:

This calculation shows that doubling the batch size from 16 to 32 suggests doubling the learning rate from 2e-5 to 4e-5. The linear scaling rule gives a principled starting point, though practitioners should still validate that the scaled learning rate works well for their specific task.

Let's walk through a complete fine-tuning implementation, examining each component in detail. Understanding the underlying mechanics of the training loop gives you the ability to debug problems, implement custom modifications, and reason about what is happening during training. High-level APIs are convenient, but knowing what they are doing under the hood is needed for serious practitioners.

The training loop has a deceptively simple structure: iterate over batches, compute loss, compute gradients, update parameters. But each of these steps involves important decisions that affect training stability and final performance. The details of gradient clipping, optimizer parameter grouping, and evaluation frequency all matter for achieving good results.

While high-level APIs handle many details, understanding the underlying training loop gives insight into what is happening during fine-tuning. The key components are the optimizer, the learning rate scheduler, and the parameter grouping strategy.

Notice the parameter grouping: we apply weight decay to most parameters but exclude biases and layer normalization weights. This follows the original BERT fine-tuning recipe and prevents these parameters from being pushed toward zero unnecessarily. The no_decay list identifies parameters that should remain free from regularization pressure. Biases are typically small in magnitude and contribute little to overfitting. Layer normalization parameters are scaling and shifting factors that should be free to take whatever values the model needs for proper normalization.

The AdamW optimizer is the standard choice for fine-tuning transformers. It implements decoupled weight decay, maintaining the beneficial adaptive learning rate behavior of Adam while correctly separating the regularization effect of weight decay. Using standard Adam with L2 regularization added to the loss produces suboptimal results for transformers because the adaptive learning rates interact with the gradient in a way that diminishes the regularization effect. AdamW avoids this issue by applying weight decay directly to parameters after the gradient update.

Each training step involves a forward pass, loss computation, backward pass, and parameter update. The implementation below shows the needed operations, including gradient clipping, which is a necessary stability measure for fine-tuning.

Gradient clipping prevents any single batch from causing excessively large updates. The maximum gradient norm of 1.0 is a common choice for fine-tuning, constraining the total magnitude of gradients across all parameters to prevent training instability. The clip_grad_norm_ function computes the global gradient norm (the L2 norm across all parameter gradients), and if it exceeds the specified maximum, scales all gradients down proportionally so that the total norm equals exactly 1.0. This operation does not change the direction of the update, only its magnitude.

Gradient clipping is particularly important in the early epochs of fine-tuning, when the classification head is still learning rapidly and may produce large gradients. Without clipping, a single bad batch could push the backbone parameters far from their pre-trained values. The clipping threshold of 1.0 was chosen empirically and has become a community standard for BERT-style fine-tuning. For larger models, you may occasionally need to adjust this threshold if you observe that gradients are being clipped too aggressively.

Evaluating during fine-tuning helps monitor for overfitting and select the best checkpoint. Evaluation should be done frequently enough to detect overfitting early, but not so frequently that it materially slows down training. Evaluating once per epoch is usually sufficient for fine-tuning runs of 2-5 epochs.

The torch.no_grad() context manager is necessary during evaluation: it disables gradient computation, reducing memory usage and speeding up the forward pass. During evaluation, we don't need gradients because we are not updating parameters. Forgetting to use torch.no_grad() during evaluation can cause memory exhaustion and materially slow down your training loop, especially when evaluating on large validation sets.

The model.eval() call is equally important: it switches certain layers from their training behavior to their evaluation behavior. Dropout layers, for example, randomly zero out activations during training to prevent co-adaptation of neurons, but should be disabled during evaluation to produce consistent predictions. Batch normalization layers similarly behave differently during training and evaluation. Failing to call model.eval() before evaluation will produce inconsistent results and may underestimate the model's true performance.

Putting it all together into a complete training run:

The key parameters for full fine-tuning are:

• learning_rate: Controls how much weights change per update. Typically 1e-5 to 5e-5 for transformer fine-tuning.
• num_train_epochs: Number of passes through the training data. Usually 2-5 epochs for fine-tuning.
• per_device_train_batch_size: Samples per batch on each device. Common values are 16-32 for BERT-sized models.
• warmup_ratio: Fraction of training for learning rate warmup. Typically 0.06-0.1 (6-10%).
• weight_decay: L2 regularization strength. Common value is 0.01.
• gradient_accumulation_steps: Number of batches to accumulate before updating. Use to simulate larger batch sizes.

The training progresses through multiple epochs, with both training and validation metrics tracked at each step. The best model checkpoint is saved based on validation accuracy, which helps prevent overfitting by selecting the model state that generalizes best to unseen data. The final validation accuracy shows how well the fine-tuned model performs on held-out examples.

Tracking loss and accuracy across epochs helps diagnose training issues. The training curves encode a wealth of information about what is happening during optimization, and learning to read them fluently is one of the most useful skills for a practitioner.

The training curves reveal the dynamics of fine-tuning. Training loss should decrease steadily across epochs. Validation loss may decrease initially but can increase if the model begins overfitting. The gap between training and validation loss indicates the degree of overfitting, and a widening gap is a signal to consider stronger regularization or reducing the number of epochs. The validation accuracy curve gives the most direct measure of task performance and is typically the metric used to select the best checkpoint for deployment.

To illustrate the impact of hyperparameter choices, let's compare different learning rate configurations across the training process. This comparison reveals several important patterns that help build intuition for how fine-tuning behaves under different conditions.

The comparison reveals several patterns characteristic of fine-tuning:

• Learning rates in the 2e-5 range typically perform best for BERT-style models, confirming the empirically established community recommendation
• Lower rates (1e-5) learn more slowly but can reach good results with more epochs, making them suitable when computational budget allows for longer training runs
• Higher rates (5e-5) show faster initial progress but can plateau earlier, showing that more aggressive updates quickly reach a local optimum that may not be globally optimal
• Excessively high rates (1e-4) damage pre-trained representations and lead to poor, unstable performance, with non-monotonic accuracy curves that reveal the destructive effect of oversized updates

The non-monotonic behavior of the 1e-4 configuration is particularly instructive. Accuracy rises in epoch 1, as even large random updates occasionally improve performance, then falls in epoch 2 as the pre-trained representations are corrupted, then partially recovers in epoch 3 as the model attempts to find a new equilibrium. This pattern is a clear diagnostic for a learning rate that is too high: the model cannot maintain consistent progress because each update is large enough to move the parameters away from good configurations.

Full fine-tuning is powerful but comes with significant limitations that practitioners must understand. These limitations are basic constraints rather than minor inconveniences, and they have driven entire research directions aimed at making fine-tuning more efficient and accessible.

Computational cost scales directly with model size. Fine-tuning BERT-base requires moderate resources: a single GPU can complete training in minutes to hours depending on dataset size. Fine-tuning a 7B parameter model demands multiple high-end GPUs and can take days. Each forward and backward pass touches every parameter, and the cost grows linearly with the number of parameters. For the largest modern models, such as GPT-4 class systems with hundreds of billions of parameters, full fine-tuning is computationally impractical for most organizations. Even a single fine-tuning run would require thousands of GPU hours and significant infrastructure.

The memory requirements compound the computational challenge. For each parameter of size $p$ bytes, Adam stores the parameter itself ($p$ bytes), a first-moment buffer ($p$ bytes), and a second-moment buffer ($p$ bytes), totaling $3p$ bytes just for parameters and optimizer states. For a 7B parameter model in 16-bit precision, this amounts to approximately 42 GB, which exceeds the memory of any consumer GPU and most professional GPUs. Practitioners working with large models must use distributed training across multiple GPUs, adding complexity and infrastructure cost. This constraint is precisely what motivates the parameter-efficient fine-tuning approaches we will explore in Part XXXV of this handbook.

Storage requirements compound when serving multiple tasks. Each fine-tuned model is a complete copy of the original. If you fine-tune BERT for 10 different tasks, you need to store 10 complete models, each at full size. For a 3B parameter model at 16-bit precision, each task requires roughly 6 GB of storage (3 billion parameters times 2 bytes per parameter). Ten tasks require 60 GB, a hundred tasks require 600 GB, and so on. This creates significant infrastructure challenges for multi-task deployments, especially in production environments where models must be loaded and served with low latency. The storage cost alone can make full fine-tuning impractical for applications requiring many specialized variants of a model.

Catastrophic forgetting remains a basic challenge. As the model adapts to new data, it can lose capabilities from pre-training. A model fine-tuned heavily on medical text might perform worse on general language understanding than the original. The fine-tuning updates that improve performance on the target task necessarily move parameters away from the configurations that encoded general capabilities. This is not a bug but a consequence of gradient descent: the optimizer minimizes the target task loss, not the pre-training loss, so parameters shift toward configurations that favor the new task at the expense of old capabilities. We will examine this phenomenon in detail in the next chapter, covering why it occurs and techniques to mitigate it.

The magnitude of catastrophic forgetting depends on the learning rate, the number of epochs, and the similarity between the pre-training distribution and the fine-tuning distribution. Large learning rates and many epochs increase forgetting. Fine-tuning on a distribution similar to pre-training causes less forgetting than fine-tuning on a highly specialized domain. Understanding these dependencies helps you calibrate your fine-tuning configuration to reach good task performance while minimizing capability degradation.

Despite these limitations, full fine-tuning remains the standard for maximum task performance. When you have sufficient compute and storage, along with enough task-specific data, full fine-tuning typically outperforms parameter-efficient alternatives. The entire model can adapt its representations to the target domain, using every parameter toward the task objective. Parameter-efficient methods give useful approximations, but they constrain the space of representational changes the model can make, which sometimes limits their ceiling performance. When maximum accuracy is the goal and resources permit, full fine-tuning is the right choice.

Full fine-tuning adapts all parameters of a pre-trained model to a downstream task. The procedure involves loading a pre-trained checkpoint, adding a task-specific head, and training on labeled data with carefully selected hyperparameters. The configuration choices encode empirical wisdom accumulated from thousands of experiments: learning rates in the $10^{-5}$ range, warmup periods covering the first 6-10% of training, and 2-5 epochs of optimization.

Here are the necessary hyperparameters for fine-tuning:

• Learning rate: typically $1 \times 10^{-5}$ to $5 \times 10^{-5}$ (much lower than pre-training rates)
• Number of epochs: usually 2-5 (just enough for adaptation without overfitting)
• Batch size: 16-32 for consumer GPUs; use gradient accumulation for larger effective sizes
• Warmup ratio: 0.06 to 0.1 (6% to 10%) of training steps to stabilize early training
• Weight decay: around 0.01, applied to most but not all parameters

Learning rate schedules like linear or cosine decay, combined with warmup, give stable training dynamics. The batch size-learning rate relationship suggests scaling both together for consistent behavior. Gradient accumulation decouples memory requirements from effective batch size. This makes possible flexible training across hardware configurations.

Full fine-tuning excels when compute resources are available and maximum task performance is the goal. The complete update of all parameters allows the model to develop deep task-specific specialization that parameter-efficient methods cannot fully replicate. For resource-constrained scenarios or multi-task deployments, parameter-efficient methods offer attractive alternatives that sacrifice some peak performance for large reductions in memory and storage requirements. In the following chapters, we will examine catastrophic forgetting in depth and explore the parameter-efficient fine-tuning methods that have become the practical standard for large-scale deployments.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about full fine-tuning.