Full Fine-tuning › The Full Fine-tuning Procedure

Full fine-tuning follows a structured procedure that turns a general-purpose pre-trained model into a task-specific one. While conceptually simple, each step involves decisions that affect the final model's performance, representing a careful balance between preserving pre-trained knowledge and adapting the model to excel at a specific downstream task. The procedure has four main phases: loading the pre-trained model and attaching a task head, preparing and tokenizing the data, configuring the training hyperparameters, and running the optimization loop while monitoring for convergence and overfitting.

Think of full fine-tuning as adapting a working system for a new purpose. The pre-trained model already has useful internal structure, and the task is to adjust the components needed for a particular output. You preserve pre-trained representations while adapting task-specific outputs. Fine-tuning therefore requires different hyperparameter choices than training from scratch.

The key insight is that the pre-trained backbone and the newly added task head have fundamentally different characteristics at the start of fine-tuning. The backbone contains carefully learned representations developed over billions of training examples. The task head is randomly initialized and has never processed a single input. These two components need to be synchronized during training, which shapes almost every hyperparameter choice in fine-tuning.

The process begins by loading a pre-trained checkpoint. This checkpoint contains the learned weights from pre-training on large corpora. For encoder models like BERT, these weights encode rich contextual representations that capture syntax and semantics, including pragmatic patterns and some factual world knowledge. For decoder models like GPT, they capture language generation capabilities that let fluent text production across diverse domains. The checkpoint is months of compute time and billions of training examples. All of this is distilled into a set of parameter values that serve as our starting point.

The magnitude of what a checkpoint contains is easy to underestimate. When BERT was pre-trained, it processed 3.3 billion words from BooksCorpus and English Wikipedia. It learned to predict masked tokens by developing internal representations that capture long-range dependencies, coreference relationships, syntactic structures, and semantic similarities. These representations are not stored as explicit rules or lookup tables but as patterns distributed across 110 million parameters. Loading the checkpoint loads all of that accumulated knowledge, instantly.

The key parameters for loading the model are:

• model_name: The identifier for the pre-trained model on Hugging Face Hub.
• num_labels: Number of output classes for the classification task.

When loading a model for a downstream task, the library adds a task-specific head on top of the pre-trained backbone. This head converts the model's internal representations into task-specific outputs, such as class probabilities for classification. For sequence classification, this is typically a linear layer that maps the final hidden states to class logits. The architecture follows a simple principle: the pre-trained model produces rich, contextual representations, and the task head turns these representations into the specific format required by your task.

Key concepts:

• task-specific head: a small neural network layer added on top of the pre-trained model to produce outputs for the target task
• final hidden states: the contextual representations produced by the pre-trained backbone for each token
• class logits: unnormalized scores for each class that will be converted to probabilities via softmax

The task-specific head acts as a bridge between the model's rich internal representations and the specific output format required by the task. For classification, it reduces the high-dimensional hidden states to a simple score per class. This reduction is mathematically straightforward: a linear transformation projects from the hidden dimension (768 for BERT-base) to the number of classes (2 for binary classification), but the simplicity of this mapping belies the complexity of the representations it receives.

The backbone parameters are initialized from pre-training, while the classification head is randomly initialized. This asymmetry changes the interpretation for training. The backbone starts with excellent representations, while the head must learn from scratch to interpret them correctly. The head is like a new translator who has just been handed a beautifully written document in a language they don't yet know how to interpret. The document is full of meaning; the translator just needs to learn the mapping between the representations and the output format required.

Fine-tuning requires task-specific labeled data. The data must be tokenized using the same tokenizer that was used during pre-training to ensure consistency in how text is represented. This consistency is important because the model learned to interpret specific token patterns during pre-training, and using a different tokenization scheme would create a mismatch between the input format and the model's learned expectations. The model expects its inputs to look exactly like the inputs it saw during pre-training, down to the specific subword boundaries and special tokens.

Think of tokenizer consistency like communicating with someone in a particular dialect. If you learned to speak with someone using British English, then suddenly switched to a completely different dialect, your communication partner would struggle even if both dialects convey the same underlying meaning. The model learned patterns over a specific tokenization scheme, and those patterns are now embedded in its weights. Switching tokenizers would be like speaking to it in an unfamiliar dialect that uses the same words but arranges them differently.

The tokenization process also introduces several structural elements that the model expects to see. For BERT, every sequence begins with a [CLS] token, ends with a [SEP] token, and may contain a second [SEP] token for paired-sequence tasks. These special tokens are not present in the raw text but are necessary for the model to function correctly: the [CLS] token's final hidden state is what BERT uses as the sequence-level representation for classification tasks.

The Rotten Tomatoes dataset gives thousands of movie reviews divided into a training set and separate validation and test sets. The training set size determines how many parameter updates will occur per epoch, while the validation and test sets allow us to monitor generalization and measure final performance on unseen data. The tokenization process converts text into the input format expected by the model. This format includes input IDs, attention masks, and optionally token type IDs. Using max_length=128 truncates longer sequences and pads shorter ones to a fixed length, letting efficient batched processing.

The key parameters for dataset preparation are:

• padding: Strategy for padding sequences. "max_length" pads all sequences to a fixed length.
• truncation: Whether to truncate sequences longer than max_length.
• max_length: Maximum sequence length, chosen to balance computational efficiency with information retention.

Key tokenization concepts:

• input IDs: integer tokens representing each word or subword piece in the vocabulary, mapping text to the numerical format required by the model
• attention masks: binary values showing which tokens are real content (1) versus padding (0), so the model ignores padding when computing attention
• token type IDs: optional integers distinguishing different segments in paired-sequence tasks, such as separating question from context

The attention mask is important because it prevents padded positions from influencing the model's computations. This keeps only real content contributes to the final representations. Without this mechanism, padding tokens would distort the attention patterns and corrupt the contextual representations that make transformers so effective. When computing self-attention, the model would otherwise treat padding tokens as real content. This allows them to influence the contextual representations of real tokens. The attention mask zeros out these contributions, effectively rendering padding tokens invisible to the attention mechanism.

Choosing max_length involves a practical tradeoff. Longer sequences capture more context but require more memory, limiting batch size. For sentiment classification of movie reviews, most reviews fit comfortably within 128 tokens. For longer documents, such as legal contracts or scientific papers, you would need larger max lengths, which come at the cost of smaller batch sizes or more GPU memory. This tradeoff between context length and computational cost recurs throughout transformer-based NLP.

Fine-tuning hyperparameters differ substantially from pre-training settings. Pre-training uses large batch sizes, high learning rates, and millions of steps. Fine-tuning uses smaller batch sizes, much lower learning rates, and only a few epochs. This approach prevents the pre-trained knowledge from being overwritten too quickly, preserving the model's existing capabilities while letting task-specific adaptation. The philosophy underlying these choices shows a basic insight: we are not learning from scratch, but rather making targeted adjustments to an already capable model.

The configuration choices for fine-tuning encode decades of accumulated empirical wisdom. The specific values, such as learning rates in the $10^{-5}$ range and warmup ratios of 0.06 to 0.1, were not derived from first principles but were discovered through extensive experimentation. The original BERT paper by Devlin et al. (2018) ran systematic experiments across multiple fine-tuning tasks and reported these ranges as reliably effective. Subsequent research has largely confirmed these findings, with some task-specific adjustments for very large or very small datasets.

These settings stand for a reasonable starting point for fine-tuning BERT on a classification task. The learning rate of $2 \times 10^{-5}$ is roughly 100 times smaller than typical pre-training learning rates. The warmup ratio of 0.1 means the learning rate will increase linearly for the first 10% of training steps before following the decay schedule. Each of these choices shows accumulated wisdom from thousands of fine-tuning experiments across the research community.

The fp16=True setting lets 16-bit floating-point precision during training, which roughly halves the memory requirements compared to 32-bit precision. This allows larger batch sizes on the same hardware. Modern GPUs have special hardware support for 16-bit operations, making them faster than 32-bit computations for many matrix operations. The tradeoff is a small risk of numerical instability due to reduced precision, which is why gradient scaling is applied automatically when fp16 training is enabled.

With the model loaded and the data prepared under the chosen configuration, training proceeds through the standard optimization loop. The Trainer API handles batching, gradient computation, parameter updates, and evaluation. Behind this convenient abstraction lies a advanced process that coordinates data loading, forward passes through the model, loss computation, backward passes for gradient calculation, and parameter updates according to the optimizer's rules.

Each training step involves the following sequence of operations. First, a batch of tokenized sequences is loaded from the training dataset. Second, the model processes the batch in a forward pass, computing representations at each layer until the classification head produces logits. Third, the cross-entropy loss between the logits and the true labels is computed. Fourth, gradients are computed via backpropagation, flowing from the loss through the classification head and then through every layer of the backbone. Fifth, the optimizer uses these gradients to update all parameters. This cycle repeats for every batch in every epoch.