Fine-tuning Data Efficiency › Few-Shot Fine-Tuning

Few-shot fine-tuning adapts a pre-trained model using only a handful of labeled examples per class. This setting is particularly challenging because small datasets are prone to overfitting, and variance across different training runs can be high. The basic tension in few-shot learning lies between extracting maximum information from limited examples while avoiding the trap of memorizing noise or spurious patterns.

Pattern-Exploiting Training (PET) reformulates classification tasks as cloze-style fill-in-the-blank problems that match the pre-training objective. Instead of training a separate classification head, PET uses the model's masked language modeling capability. This approach reuses the skill the model developed during pre-training rather than asking it to learn an entirely new type of prediction.

Consider sentiment classification. Rather than adding a classification layer that must learn from scratch how to map hidden states to class probabilities, PET turns the input into a format the model already understands:

The model predicts probabilities for tokens in the mask position. We map specific tokens (verbalizers) to class labels. Since the model was pre-trained on exactly this kind of task, it can make reasonable predictions even with very few examples. The model already knows that after describing something as "terrible," the word "bad" is more likely than "great" in the masked position, giving it a strong prior even before seeing any task-specific training data.

The pattern design materially impacts performance because it determines how well the classification task aligns with what the model learned during pre-training. Effective patterns share several characteristics:

• They sound natural and fluent, resembling text the model would have encountered during pre-training.
• They match the pre-training distribution, using constructions common in the training corpus.
• They place the mask in a semantically appropriate position where the predicted word indicates the class.
• They use verbalizers that clearly distinguish classes, selecting words whose meanings unambiguously correspond to each label.

SetFit (Sentence Transformer Fine-tuning) takes a different approach, using contrastive learning to maximize data efficiency. Rather than reformulating the task to match pre-training, SetFit amplifies the available training signal through strategic pair generation. The method works in two stages that together transform a small set of labeled examples into a much richer training signal.

Stage 1: Contrastive fine-tuning. Generate pairs of examples from the few-shot dataset. Pairs with the same label are positive pairs; pairs with different labels are negative pairs. Fine-tune a sentence transformer to produce similar embeddings for positive pairs and dissimilar embeddings for negative pairs. This stage teaches the model which examples should be grouped together in the embedding space.

Stage 2: Classification head training. Use the fine-tuned sentence transformer to embed all training examples. Train a simple classifier (logistic regression or small neural network) on these embeddings. Because the embeddings now cluster by class, even a simple classifier can reach strong performance.

SetFit's power comes from data amplification through pairing. With $n$ examples per class and $k$ classes, you can generate $\binom{n}{2} \cdot k$ positive pairs and many more negative pairs. Even 8 examples per class yields hundreds of training pairs for the contrastive objective. This combinatorial explosion turns a seemingly inadequate training set into one rich enough for real learning.

Few-shot fine-tuning exhibits high variance across runs, which is one of its most challenging aspects. Different random seeds for weight initialization, data ordering, or the specific examples selected can sharply change results. A model that reaches 90% accuracy in one training run might only reach 70% in another, even with identical hyperparameters. Several techniques improve stability and make results more reliable.

The key insight is that with very few examples, the training signal is inherently noisy. When you have only eight labeled examples, one mislabeled or unrepresentative example constitutes 12.5% of your entire training set. This means the model's behavior can be dominated by a single data point in ways that would be washed out in a larger dataset. Recognizing this fragility is the first step toward mitigating it.

Multiple random restarts train the model multiple times with different seeds and either ensemble the predictions or select the best model based on a small validation set. This reduces the risk of unlucky initialization and gives a more representative picture of achievable performance.

Training set sampling creates multiple few-shot subsets from a larger pool. If you have 100 labeled examples but want to simulate 10-shot learning, create multiple 10-shot subsets and train on each. This helps identify reliable patterns that persist across different example selections rather than patterns specific to one particular subset.

Gradual unfreezing starts by training only the classification head, then progressively unfreezes deeper layers. This prevents early training instability from corrupting pre-trained representations. By letting the classification head to stabilize before introducing changes to earlier layers, the model maintains the useful features learned during pre-training.

As we discussed in the previous chapter on fine-tuning learning rates, using lower learning rates is particularly important in few-shot settings. The limited data gives weak gradient signals, so aggressive updates can quickly overwrite useful pre-trained knowledge. A learning rate that works well with 10,000 examples might be catastrophically high with 100 examples.

In practice, it is worth treating the variance in few-shot results as a measurement problem rather than a model problem. If you train five models with different random seeds and observe accuracy ranging from 72% to 91%, you have learned something important: the best performance achievable with this data and model combination is around 91%, but that result is not reliably reproducible without ensembling or careful seed selection. Reporting the mean and standard deviation across multiple runs, rather than cherry-picking the best single run, gives an honest picture of what to expect when the system is deployed on new data.