Fine-tuning Data Efficiency: Few-Shot Learning, Augmentation

One of the most useful properties of pre-trained language models is their ability to adapt to new tasks with surprisingly few examples. Where traditional machine learning often requires thousands or millions of labeled samples, a pre-trained transformer can reach strong performance with just dozens or hundreds. This chapter covers methods that improve data efficiency, from few-shot fine-tuning to text-specific data augmentation.

Understanding data efficiency is important for practical applications. Most real-world NLP problems don't come with large labeled datasets. Medical text classification might have only a few hundred expert-annotated examples. A new product category might have just a handful of customer reviews. Legal document analysis often relies on expensive attorney annotations. The ability to reach good performance with limited data often determines whether a project is viable.

Think of pre-training as giving a model an enormous head start. Before seeing a single labeled example, a pre-trained model already understands that "exquisite" and "superb" are semantically similar, that a sentence beginning with "Unfortunately, the service was..." is heading toward something negative, and that medical terms like "hypertension" and "myocardial infarction" belong to a specific domain. All of this knowledge was acquired for free from unlabeled text. When you then give even a handful of labeled examples, you are not teaching the model language from scratch. You are simply redirecting capabilities it already possesses toward a specific output format.

This has an important practical consequence. Data collection is often the most time-consuming and expensive part of building an NLP system. Expert annotators are costly, annotation tasks require detailed guidelines, and achieving consistent labeling across multiple annotators demands careful quality control. Any technique that reduces the number of labeled examples needed translates directly into faster development cycles, lower costs, and the ability to address problems that would otherwise be economically unviable. The methods in this chapter help practitioners do more with less.

Data efficiency in language AI spans a wide spectrum, from tasks requiring no task-specific data at all to those benefiting from millions of examples. Understanding where your problem falls on this spectrum helps you choose the right approach and set realistic expectations for what can be achieved with your available resources.

Zero-shot inference uses the model's pre-trained capabilities without any task-specific fine-tuning. As we saw with GPT-3's in-context learning in Part XXVIII, large language models can perform many tasks simply through careful prompting. The model generalizes from its pre-training data to new tasks, drawing on the large knowledge encoded in its parameters during the initial training phase.

Few-shot prompting gives a handful of examples in the prompt itself, guiding the model's behavior without updating any weights. This approach works well for tasks similar to what the model encountered during pre-training but doesn't create a specialized model. The examples serve as demonstrations that help the model understand the desired input-output relationship.

Few-shot fine-tuning updates model weights using a small labeled dataset, typically 10 to 1,000 examples. This creates a specialized model that often outperforms few-shot prompting, especially for tasks that differ materially from pre-training. By modifying the model's parameters, few-shot fine-tuning creates a more permanent adaptation than prompt-based approaches.

Standard fine-tuning uses moderate amounts of data, typically 1,000 to 100,000 examples. This is the regime where most practical fine-tuning occurs, balancing data collection costs against model performance. At this scale, practitioners can expect reliable, reproducible results with lower variance across training runs.

Large-scale fine-tuning applies when you have hundreds of thousands or millions of labeled examples. At this scale, even smaller pre-trained models can reach excellent results, and the benefits of larger pre-trained models diminish relative to smaller ones. The abundance of task-specific data can compensate for less advanced pre-training.

The key insight is that pre-training fundamentally changes the relationship between data quantity and model performance. A pre-trained BERT fine-tuned on 100 examples often outperforms a randomly initialized model trained on 10,000 examples. The representations learned during pre-training give such strong priors that far less task-specific data is needed. This shift in the learning curve is one of the most significant practical advances in modern NLP.

Few-shot fine-tuning adapts a pre-trained model using only a handful of labeled examples per class. This setting is particularly challenging because small datasets are prone to overfitting, and variance across different training runs can be high. The basic tension in few-shot learning lies between extracting maximum information from limited examples while avoiding the trap of memorizing noise or spurious patterns.

Pattern-Exploiting Training (PET) reformulates classification tasks as cloze-style fill-in-the-blank problems that match the pre-training objective. Instead of training a separate classification head, PET uses the model's masked language modeling capability. This approach reuses the skill the model developed during pre-training rather than asking it to learn an entirely new type of prediction.

Consider sentiment classification. Rather than adding a classification layer that must learn from scratch how to map hidden states to class probabilities, PET turns the input into a format the model already understands:

The model predicts probabilities for tokens in the mask position. We map specific tokens (verbalizers) to class labels. Since the model was pre-trained on exactly this kind of task, it can make reasonable predictions even with very few examples. The model already knows that after describing something as "terrible," the word "bad" is more likely than "great" in the masked position, giving it a strong prior even before seeing any task-specific training data.

The pattern design materially impacts performance because it determines how well the classification task aligns with what the model learned during pre-training. Effective patterns share several characteristics:

• They sound natural and fluent, resembling text the model would have encountered during pre-training.
• They match the pre-training distribution, using constructions common in the training corpus.
• They place the mask in a semantically appropriate position where the predicted word indicates the class.
• They use verbalizers that clearly distinguish classes, selecting words whose meanings unambiguously correspond to each label.

SetFit (Sentence Transformer Fine-tuning) takes a different approach, using contrastive learning to maximize data efficiency. Rather than reformulating the task to match pre-training, SetFit amplifies the available training signal through strategic pair generation. The method works in two stages that together transform a small set of labeled examples into a much richer training signal.

Stage 1: Contrastive fine-tuning. Generate pairs of examples from the few-shot dataset. Pairs with the same label are positive pairs; pairs with different labels are negative pairs. Fine-tune a sentence transformer to produce similar embeddings for positive pairs and dissimilar embeddings for negative pairs. This stage teaches the model which examples should be grouped together in the embedding space.

Stage 2: Classification head training. Use the fine-tuned sentence transformer to embed all training examples. Train a simple classifier (logistic regression or small neural network) on these embeddings. Because the embeddings now cluster by class, even a simple classifier can reach strong performance.

SetFit's power comes from data amplification through pairing. With $n$ examples per class and $k$ classes, you can generate $\binom{n}{2} \cdot k$ positive pairs and many more negative pairs. Even 8 examples per class yields hundreds of training pairs for the contrastive objective. This combinatorial explosion turns a seemingly inadequate training set into one rich enough for real learning.

Few-shot fine-tuning exhibits high variance across runs, which is one of its most challenging aspects. Different random seeds for weight initialization, data ordering, or the specific examples selected can sharply change results. A model that reaches 90% accuracy in one training run might only reach 70% in another, even with identical hyperparameters. Several techniques improve stability and make results more reliable.

The key insight is that with very few examples, the training signal is inherently noisy. When you have only eight labeled examples, one mislabeled or unrepresentative example constitutes 12.5% of your entire training set. This means the model's behavior can be dominated by a single data point in ways that would be washed out in a larger dataset. Recognizing this fragility is the first step toward mitigating it.

Multiple random restarts train the model multiple times with different seeds and either ensemble the predictions or select the best model based on a small validation set. This reduces the risk of unlucky initialization and gives a more representative picture of achievable performance.

Training set sampling creates multiple few-shot subsets from a larger pool. If you have 100 labeled examples but want to simulate 10-shot learning, create multiple 10-shot subsets and train on each. This helps identify reliable patterns that persist across different example selections rather than patterns specific to one particular subset.

Gradual unfreezing starts by training only the classification head, then progressively unfreezes deeper layers. This prevents early training instability from corrupting pre-trained representations. By letting the classification head to stabilize before introducing changes to earlier layers, the model maintains the useful features learned during pre-training.

As we discussed in the previous chapter on fine-tuning learning rates, using lower learning rates is particularly important in few-shot settings. The limited data gives weak gradient signals, so aggressive updates can quickly overwrite useful pre-trained knowledge. A learning rate that works well with 10,000 examples might be catastrophically high with 100 examples.

In practice, it is worth treating the variance in few-shot results as a measurement problem rather than a model problem. If you train five models with different random seeds and observe accuracy ranging from 72% to 91%, you have learned something important: the best performance achievable with this data and model combination is around 91%, but that result is not reliably reproducible without ensembling or careful seed selection. Reporting the mean and standard deviation across multiple runs, rather than cherry-picking the best single run, gives an honest picture of what to expect when the system is deployed on new data.

Data augmentation artificially expands the training set by creating modified versions of existing examples. While augmentation has been especially effective for computer vision (random crops, rotations, color changes), text augmentation is more challenging because small changes can alter meaning sharply. Changing a single word can flip the sentiment of a sentence, and rearranging phrases can make text ungrammatical. The goal is to introduce real variation while preserving the label's validity.

The simplest augmentation replaces words with synonyms:

This preserves the overall meaning while introducing lexical variation. The model learns that both "quick" and "fast," or "jumps" and "leaps," can express similar concepts, reducing its dependence on specific vocabulary. However, synonym replacement has limitations: not all words have good synonyms, context matters (a "bank" isn't always replaceable with "shore"), and some domains have specialized vocabulary where synonyms don't exist.

Back-translation generates paraphrases by translating text to another language and back:

The translation process naturally introduces variation in word choice and sentence structure while typically preserving meaning. Each translation step makes decisions about how to express concepts, and these decisions may differ from the original phrasing. Using multiple intermediate languages generates more diverse paraphrases, as each language brings its own structural biases and vocabulary choices to the translation process.

Several random operations can augment text data, each introducing a different kind of variation.

Random insertion adds random words from the vocabulary into sentences. While this might seem likely to introduce noise, it encourages the model to focus on key words rather than word order. The model learns to identify important content words even when surrounded by distractors.

Random swap exchanges the positions of two words in a sentence. Again, this seems potentially harmful, but it helps models tolerate minor ordering variations. Many classification tasks do not depend on exact word order, and this augmentation reflects that fact.

Random deletion removes words with some probability. This simulates incomplete or noisy input and prevents over-reliance on specific words. If a model can still classify correctly with some words missing, it has learned more reliable patterns than one requiring all words present.

These random operations work best when applied conservatively. Augmenting too aggressively can create examples that are too different from real data, training the model on a distribution that doesn't match what it will encounter at test time.

Mixup, originally developed for images, creates new training examples by interpolating between existing ones. The intuition behind mixup is simple but powerful: by training on blended examples with soft labels, we encourage the model to behave linearly in the space between training points, leading to smoother decision boundaries. For text, this is less straightforward since discrete tokens cannot be interpolated directly. You cannot average the word "happy" and "sad" to get a word that is half-positive. Instead, text mixup operates in embedding space, where continuous representations make interpolation real:

Given two examples $(x_1, y_1)$ and $(x_2, y_2)$, we create a synthetic example by interpolating their embeddings and labels:

where:

• $\tilde{e}$: the embedding vector for the new synthetic example
• $\tilde{y}$: the label for the new example (a "soft" target)
• $\lambda$: the mixing coefficient sampled from a Beta distribution ($\lambda \in [0, 1]$), controlling the ratio between the two examples
• $e_1, e_2$: the embeddings of the original examples (typically [CLS] token embeddings or pooled representations)
• $y_1, y_2$: the labels of the original examples

The first equation creates a new embedding that lies somewhere on the line segment connecting the two original embeddings in the high-dimensional space. When $\lambda = 1$, we recover the first example exactly; when $\lambda = 0$, we recover the second. Intermediate values produce embeddings that blend characteristics of both inputs.

The second equation applies the same interpolation to the labels. This is what makes mixup so distinctive: rather than assigning a hard label to the synthetic example, we assign a soft target that shows the mixture. This process smooths the decision boundary in the embedding space by teaching the model that intermediate regions should have intermediate predictions. For instance, if a positive example ($y=1$) is mixed with a negative example ($y=0$) with $\lambda=0.7$, the target label becomes $0.7$, encouraging the model to reflect this ambiguity in its prediction confidence. The model learns that it should be more confident near pure examples and less confident in the interpolated regions between classes.

The Beta distribution for sampling $\lambda$ is chosen because it naturally produces values between 0 and 1, and its shape parameter controls how often extreme values (near 0 or 1) versus moderate values (near 0.5) are sampled. Lower shape parameters favor interpolations closer to the original examples, while higher values encourage more balanced mixing.

More advanced augmentation uses language models themselves to generate variations. Given a sentence, mask out one or more words and use a masked language model to predict replacements:

This approach generates contextually appropriate substitutions that maintain grammaticality better than random synonym replacement. The language model considers the full context when selecting a replacement word. This keeps the augmented word fits naturally in its surroundings. A word like "dull" is chosen because the model has learned that it appears in similar contexts to "boring."

Generative models can create even more diverse augmentations. Given a few examples of a class, a language model can generate entirely new examples in the same style, potentially multiplying the effective dataset size many times over. This approach can create examples that express the same sentiment or class membership through entirely different phrasings. This gives richer training signal than modifications of existing examples.

Why are pre-trained models so sample-efficient? Several interconnected factors contribute to this remarkable property, each building on the others to create models that can learn from minimal supervision.

Pre-training encodes large amounts of linguistic knowledge into model weights. The model learns syntax, semantics, factual knowledge, and reasoning patterns from billions of tokens of text. This knowledge acts as a strong prior that guides learning on new tasks. Rather than starting from a blank slate, the model begins with rich expectations about how language works.

Consider a sentiment classifier. A randomly initialized model must learn from scratch that "excellent" is positive and "terrible" is negative. It has no prior knowledge connecting these words to positive or negative sentiments. A pre-trained model already knows these associations from context. During pre-training, the model observed sentences like "The excellent performance earned applause" and "The terrible weather ruined the picnic," learning that "excellent" typically appears in positive contexts and "terrible" in negative ones. Fine-tuning merely teaches the model how to apply this existing knowledge to the classification format, connecting what it already knows to the specific output structure required.

As we discussed in Part XXXIV on transfer learning, pre-trained representations are highly reusable across different tasks. Lower layers encode general linguistic patterns like part-of-speech and syntax. These foundational features are useful for virtually any language task. Middle layers capture phrase-level semantics, understanding how words combine to form real units. Upper layers stand for more task-needed features that can be adapted to specific applications.

When fine-tuning on limited data, most of these features remain useful. The model doesn't need to learn basic language understanding from scratch; it only needs to learn task-specific adjustments on top of already-good representations. This is analogous to how a trained musician learning a new instrument doesn't need to relearn music theory or rhythm. They already possess foundational knowledge that transfers, and only need to learn the specifics of the new instrument.

Pre-training also gives implicit regularization that helps prevent overfitting in few-shot settings. The representations aren't perfectly aligned with the downstream task, which acts like noise injection. This imperfect alignment prevents the model from immediately memorizing the few training examples, forcing it to find more general patterns.

Also, the pre-trained classification token ([CLS] in BERT) isn't specialized for any particular task. This "neutral initialization" means the model must learn real class distinctions rather than exploiting spurious correlations that happen to align with the initialization. If the CLS token were initialized to produce outputs that accidentally correlated with the training labels, the model might find shortcuts rather than learning real features. The neutral initialization ensures the model must earn its performance through real learning.

Understanding the mechanisms behind sample efficiency also clarifies where the approach breaks down. Pre-training gives strong priors, but only for the types of patterns that were abundant in the pre-training corpus. If your task requires understanding highly specialized notation, recognizing subtle domain-specific discourse conventions, or making fine-grained distinctions that are rare in general text, the pre-training advantage weakens substantially.

Consider the difference between classifying movie reviews as positive or negative versus classifying radiology reports by severity of finding. Sentiment words like "excellent," "terrible," and "disappointing" appear constantly in web text, giving pre-trained models extremely strong priors. Radiology-specific severity distinctions ("mild bilateral basilar atelectasis" vs. "moderate consolidation suggesting pneumonia") appear far less frequently and require integrating specialized domain knowledge that most general pre-training corpora contain only sparsely. In the first case, a handful of examples can be sufficient. In the second case, the model may still need hundreds of domain-specific examples to build reliable representations, no matter how advanced the few-shot technique. Knowing which situation you are in before committing to a low-data strategy is a practical skill that separates successful deployments from frustrating ones.

Let's implement several data efficiency techniques, starting with data augmentation.

The dataset size has expanded by a factor of four. This gives materially more training signal. The examples show that the augmented versions retain the core sentiment of the original ("positive" or "negative") while introducing variations in phrasing, which helps prevent overfitting to specific words.

Now let's implement the core idea behind SetFit: generating contrastive pairs from few-shot data.

This shows how few-shot data can be amplified. From just 8 examples (4 per class), we generated many training pairs for contrastive learning.

Let's implement few-shot fine-tuning using the SetFit approach with a pre-trained sentence transformer.

Data augmentation typically yields a performance improvement by helping the model generalize from the limited examples. By seeing variations in phrasing and word choice, the classifier becomes less reliant on specific keywords and more focused on the underlying sentiment.

For a more advanced approach, let's implement contrastive fine-tuning on the sentence embeddings.

Let's examine how performance scales with training data size.

The key parameters used in these data efficiency techniques are:

• alpha_sr / alpha_ri / alpha_rs: Percentage of words to change for synonym replacement, random insertion, and random swap. The original EDA paper recommends values around 0.1 (10% of words), but this depends on sentence length. Short sentences should use lower values to avoid generating nonsensical text.
• p_rd: Probability of deleting a word in random deletion. Values between 0.05 and 0.15 work well in practice. Higher values produce more aggressive deletions that can break sentence structure.
• margin: Distance threshold in contrastive loss. Negative pairs are pushed apart until their cosine similarity drops below this margin. A margin of 0.5 is a common starting point, though domain-specific tasks may benefit from tuning this value based on how naturally separable the classes are.
• max_iter: Maximum iterations for the logistic regression solver. Increasing this value ensures convergence but adds computation time. With small datasets, 1,000 iterations is typically sufficient.
• num_aug: Number of augmented versions to generate per original example. Values between 2 and 8 are typical. The original EDA paper found that generating 4 augmented versions per example provided the best balance between expansion and noise introduction.

Notice that these parameters interact. Setting both alpha_sr and alpha_ri high simultaneously can produce examples that have changed so substantially that the label no longer applies. A sentence that was originally a clear positive review might become ambiguous or even negative after heavy simultaneous synonym replacement and random insertion. Conservative combinations of augmentation techniques are generally safer than aggressive application of any single one.

When working with limited data, several strategies maximize your chances of success. The guiding principle across all of them is the same: every labeled example is precious, so the goal is to extract as much signal as possible from what you have while avoiding the pitfalls of overfitting to noise.

Choose pre-trained models whose training data aligns with your target domain. A model pre-trained on scientific papers will be more sample-efficient for scientific text classification than a model trained on web text. This domain alignment gives stronger priors for the specific vocabulary and style of your target task.

The importance of this choice is easy to underestimate. A general model may have seen very few examples of clinical abbreviations, legal boilerplate, or domain-specific jargon during pre-training. As a result, its representations for these terms are weak or borrowed from superficially similar everyday language. A domain-specific model that was exposed to thousands of such examples during pre-training arrives with far richer priors, meaning your few-shot examples only need to refine existing knowledge rather than build it from scratch. Resources like BioBERT for biomedical text, LegalBERT for legal documents, and FinBERT for financial text exist precisely because domain alignment matters for sample efficiency.

When possible, reformulate your task to better match the pre-training objective. Classification as fill-in-the-blank (PET), question answering as span extraction, and other reformulations can sharply improve few-shot performance by using what the model already knows how to do.

With limited data, overfitting is the primary enemy. Apply strong regularization: high dropout, weight decay, early stopping based on validation loss (even with few validation examples), and data augmentation. It's better to underfit slightly than to memorize the few training examples.

Train multiple models with different random seeds and combine their predictions. This reduces variance and gives more stable results than any single model. With few-shot data, the specific random initialization and training order can materially impact results.

Sometimes it's better to use a larger pre-trained model with fewer labeled examples than a smaller model with more labels. The next part of this book covers parameter-efficient fine-tuning methods like LoRA that make it practical to adapt very large models with limited computational resources.

When you can only afford to label a small number of examples, the quality and representativeness of those examples matters more than it does at scale. A carefully chosen set of 50 examples that covers the space of variation in your task can outperform 200 hastily chosen examples concentrated in the easiest cases. Active learning, which selects the most informative examples to label based on model uncertainty, formalizes this intuition into an algorithmic strategy. Even without a full active learning setup, spending time reviewing your few-shot examples and so they span the full range of difficulty and surface forms in your target domain is time well invested.

Data augmentation and few-shot techniques have important limitations that practitioners must understand. Text augmentation, unlike image augmentation, can easily change meaning or introduce grammatical errors. Synonym replacement might substitute words with subtly different connotations ("slender" vs. "skinny"), and random operations can create awkward or nonsensical sentences. The quality of augmented data is often lower than real data, meaning there are diminishing returns to aggressive augmentation.

Few-shot fine-tuning also exhibits concerning instabilities. Results can vary sharply based on which specific examples are selected for training, the random seed used for initialization, and the order in which examples are presented. A model that appears excellent on one run might perform poorly on another. This variance is especially problematic when the "best" model is selected based on a small validation set, as the selection itself may just be fitting to noise.

Domain shift presents another challenge. Few-shot techniques work best when the target task is similar to what the model encountered during pre-training. For highly specialized domains with unique vocabulary, unusual document structures, or technical content unlike anything in the pre-training data, more labeled examples may be necessary regardless of the techniques applied.

There is also a subtler limitation worth understanding: few-shot methods tend to amplify whatever biases exist in the small training set. When a model is trained on thousands of examples, individual biases in labeling or example selection average out. When a model is trained on eight examples, a single annotator's idiosyncratic judgment can become a defining feature of the model's behavior. If the three positive examples in your training set all happen to contain exclamation marks, the model might learn to use punctuation as a proxy for positive sentiment, a spurious correlation that will fail on formal positive text. Careful curation of few-shot examples is good practice and a direct safeguard against this kind of bias amplification.

Despite these limitations, data efficiency techniques have transformed what is practically achievable with NLP. Tasks that once required months of annotation can now be tackled with days of labeling. This broader access allows smaller organizations, practitioners with limited budgets, and applications in data-scarce domains to benefit from state-of-the-art language AI. The combination of large pre-trained models with smart adaptation strategies has fundamentally shifted the trade-offs in building NLP systems. A solo researcher working on a niche domain problem now has tools that would have required a large team and substantial annotation budget just five years ago.

This chapter explored techniques for maximizing the value of limited labeled data in fine-tuning.

The sample efficiency spectrum ranges from zero-shot inference through few-shot fine-tuning to large-scale adaptation. Pre-trained models fundamentally change the relationship between data quantity and model performance, letting strong results with far fewer examples than traditional approaches require.

Few-shot fine-tuning methods like Pattern-Exploiting Training (PET) reformulate tasks to match pre-training objectives, while SetFit uses contrastive learning to amplify few-shot data through pair generation. These techniques can reach strong performance with just 8-32 labeled examples per class.

Data augmentation techniques for text include synonym replacement, back-translation, random operations (deletion, swap, insertion), embedding-space mixup, and contextual augmentation using language models. While less straightforward than image augmentation, these methods can substantially expand small datasets.

Sample efficiency patterns emerge from pre-training, which gives strong linguistic priors that reduce the need for task-specific data. Features learned from large unlabeled corpora transfer effectively to downstream tasks with minimal adaptation.

Small data strategies emphasize domain-aligned model selection, task reformulation, aggressive regularization, ensembling across random seeds, and considering the trade-off between label efficiency and model size.

The next part explores parameter-efficient fine-tuning methods like LoRA, which extend these efficiency principles to model adaptation itself, letting large models to be customized with minimal computational cost and memory overhead.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about fine-tuning data efficiency and few-shot learning techniques.