Fine-tuning Data Efficiency › Small Data Strategies

When working with limited data, several strategies maximize your chances of success. The guiding principle across all of them is the same: every labeled example is precious, so the goal is to extract as much signal as possible from what you have while avoiding the pitfalls of overfitting to noise.

Choose pre-trained models whose training data aligns with your target domain. A model pre-trained on scientific papers will be more sample-efficient for scientific text classification than a model trained on web text. This domain alignment gives stronger priors for the specific vocabulary and style of your target task.

The importance of this choice is easy to underestimate. A general model may have seen very few examples of clinical abbreviations, legal boilerplate, or domain-specific jargon during pre-training. As a result, its representations for these terms are weak or borrowed from superficially similar everyday language. A domain-specific model that was exposed to thousands of such examples during pre-training arrives with far richer priors, meaning your few-shot examples only need to refine existing knowledge rather than build it from scratch. Resources like BioBERT for biomedical text, LegalBERT for legal documents, and FinBERT for financial text exist precisely because domain alignment matters for sample efficiency.

When possible, reformulate your task to better match the pre-training objective. Classification as fill-in-the-blank (PET), question answering as span extraction, and other reformulations can sharply improve few-shot performance by using what the model already knows how to do.

With limited data, overfitting is the primary enemy. Apply strong regularization: high dropout, weight decay, early stopping based on validation loss (even with few validation examples), and data augmentation. It's better to underfit slightly than to memorize the few training examples.

Train multiple models with different random seeds and combine their predictions. This reduces variance and gives more stable results than any single model. With few-shot data, the specific random initialization and training order can materially impact results.

Sometimes it's better to use a larger pre-trained model with fewer labeled examples than a smaller model with more labels. The next part of this book covers parameter-efficient fine-tuning methods like LoRA that make it practical to adapt very large models with limited computational resources.

When you can only afford to label a small number of examples, the quality and representativeness of those examples matters more than it does at scale. A carefully chosen set of 50 examples that covers the space of variation in your task can outperform 200 hastily chosen examples concentrated in the easiest cases. Active learning, which selects the most informative examples to label based on model uncertainty, formalizes this intuition into an algorithmic strategy. Even without a full active learning setup, spending time reviewing your few-shot examples and so they span the full range of difficulty and surface forms in your target domain is time well invested.