Fine-tuning Data Efficiency › The Sample Efficiency Spectrum

Data efficiency in language AI spans a wide spectrum, from tasks requiring no task-specific data at all to those benefiting from millions of examples. Understanding where your problem falls on this spectrum helps you choose the right approach and set realistic expectations for what can be achieved with your available resources.

Zero-shot inference uses the model's pre-trained capabilities without any task-specific fine-tuning. As we saw with GPT-3's in-context learning in Part XXVIII, large language models can perform many tasks simply through careful prompting. The model generalizes from its pre-training data to new tasks, drawing on the large knowledge encoded in its parameters during the initial training phase.

Few-shot prompting gives a handful of examples in the prompt itself, guiding the model's behavior without updating any weights. This approach works well for tasks similar to what the model encountered during pre-training but doesn't create a specialized model. The examples serve as demonstrations that help the model understand the desired input-output relationship.

Few-shot fine-tuning updates model weights using a small labeled dataset, typically 10 to 1,000 examples. This creates a specialized model that often outperforms few-shot prompting, especially for tasks that differ materially from pre-training. By modifying the model's parameters, few-shot fine-tuning creates a more permanent adaptation than prompt-based approaches.

Standard fine-tuning uses moderate amounts of data, typically 1,000 to 100,000 examples. This is the regime where most practical fine-tuning occurs, balancing data collection costs against model performance. At this scale, practitioners can expect reliable, reproducible results with lower variance across training runs.

Large-scale fine-tuning applies when you have hundreds of thousands or millions of labeled examples. At this scale, even smaller pre-trained models can reach excellent results, and the benefits of larger pre-trained models diminish relative to smaller ones. The abundance of task-specific data can compensate for less advanced pre-training.

The key insight is that pre-training fundamentally changes the relationship between data quantity and model performance. A pre-trained BERT fine-tuned on 100 examples often outperforms a randomly initialized model trained on 10,000 examples. The representations learned during pre-training give such strong priors that far less task-specific data is needed. This shift in the learning curve is one of the most significant practical advances in modern NLP.