GPT-3 & In-Context Learning (2020) › The Problem

Despite the remarkable success of the transfer learning paradigm established by GPT-1, GPT-2, and BERT, several significant problems remained unresolved by 2020. The fine-tuning approach that had become standard required collecting labeled training data for each new task, which was expensive, time-consuming, and often impractical. For many applications, sufficient labeled data might not exist, and creating it required domain expertise and significant human effort. This requirement created a bottleneck that limited the speed at which language models could be adapted to new tasks and domains.

Consider a scenario where a developer wanted to build a system that could extract information from medical records, translate technical documentation, answer questions about legal contracts, and generate creative writing prompts. With the fine-tuning paradigm, each of these tasks would require collecting hundreds or thousands of labeled examples, carefully preparing training data, fine-tuning separate models or task-specific heads, and then deploying multiple models. This process could take weeks or months for each task, making it difficult to rapidly prototype new applications or adapt to changing requirements.

The fine-tuning approach also suffered from a problem known as "catastrophic forgetting," where adapting a pretrained model to a new task could degrade its performance on previous tasks. If an organization had fine-tuned a GPT model for sentiment analysis and then wanted to fine-tune it again for question answering, the second fine-tuning process might cause the model to forget what it learned about sentiment analysis. This limitation meant that organizations often needed to maintain separate fine-tuned models for each task, multiplying computational costs and infrastructure complexity.

The computational cost of fine-tuning was another significant barrier. While pretrained models could be used by downloading pre-computed weights, fine-tuning required running backpropagation on task-specific datasets, which demanded substantial computational resources. For resource-constrained organizations or researchers, the cost of fine-tuning large models could be prohibitive, limiting access to state-of-the-art language understanding capabilities.

GPT-2 had demonstrated promising zero-shot capabilities, where the model could perform tasks based on natural language task descriptions without any task-specific training. However, these capabilities were unreliable and inconsistent. Performance varied dramatically depending on how the task was described, what examples were included in the prompt, and the specific formulation of the problem. Zero-shot performance also typically lagged significantly behind fine-tuned models, making it unsuitable for production applications requiring high accuracy.

The unreliability of zero-shot learning created a fundamental tension. On one hand, researchers recognized that large language models seemed to have learned to recognize structured task examples during pretraining, enabling them to perform tasks without explicit training. On the other hand, this capability was too unreliable to be practically useful for most applications. The question became whether this capability could be made more reliable through better prompt design, more examples, or simply by scaling the model further.

There was also a deeper question about what language models were learning during pretraining. If a model trained only to predict the next token in text could perform diverse tasks when prompted appropriately, what kind of knowledge or capabilities had emerged from this simple training objective? Answering this question would help researchers develop more capable models and determine when to apply them.

The field needed a demonstration that would clarify whether in-context learning was a practical alternative to fine-tuning, whether it could achieve comparable or better performance, and under what conditions it would be reliable. GPT-3 was designed to provide definitive answers to these questions by scaling the model to an unprecedented size and systematically evaluating in-context learning across diverse tasks and settings.