GPT-3 › GPT-3: Scale, Few-Shot Learning, and In-Context Adaptation

GPT-2 demonstrated that language models could perform tasks without task-specific training, achieving modest zero-shot performance on reading comprehension alongside translation and summarization. But this capability was limited: the model often required fine-tuning to match specialized systems, and zero-shot performance lagged well behind purpose-built models trained on labeled data. For most real-world applications, fine-tuning remained the dominant paradigm. You would take a pretrained model, collect thousands of labeled examples for your specific task, and spend GPU time adapting the model's weights to your domain. GPT-3 changed this equation through sheer scale.

With 175 billion parameters, over 100 times larger than GPT-2, GPT-3 revealed that massive language models could learn new tasks from just a few examples provided in the prompt, a capability called few-shot learning. The model did not need gradient updates, additional training, or labeled datasets. You simply described the task and showed a handful of demonstrations inside the prompt, and the model completed the pattern. This was not a minor incremental improvement. It was a qualitative shift in how language models could be used.

This discovery reshaped how the field thinks about language model capabilities. Instead of training separate models for each task, a single pretrained model could adapt on the fly. Prompting replaced fine-tuning for many applications. Practitioners who once spent weeks collecting and labeling data for a new task could instead spend minutes crafting a good prompt. The economic and practical implications were enormous. The implications also extended beyond NLP: GPT-3 demonstrated that scale itself could be a source of emergent capabilities, behaviors that appear suddenly as models grow larger rather than improving smoothly and gradually with size.

The paper introducing GPT-3, "Language Models are Few-Shot Learners" by Brown et al. (2020), became one of the most influential papers in the history of natural language processing. It presented systematic empirical evidence that performance across dozens of diverse NLP benchmarks scaled predictably with model size and the number of in-context examples. It introduced rigorous definitions distinguishing zero-shot, one-shot, and few-shot evaluation. And it raised fundamental questions about what it means for a model to "learn" a task, questions that researchers are still working through today.

In this chapter, we explore GPT-3's architecture and training, the discovery of in-context learning, the difference between zero-shot, one-shot, and few-shot prompting, and what GPT-3 revealed about the relationship between scale and capability. We also examine the limitations that scale did not solve and the lasting impact GPT-3 had on research directions and industry practice.