GPT-3 & In-Context Learning (2020) › The Solution

GPT-3 addressed these problems through a simple but radical approach: scale the autoregressive language model to 175 billion parameters and demonstrate that in-context learning could match or exceed fine-tuned performance across many tasks without any gradient-based adaptation. The solution relied on the insight that during pretraining on diverse internet text, the model encountered countless structured task examples embedded in the text. At sufficient scale, the model could learn to recognize these patterns and apply them appropriately when similar patterns appeared in its input context.

The architecture of GPT-3 was based on the transformer decoder used in GPT-1 and GPT-2, but scaled to unprecedented size. The model consisted of 96 transformer layers with 175 billion total parameters. Each layer contained multi-head self-attention mechanisms and feedforward networks, with layer normalization and residual connections. The model used the same autoregressive objective as previous GPT models: given a sequence of tokens, predict the next token. This simple objective, when applied at massive scale with diverse training data, produced a model with remarkable emergent capabilities.

The training data for GPT-3 was far larger and more diverse than previous models. The training corpus included Common Crawl, web texts, books, Wikipedia, and other text sources, totaling hundreds of billions of tokens. This diverse corpus exposed the model to an enormous variety of text formats, including question-answer pairs, translations, code examples, mathematical problems, explanations, and countless other structured formats. During training, the model learned patterns such as how questions are typically answered, how translations map between languages, how code functions are structured, and how explanations follow examples.

The key innovation was demonstrating that few-shot learning, where the model sees several examples of a task in its prompt, could achieve performance comparable to fine-tuned models. In few-shot learning, the prompt contains several input-output examples demonstrating the task, followed by a new input for which the model should generate the output. For example, a sentiment analysis prompt might contain several examples like "The movie was fantastic! -> positive" and "I hated this book. -> negative", followed by "The weather is nice today. ->". The model would recognize the pattern and generate "positive".

The researchers evaluated GPT-3 across three learning paradigms: zero-shot, one-shot, and few-shot. In zero-shot learning, the model receives only a natural language task description with no examples. In one-shot learning, the model receives one example. In few-shot learning, the model receives several examples. The results showed that performance improved dramatically from zero-shot to one-shot to few-shot, with few-shot learning often matching or exceeding fine-tuned baseline models.

The model's ability to perform in-context learning emerged from the diversity and scale of its training data. Because the training corpus contained so many structured task examples, the model learned to recognize them during pretraining. When similar patterns appeared in the input prompt, the model's attention mechanisms could identify the relevant context and apply the learned patterns to generate appropriate continuations. This process required no gradient updates or weight changes; it was purely a function of the model's learned representations and how it processed the prompt.

The model's scale enabled this capability. With 175 billion parameters, GPT-3 had sufficient capacity to store and recognize an enormous variety of task structures and textual patterns. Smaller models trained on the same data might recognize some patterns but would lack the capacity to reliably recognize and apply the diverse patterns needed for effective in-context learning across many different tasks.

The training process itself used standard autoregressive language modeling, but at unprecedented scale. The model learned to maximize the likelihood of each token given previous tokens across hundreds of billions of tokens of diverse text. This training objective, simple in principle, forced the model to develop sophisticated internal representations of language structure, task formats, reasoning patterns, and world knowledge that enabled it to perform diverse tasks when prompted appropriately.

The researchers also developed techniques for effectively using GPT-3 through prompt engineering. The format of examples, the number of examples, and the way tasks were described all influenced performance. For best results, examples needed to be clearly formatted, representative of the task, and sufficient in number. This led to the development of systematic approaches to prompt design that maximized the model's in-context learning capabilities.