GPT-3 › The Discovery of In-Context Learning
Language AI HandbookAll chapters GPT-3 › The Discovery of In-Context LearningOne section · Sections in this chapter5 / 25 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
GPT-3's most significant contribution wasn't its architecture but what it revealed about large-scale language models: they can learn new tasks from examples provided in the prompt, without any gradient updates. This capability, called in-context learning (ICL), emerged as a surprise during evaluation.
In-context learning is the ability of a language model to perform a task by conditioning on a few examples (demonstrations) in the input prompt, without updating the model's parameters. The model "learns" the task pattern from the examples and applies it to new inputs.
To appreciate how surprising this was, recall the standard pre-GPT-3 paradigm. When you wanted a model to perform sentiment classification, you collected thousands of labeled reviews, fine-tuned a BERT-style model on them, and deployed the resulting specialized model. This process required labeled data (expensive to collect), GPU time (expensive to run), and engineering infrastructure (expensive to maintain). And you had to repeat the entire process for every new task.
In-context learning throws this paradigm out. Instead of updating the model's weights, you modify the input. You write a few examples of the task directly into the prompt, and the model infers what you want from context. The weights never change. No training loop runs. The adaptation happens purely in the model's forward pass, through the same attention mechanism it uses for everything else.
Consider a sentiment classification task. Instead of fine-tuning a model on thousands of labeled examples, you can simply show GPT-3 a few examples in the prompt:
Review: "This movie was absolutely fantastic!"
Sentiment: Positive
Review: "I wasted two hours of my life on this garbage."
Sentiment: Negative
Review: "The acting was decent but the plot made no sense."
Sentiment:
GPT-3 would then complete this with "Negative" or "Mixed", having inferred the classification pattern from just two examples. The key insight is that the model treats the prompt as part of its input context and continues the pattern it observes. It sees that each review is followed by a sentiment label, and when it encounters a review without a label, it generates one.
This seems almost magical at first. How can the model infer what to do without being trained on it? The answer lies in pretraining. GPT-3 has seen billions of examples of people explaining concepts. This provides examples, and asking others to complete patterns. The format of a few-shot prompt, while novel as a machine learning technique, resembles things GPT-3 has encountered many times: educational texts, quizzes, worked examples, and teaching materials where an instructor shows examples and expects the student to continue the pattern. In-context learning may be less a new capability and more the application of a deeply learned skill, the skill of learning from examples, to the machine learning context.
Zero-Shot, One-Shot, and Few-Shot Learning
OpenAI's GPT-3 paper systematically compared three prompting strategies. Each represents a different level of guidance provided to the model:
- Zero-shot: The model receives only a task description, with no examples. It must understand the task from the description alone and generalize immediately.
- One-shot: The model sees one example before the test input. This provides a template for the expected input/output format, but offers minimal statistical signal about the task.
- Few-shot: The model sees multiple examples (typically 10-100, limited by context length). This provides both format information and statistical patterns about what kinds of inputs map to what kinds of outputs.
The distinction matters because it reveals different aspects of the model's capability. Zero-shot performance measures how well the model can understand natural language descriptions of tasks, essentially testing whether it has built sufficient meta-knowledge to interpret instructions. One-shot performance tests whether a single example is enough to fix the input/output format in the model's representation. Few-shot performance tests whether the model can pick up statistical regularities from a small sample, something closer to what we intuitively mean by "learning."
In practice, GPT-3's zero-shot performance was often impressive but inconsistent. Its one-shot performance was typically better across most tasks. And few-shot performance, with 10-50 examples filling the context window, was usually the strongest. This gradient from zero to few-shot suggested that each type of information, task description, format template, and statistical demonstration, provides complementary signal that the model can use.
def create_prompt(examples, test_input, task_description=None):
"""
Create a few-shot learning prompt.
Args:
examples: List of (input, output) tuples for demonstrations
test_input: The input to classify/complete
task_description: Optional zero-shot task description
Returns:
Formatted prompt string
"""
prompt_parts = []
if task_description:
prompt_parts.append(task_description + "\n")
for inp, out in examples:
prompt_parts.append(f"Input: {inp}")
prompt_parts.append(f"Output: {out}\n")
prompt_parts.append(f"Input: {test_input}")
prompt_parts.append("Output:")
return "\n".join(prompt_parts)
# Zero-shot example
zero_shot = create_prompt(
examples=[],
test_input="The food was cold and the service was slow.",
task_description="Classify the sentiment of the following review as Positive or Negative.",
)
# One-shot example
one_shot = create_prompt(
examples=[("Great product, highly recommend!", "Positive")],
test_input="The food was cold and the service was slow.",
)
# Few-shot example
few_shot = create_prompt(
examples=[
("Great product, highly recommend!", "Positive"),
("Terrible experience, never again.", "Negative"),
("Best purchase I've ever made!", "Positive"),
("Broken on arrival, what a waste.", "Negative"),
],
test_input="The food was cold and the service was slow.",
)============================================================ ZERO-SHOT PROMPT ============================================================ Classify the sentiment of the following review as Positive or Negative. Input: The food was cold and the service was slow. Output: ============================================================ ONE-SHOT PROMPT ============================================================ Input: Great product, highly recommend! Output: Positive Input: The food was cold and the service was slow. Output: ============================================================ FEW-SHOT PROMPT (4 examples) ============================================================ Input: Great product, highly recommend! Output: Positive Input: Terrible experience, never again. Output: Negative Input: Best purchase I've ever made! Output: Positive Input: Broken on arrival, what a waste. Output: Negative Input: The food was cold and the service was slow. Output:
Look at the difference in what each prompt gives the model. The zero-shot prompt contains the complete task description but leaves the model to infer what the expected output format should be. The one-shot prompt does not explain the task but demonstrates the format. The few-shot prompt provides four demonstrations spanning both classes, giving the model both format information and evidence about what "Positive" and "Negative" mean in this context.
The key insight from GPT-3's evaluation was that performance scaled predictably with both model size and number of examples:
This scaling pattern revealed something important: in-context learning is not simply pattern matching or memorization. It requires sufficient model capacity to represent the task and generalize from examples. Smaller models plateau quickly, while larger models continue to benefit from additional demonstrations. The 175B model shows particularly steep improvement as examples are added, suggesting it can extract and apply statistical regularities from the demonstrations in a way that smaller models cannot. This is consistent with the hypothesis that in-context learning is an emergent property, one that requires a certain threshold of model capacity before it becomes effective.