In-Context Learning › What Is In-Context Learning?
Language AI HandbookAll chapters In-Context Learning › What Is In-Context Learning?One section · Sections in this chapter2 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
In-context learning refers to a language model's ability to learn new tasks from examples provided in the input prompt, without updating any model parameters. The model "learns" the task pattern by conditioning on the demonstrations and applies that pattern to new inputs.
In-context learning (ICL) is the ability of a pretrained language model to perform a task by conditioning on a few input-output examples (demonstrations) in the prompt, without any gradient-based training or weight updates. The model infers the task from the examples and generalizes to new inputs.
Consider a translation task. Instead of fine-tuning a model on thousands of parallel sentences, you can provide a few examples directly in the prompt:
English: The weather is beautiful today.
French: Le temps est magnifique aujourd'hui.
English: I would like a cup of coffee.
French: Je voudrais une tasse de café.
English: Where is the nearest train station?
French:
The model completes this with "Où est la gare la plus proche?" having inferred the translation pattern from just two examples. No training occurred. The model's weights remain unchanged. Yet it performs the task correctly.
This behavior was surprising because traditional machine learning assumes you need gradient updates to learn. The model must see many examples, compute a loss, and adjust its parameters. ICL breaks this assumption: learning happens within a single forward pass through the network.
The Three Regimes of Prompting
The GPT-3 paper formalized three prompting regimes based on how many examples are provided:
def create_icl_prompt(task_description, examples, test_input):
"""
Create an in-context learning prompt with demonstrations.
Args:
task_description: Optional natural language description of the task
examples: List of (input, output) tuples as demonstrations
test_input: The input to be completed by the model
Returns:
Formatted prompt string
"""
parts = []
if task_description:
parts.append(task_description)
parts.append("")
for inp, out in examples:
parts.append(f"Input: {inp}")
parts.append(f"Output: {out}")
parts.append("")
parts.append(f"Input: {test_input}")
parts.append("Output:")
return "\n".join(parts)
# Zero-shot: Task description only, no examples
zero_shot = create_icl_prompt(
task_description="Classify the sentiment as Positive or Negative.",
examples=[],
test_input="The movie was a complete waste of time.",
)
# One-shot: Single demonstration
one_shot = create_icl_prompt(
task_description=None,
examples=[("I love this product!", "Positive")],
test_input="The movie was a complete waste of time.",
)
# Few-shot: Multiple demonstrations
few_shot = create_icl_prompt(
task_description=None,
examples=[
("I love this product!", "Positive"),
("Terrible customer service.", "Negative"),
("Exceeded all my expectations.", "Positive"),
("Would not recommend to anyone.", "Negative"),
],
test_input="The movie was a complete waste of time.",
)================================================== ZERO-SHOT (task description only) ================================================== Classify the sentiment as Positive or Negative. Input: The movie was a complete waste of time. Output: ================================================== ONE-SHOT (1 example) ================================================== Input: I love this product! Output: Positive Input: The movie was a complete waste of time. Output: ================================================== FEW-SHOT (4 examples) ================================================== Input: I love this product! Output: Positive Input: Terrible customer service. Output: Negative Input: Exceeded all my expectations. Output: Positive Input: Would not recommend to anyone. Output: Negative Input: The movie was a complete waste of time. Output:
Each regime has distinct characteristics:
- Zero-shot: The model relies entirely on its pretrained knowledge and the task description. Works best for common tasks the model encountered during pretraining.
- One-shot: A single example clarifies the expected format and task. Often sufficient for simple classification or formatting tasks.
- Few-shot: Multiple examples help the model distinguish between classes, understand edge cases, and calibrate its confidence. Typically 4-32 examples, limited by context length.
The number of examples you can provide is constrained by the model's context window. With a 2,048 token limit (GPT-3) or 8,192+ tokens (later models), you must balance demonstration count against prompt length.