In-Context Learning Emergence › In-Context Learning Emergence

In Part XXVIII, we introduced in-context learning (ICL) as a remarkable capability of large language models: the ability to perform new tasks simply by observing a few demonstrations in the prompt, without any gradient updates. But when we introduced ICL, we treated it as a feature of GPT-3 and its successors. What we didn't fully address was the mystery of when and how this capability appears.

ICL doesn't exist at small scales. Train a language model with 100 million parameters on the same data, with the same architecture, and you'll find something peculiar. The model doesn't learn from examples in the prompt. It will generate text, predict next tokens, and even produce coherent sentences, but showing it examples of a task at inference time has little effect on its behavior. Then, somewhere between 1 billion and 100 billion parameters, something changes. The model starts using those examples. It begins to generalize from demonstrations it has never seen during training to produce correct outputs for novel inputs.

This emergence of in-context learning is one of the most consequential transitions in the scaling of language models. It changed LLMs from pattern-matching text generators into flexible few-shot learners, fundamentally changing how we interact with AI systems. Understanding when this transition occurs, how it scales compared to traditional fine-tuning, and what mechanisms might explain it remains an active area of research with measurable implications for both theory and practice.