In-Context Learning › ICL Scaling Behavior

Michael BrenndoerferUpdated July 27, 20256 min read
Language AI HandbookAll chapters
Part 28 of 67GPT Architecture
Chapter 200In-Context Learning › ICL Scaling Behavior
Reading section 5 of 11 · ICL Scaling Behavior · 6 minSection 5 of 11 · 6 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 200 of 513

Section 5 of 11 · 6 min read

In-Context Learning › ICL Scaling BehaviorOne section · Sections in this chapter5 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One of the most interesting aspects of in-context learning is how it scales. Both the number of examples and the model size affect performance, but the relationship is not simply additive.

More Examples Help (Up to a Point)

Adding more demonstrations generally improves performance, but with diminishing returns. The first few examples provide the largest gains, while additional examples contribute less.

Out[16]:
Visualization
Line plot showing accuracy vs number of examples for simple, medium, and complex tasks.
Performance as a function of demonstration count for different task types. Simple tasks saturate quickly, while complex tasks benefit from more examples. All curves show diminishing returns, and performance eventually plateaus due to context length limits or task ceiling.

The optimal number of examples depends on:

  • Task complexity: Simple tasks need fewer examples
  • Model capacity: Larger models extract more from each example
  • Example quality: High-quality examples provide more information per demonstration
  • Context budget: Longer examples leave less room for more demonstrations

Model Size Enables ICL

The most dramatic scaling effect is the relationship between model size and ICL capability. Small models show minimal benefit from demonstrations. As models grow larger, they become increasingly able to use in-context examples.

Out[17]:
Visualization
Line plot showing ICL improvement over zero-shot for different model sizes from 100M to 175B parameters.
In-context learning capability as a function of model size. Smaller models (under 1B parameters) show minimal ICL ability, treating demonstrations almost as irrelevant context. ICL capability emerges around 1-10B parameters and continues improving with scale.

This emergence pattern changes the interpretation. It suggests that in-context learning represents a qualitatively different capability that activates at enough scale, rather than a smooth scaling up of pattern matching. Models below a certain threshold don't just do ICL poorly; they essentially don't do it at all.

The Interplay of Scale and Examples

Model size and example count interact in interesting ways. Larger models:

  • Extract more information from each example
  • Continue improving with more examples for longer
  • Generalize better from diverse demonstrations
  • Are more reliable to example ordering and selection
Out[18]:
Visualization
Multi-line plot showing performance curves for different model sizes as examples increase.
Interaction between model size and number of in-context examples. Larger models show steeper improvement curves and higher performance ceilings. The gap between zero-shot and few-shot widens with scale, showing that larger models leverage demonstrations more effectively.

The practical takeaway: if you're working with a smaller model, investing heavily in example selection and count may not pay off. But with large models, careful prompt engineering with well-chosen examples can yield substantial gains.

Out[19]:
Visualization
Contour plot showing ICL accuracy as function of model parameters and demonstration count.
ICL performance landscape across model sizes and example counts. Contour lines show constant accuracy levels. Small models (bottom) show minimal improvement regardless of example count. Large models (top) achieve higher performance and continue improving with more examples. The diagonal pattern indicates that model scale and example count are complementary: larger models extract more value from each additional demonstration.