In-Context Learning Emergence › Key Parameters

The key parameters affecting ICL emergence and performance are:

• Model scale: Number of parameters, typically measured in billions. ICL emergence thresholds vary by task but generally require 1B+ parameters for basic tasks and 10B+ for complex reasoning.
• Context length: Maximum number of tokens the model can process. Longer contexts allow more demonstrations but increase computational cost.
• Number of demonstrations (k-shot): More examples generally improve performance up to context limits, with diminishing returns.
• Temperature: Controls output randomness during generation. Lower values (0.0-0.3) typically work better for ICL tasks requiring precise answers.
• Demonstration ordering: The sequence of examples can materially affect performance, with more recent examples often having stronger influence.

The emergence of in-context learning has changed how we build AI systems. Before ICL, deploying a model for a new task required collecting labeled data and fine-tuning the model. Teams then had to validate it before deploying task-specific weights. This process took days to weeks and required machine learning engineering expertise.

With ICL, task adaptation becomes a matter of prompt design. A domain expert without machine learning training can configure a model for their specific use case by giving good examples. This shift has expanded who can build AI-powered applications and how quickly they can be developed.

The emergence of ICL also raised questions about what language models learn during pre-training. If a model can solve novel tasks from demonstrations, what does this say about its understanding? The mechanistic hypotheses we explored suggest that these models may be learning something deeper than text statistics, perhaps something closer to general-purpose reasoning and adaptation algorithms.