Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

In-Context Learning Emergence › ICL Emergence Curves

Michael BrenndoerferNovember 6, 202511 min read

Language AI HandbookAll chapters

Part 30 of 67Emergent Capabilities

Chapter 215In-Context Learning Emergence › ICL Emergence Curves

Reading section 2 of 10 · ICL Emergence Curves · 11 minSection 2 of 10 · 11 min

Section 2 of 10 · 11 min read

In-Context Learning Emergence › ICL Emergence CurvesOne section · Sections in this chapter2 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The emergence of in-context learning follows characteristic patterns that researchers have documented across multiple model families and scales. Unlike smooth improvements in perplexity or basic language modeling metrics, ICL ability appears relatively suddenly as models cross certain scale thresholds.

The Scale Threshold PhenomenonLink Copied

Early investigations by Brown et al. with GPT-3 revealed that in-context learning performance improves sharply with model size, but this improvement is highly non-linear. On many tasks, models below approximately 1 billion parameters show in effect random performance when given few-shot examples, while models above 10-100 billion parameters reach substantial accuracy gains from the same demonstrations.

The emergence curve for ICL typically follows a pattern with several distinct phases:

Random baseline phase: Small models (typically <1B parameters) show no real improvement from in-context examples. Performance remains near chance regardless of demonstration quality or quantity.
Transition phase: Models in an intermediate range begin showing some sensitivity to demonstrations, but performance is unstable and task-dependent.
Competent phase: Larger models consistently use demonstrations, with performance scaling log-linearly with both model size and number of examples.

In[3]:

Code

import numpy as np

np.random.seed(42)

# Simulated ICL emergence data based on published research patterns
# Model sizes in billions of parameters
model_sizes = np.array([0.1, 0.3, 0.7, 1.3, 2.7, 6.7, 13, 30, 70, 175])

# Zero-shot performance (baseline)
zero_shot = np.array(
    [0.25, 0.26, 0.27, 0.30, 0.35, 0.42, 0.48, 0.55, 0.62, 0.68]
)

# Few-shot performance (4 examples) - showing emergence
few_shot_4 = np.array(
    [0.25, 0.26, 0.28, 0.35, 0.48, 0.62, 0.72, 0.80, 0.86, 0.90]
)

# The ICL gain (difference between few-shot and zero-shot)
icl_gain = few_shot_4 - zero_shot

# Calculate key statistics for interpretation
max_icl_gain = icl_gain.max()
max_gain_idx = icl_gain.argmax()
max_gain_scale = model_sizes[max_gain_idx]

Out[4]:

Visualization

Plot showing zero-shot and few-shot performance vs model size on log scale. — Zero-shot vs few-shot performance across model scales. ICL ability emerges suddenly around 1-10B parameters.

Plot showing ICL gain vs model size on log scale. — ICL gain (benefit from demonstrations) showing sharp increase in the emergence zone.

This visualization reveals the characteristic S-curve of ICL emergence. Below 1B parameters, demonstrations give almost no benefit. Between 1B and 10B parameters, ICL ability emerges rapidly. Above 10B parameters, models reliably exploit few-shot examples, though the marginal gains continue to compound with scale.

Task-Dependent Emergence ThresholdsLink Copied

Not all tasks exhibit identical emergence thresholds. Some capabilities emerge earlier than others, creating a hierarchy of ICL difficulty:

Early-emerging tasks (emerge at smaller scales) typically include:

Simple pattern completion
Basic translation of common language pairs
Sentiment classification with clear signals
Factual question answering about high-frequency knowledge

Late-emerging tasks (require larger scales) include:

Multi-step arithmetic
Abstract reasoning and analogy
Low-resource language translation
Tasks requiring world knowledge integration

This task hierarchy suggests that ICL emergence isn't a single phenomenon but rather a spectrum of capabilities that appear progressively as models scale. Wei et al. (2022) documented this pattern systematically, showing that the emergence threshold varies by more than an order of magnitude across different task types.

In[5]:

Code

# Different tasks have different emergence thresholds
task_categories = {
    "Sentiment Analysis": {"threshold": 0.5, "max_perf": 0.92},
    "Translation (High-Resource)": {"threshold": 1.0, "max_perf": 0.85},
    "Question Answering": {"threshold": 2.0, "max_perf": 0.88},
    "Translation (Low-Resource)": {"threshold": 10.0, "max_perf": 0.75},
    "Multi-Step Arithmetic": {"threshold": 30.0, "max_perf": 0.80},
}


def sigmoid_emergence(scale, threshold, max_perf, steepness=2.0):
    """
    Model emergence as a sigmoid function centered at threshold.

    This implements the emergence curve:

    $$
    P(s) = \frac{P_{\max}}{1 + \exp(-k \cdot (\log_{10}(s) - \log_{10}(s_t)))}
    $$

    where:
    - $P(s)$: the predicted ICL performance at model scale $s$
    - $P_{\max}$: the asymptotic maximum performance (max_perf parameter)
    - $s$: model size in billions of parameters (scale parameter)
    - $s_t$: the threshold scale where performance reaches 50% of maximum
    - $k$: steepness parameter controlling transition sharpness

    Args:
    - scale: model size in billions of parameters
    - threshold: the scale at which performance reaches 50% of maximum
    - max_perf: asymptotic maximum performance
    - steepness (k): controls how sharp the transition is

    Using log-scale ensures the sigmoid is symmetric in orders of magnitude.
    """
    x = np.log10(scale) - np.log10(threshold)
    return max_perf / (1 + np.exp(-steepness * x))


scales = np.logspace(-1, 2.5, 100)  # 0.1B to ~300B

task_performance = {}
for task, params in task_categories.items():
    perf = sigmoid_emergence(3.0, params["threshold"], params["max_perf"])
    task_performance[task] = {
        "threshold": params["threshold"],
        "perf_at_3B": perf,
    }

# Calculate threshold ratio for interpretation
early_threshold = task_categories["Sentiment Analysis"]["threshold"]
late_threshold = task_categories["Multi-Step Arithmetic"]["threshold"]
threshold_ratio = late_threshold / early_threshold

The sigmoid emergence function captures a basic insight about how capabilities develop in neural networks. Rather than appearing gradually, ICL ability transitions rapidly once the model crosses a necessary scale threshold. The function operates in log-space because the relationship between model size and capability is multiplicative rather than additive. Doubling the parameters doesn't add a fixed amount of capability, but rather multiplies the model's effective capacity. The threshold parameter $s_t$ marks the inflection point where the model transitions from having in effect no ICL ability to having substantial capability. The steepness parameter $k$ controls how abrupt this transition is; higher values produce sharper phase transitions, while lower values create more gradual emergence curves. This mathematical formulation allows researchers to precisely characterize and compare emergence patterns across different tasks and model families.

Out[6]:

Console

Sentiment Analysis: threshold=0.5B, perf at 3B=0.76
Translation (High-Resource): threshold=1.0B, perf at 3B=0.61
Question Answering: threshold=2.0B, perf at 3B=0.52
Translation (Low-Resource): threshold=10.0B, perf at 3B=0.20
Multi-Step Arithmetic: threshold=30.0B, perf at 3B=0.10

Threshold ratio (hardest/easiest): 60x

Out[7]:

Visualization

Multiple sigmoid curves showing emergence thresholds varying by task type. — Different tasks exhibit different ICL emergence thresholds. Simpler tasks like sentiment analysis emerge at smaller scales, while complex reasoning tasks require much larger models.

The emergence curves reveal a clear hierarchy: sentiment analysis reaches 50% of maximum performance at just 0.5B parameters, while multi-step arithmetic requires approximately 30B parameters to reach the same relative capability. This 60-fold difference in emergence thresholds shows that ICL is not a monolithic capability but a family of skills with vastly different computational requirements.

Out[8]:

Visualization

Multiple sigmoid curves showing how steepness affects transition sharpness. — Effect of steepness parameter (k) on emergence curve sharpness.

Multiple sigmoid curves showing how threshold affects emergence timing. — Effect of threshold parameter on emergence timing.

Training Data and ComputeLink Copied

Scale alone doesn't fully determine ICL emergence. The Chinchilla scaling laws we covered in Part XXIII showed that optimal model performance requires balancing model size with training data quantity. Given a fixed compute budget, there exist optimal allocations for both model size and training data:

N_{\text{opt}} \propto C^{0.5}, \quad D_{\text{opt}} \propto C^{0.5}

where:

$N_{\text{opt}}$ : the optimal number of model parameters for a given compute budget
$D_{\text{opt}}$ : the optimal number of training tokens
$C$ : the total compute budget (measured in FLOPs)
$\propto$ : denotes proportionality (the quantities scale together but may differ by a constant factor)

The exponent of $0.5$ indicates square-root scaling: if you double your compute budget, both the optimal model size and optimal data quantity should increase by a factor of $\sqrt{2} \approx 1.41$ . This square-root relationship ensures neither parameter count nor data becomes a bottleneck. This balance also affects ICL emergence.

To understand why this relationship matters for ICL emergence, consider what happens when you deviate from optimal allocation. If you invest all additional compute into model size while keeping data fixed, the model becomes "undertrained." It has capacity it cannot effectively use because it hasn't seen enough diverse patterns during training. Conversely, if you invest all additional compute into data while keeping model size fixed, you create an "overexposed" model that has seen more patterns than it can stand for. The Chinchilla insight is that both failure modes hurt performance, and the optimal path forward maintains a balance between the two. For ICL specifically, this means that raw parameter count is insufficient. The model must also have been trained on sufficient diverse data to develop the pattern recognition capabilities that underlie few-shot learning.

Research by Hoffmann et al. demonstrated that compute-optimal training can shift emergence curves leftward, meaning ICL capabilities appear at smaller model sizes when training is more efficient. A 10B parameter model trained optimally might exhibit ICL performance comparable to a 50B parameter model trained sub-optimally.

The quality and diversity of pre-training data also matters materially. Models trained on more diverse corpora with greater task variety tend to develop stronger ICL abilities at equivalent scales. This observation connects to the meta-learning perspective we'll explore later in this chapter.