Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

In-Context Learning › Theoretical Understanding of ICL

Michael BrenndoerferUpdated July 27, 202513 min read

Language AI HandbookAll chapters

Part 28 of 67GPT Architecture

Chapter 200In-Context Learning › Theoretical Understanding of ICL

Reading section 6 of 11 · Theoretical Understanding of ICL · 13 minSection 6 of 11 · 13 min

Section 6 of 11 · 13 min read

In-Context Learning › Theoretical Understanding of ICLOne section · Sections in this chapter6 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

How can a model "learn" without updating its weights? This question strikes at the heart of what makes in-context learning so puzzling. In traditional machine learning, learning requires gradient descent: the model sees examples, computes errors, and adjusts its parameters to reduce those errors over many iterations. ICL appears to violate this basic requirement. A frozen model, with weights fixed at their pretrained values, somehow adapts its behavior based on demonstrations it has never seen before.

Understanding this phenomenon requires thinking carefully about what "learning" means in the context of a forward pass through a neural network. Three complementary hypotheses have emerged from theoretical research, each illuminating a different aspect of how ICL works. We'll explore each in turn, building from intuition to mathematical formalization.

Hypothesis 1: Task LocationLink Copied

The first hypothesis reframes the question: perhaps ICL doesn't involve learning at all in the traditional sense. Instead, demonstrations might serve as a kind of address that locates pre-existing knowledge.

Consider what happens during pretraining. The model processes billions of tokens from sources such as Wikipedia articles and books, alongside code and forum posts. Within this ocean of data, the model implicitly encounters many task formulations:

Question-answer pairs embedded in FAQ pages and forums
Translation examples in multilingual documents
Sentiment expressions in reviews and social media
Factual statements in encyclopedic content
Code with natural language comments and docstrings

The key insight is that pretraining doesn't just teach the model to predict the next token. It teaches the model to recognize patterns and activate appropriate processing strategies. When you provide few-shot demonstrations, you're not teaching the model something new. Instead, you're helping it identify which of its existing capabilities to apply.

Think of it like a library with millions of books but no catalog. The model has all the knowledge, but it needs help finding the right book. Demonstrations serve as search queries that locate the relevant "section" of the model's capabilities.

Out[20]:

Visualization

Conceptual diagram showing task regions in parameter space activated by demonstrations. — Task location hypothesis visualization. The model's parameter space contains regions specialized for different tasks. Demonstrations activate the appropriate region, effectively selecting among existing capabilities rather than learning new ones.

Several lines of evidence support the task location hypothesis:

ICL works best for familiar tasks: Performance is highest on tasks similar to those encountered during pretraining, suggesting the model is retrieving existing capabilities rather than learning new ones.
Novel operations fail: Tasks requiring new logical operations that couldn't have been learned from text data tend to fail at ICL, even with many demonstrations.
Task descriptions can substitute for examples: Sometimes a natural language description of the task works as well as demonstrations, which makes sense if the goal is location rather than learning.

The task location view is intuitive, but it doesn't fully explain the mechanics. How does the model use demonstrations to activate the right capabilities? This leads us to our second hypothesis.

Hypothesis 2: Implicit Gradient DescentLink Copied

While the task location hypothesis tells us what ICL accomplishes, the implicit gradient descent hypothesis explains how the transformer architecture implements this process. The key insight is that the attention mechanism, when viewed mathematically, performs operations that closely resemble gradient descent.

To understand this connection, we need to examine what happens during a single attention layer. When the model processes a prompt containing demonstrations followed by a test input, the attention mechanism allows information to flow from the demonstrations to the test input.

Consider the query input (the test case we want the model to complete). Before attention, it has some initial representation $h_{\text{query}}^{(0)}$ . The attention mechanism then updates this representation by aggregating information from the demonstrations:

h_{\text{query}} = h_{\text{query}}^{(0)} + \sum_{i=1}^{k} \alpha_i \cdot v_i

where:

$h_{\text{query}}$ : the updated representation of the query input after attention
$h_{\text{query}}^{(0)}$ : the initial representation of the query input before this attention layer
$\alpha_i$ : the attention weight assigned to demonstration $i$ , computed via softmax over query-key dot products
$v_i$ : the value vector derived from demonstration $i$ , encoding information about that example
$k$ : the total number of demonstrations in the prompt

This formula has the same structure as a gradient descent update. In standard gradient descent, we update parameters by:

\theta \leftarrow \theta + \eta \cdot \nabla L

where $\theta$ represents the current parameters, $\eta$ is the learning rate, and $\nabla L$ is the gradient of the loss. The analogy becomes clear when we make the following correspondences:

Correspondence between gradient descent and attention mechanism components.
Gradient Descent	Attention Mechanism
Current parameters $\theta$	Initial representation $h_{\text{query}}^{(0)}$
Learning rate $\eta$	Attention weights $\alpha_i$
Gradient $\nabla L$	Value vectors $v_i$
Updated parameters	Updated representation $h_{\text{query}}$

The attention weights $\alpha_i$ act as adaptive, example-specific learning rates. They automatically assign more weight to demonstrations that are more relevant (those with higher query-key similarity). This makes the mechanism more adaptive than standard gradient descent, where the learning rate is typically fixed.

Research has substantiated this connection with increasingly strong results:

Constructive proofs: Researchers have shown that transformers can be explicitly constructed to implement gradient descent algorithms.
Linear attention equivalence: For linear attention layers (without the softmax nonlinearity), the forward pass is mathematically equivalent to one step of gradient descent on a regression problem.
Multi-step optimization: With 96 layers like GPT-3, the model can implement many sequential "optimization steps," letting for sophisticated adaptation to the demonstrations.

In[21]:

Code

def attention_as_gradient_step(
    query_embedding, demo_embeddings, demo_labels, learning_rate=0.1
):
    """
    Illustrate how attention over demonstrations resembles gradient descent.

    This simplified model shows the conceptual parallel between:
    - Attention weighted combination of demonstration values
    - Gradient-based update toward demonstration patterns

    Args:
        query_embedding: Vector representation of the query input
        demo_embeddings: Matrix of demonstration input embeddings
        demo_labels: Labels/outputs for each demonstration
        learning_rate: Step size for the gradient interpretation

    Returns:
        Updated representation after "learning" from demonstrations
    """
    # Compute attention scores (simplified: dot product similarity)
    attention_logits = np.dot(demo_embeddings, query_embedding)
    attention_weights = np.exp(attention_logits) / np.sum(
        np.exp(attention_logits)
    )

    # The "gradient" can be viewed as a weighted combination of
    # directions toward each demonstration
    gradient = np.zeros_like(query_embedding)
    for i, (demo_emb, weight) in enumerate(
        zip(demo_embeddings, attention_weights)
    ):
        # Direction from query toward demonstration
        direction = demo_emb - query_embedding
        gradient += weight * direction

    # Apply the "gradient update"
    updated = query_embedding + learning_rate * gradient

    return updated, attention_weights

Out[22]:

Console

Attention as Implicit Gradient Descent
==================================================

Query embedding: [1.00, 0.50, 0.20]

Demonstration attention weights:
  Demo 1 (Positive): 0.405
  Demo 2 (Negative): 0.175
  Demo 3 (Positive): 0.420

Updated embedding: [0.978, 0.501, 0.206]
Shift magnitude: 0.0227

The attention weights reveal how the model allocates "learning" across demonstrations. Demo 1 and Demo 3 (both Positive) receive higher weights because their embeddings are more similar to the query. Demo 2 (Negative) receives lower weight due to its dissimilar embedding. The updated representation shifts toward the weighted average of demonstrations, with the shift magnitude showing how much the query representation changed.

This toy example captures the essence of the implicit gradient descent hypothesis: demonstrations don't just provide information, they actively reshape the query representation through the attention mechanism. Similar examples contribute more to this reshaping, just as training examples with larger gradients contribute more to parameter updates in standard learning.

Out[23]:

Visualization

Heatmap showing attention weights from query to each demonstration. — Attention pattern from test query to demonstrations. Higher attention weights (darker blue) indicate demonstrations that contribute more to updating the query representation. In this example, the model attends most strongly to Demo 1 and Demo 3, which are semantically similar to the query and share the same label (Positive).

Hypothesis 3: Bayesian InferenceLink Copied

The first two hypotheses focus on mechanism: what capabilities exist (task location) and how attention implements adaptation (implicit gradient descent). Our third hypothesis takes a probabilistic perspective, asking: how does the model's uncertainty about the task change as it sees more demonstrations?

Imagine you're trying to guess what game someone is playing by watching them take actions. Each action provides evidence about which game it might be. Early on, many games are plausible. As you observe more actions, your beliefs concentrate on the correct game. This is exactly what Bayesian inference formalizes.

Applied to ICL, the model starts with a prior distribution over possible tasks, learned during pretraining. This prior reflects how often different task types appeared in the training data. When you provide demonstrations, each one updates this distribution according to Bayes' rule:

P(\text{task} \mid D_1, \ldots, D_k) \propto P(D_1, \ldots, D_k \mid \text{task}) \cdot P(\text{task})

where:

$D_i = (x_i, y_i)$ : the $i$ -th demonstration, consisting of an input $x_i$ and its corresponding output $y_i$
$P(\text{task})$ : the prior probability of each task type, which reflects how often the model encountered similar patterns during pretraining
$P(D_1, \ldots, D_k \mid \text{task})$ : the likelihood of observing these specific demonstrations if the task is as hypothesized (demonstrations consistent with the task have high likelihood)
$P(\text{task} \mid D_1, \ldots, D_k)$ : the posterior probability of each task after observing all $k$ demonstrations
$\propto$ : indicates proportionality (the left side equals the right side divided by a normalizing constant)

Let's trace through this formula step by step to build intuition:

The prior $P(\text{task})$ encodes pretraining knowledge. If the model saw many sentiment classification examples during pretraining, sentiment analysis has a higher prior probability than, say, a specialized domain-specific task.
The likelihood $P(D_1, \ldots, D_k \mid \text{task})$ evaluates consistency. For each possible task interpretation, how likely is it that we would observe these specific demonstrations? If the demonstrations show text mapped to "Positive" and "Negative" labels, the likelihood is high under "sentiment classification" and low under "translation."
The posterior concentrates probability. Multiplying prior and likelihood (and normalizing) yields a posterior that combines our initial beliefs with the evidence from demonstrations. As more demonstrations are added, the posterior becomes sharper, putting on the task interpretation that best explains all the evidence.

This Bayesian view explains several empirical observations about ICL:

Out[24]:

Visualization

Bar chart showing uniform prior distribution over tasks. — Prior P(task): Uniform distribution before seeing any demonstrations.

Bar chart showing posterior after one demonstration. — After 1 Example: Posterior begins putting on sentiment classification.

Bar chart showing peaked posterior after four demonstrations. — After 4 Examples: High confidence in the correct task interpretation.

The visualization illustrates how the posterior evolves. Initially, probability is spread uniformly across task interpretations. After one demonstration showing an input mapped to "Positive," the posterior shifts toward sentiment classification. After four demonstrations consistently showing sentiment labels, the model has high confidence in the task identity.

Evidence supporting the Bayesian view comes from several empirical observations:

Inconsistent demonstrations hurt performance. If you mix correct and incorrect labels, the likelihood term is reduced for all task interpretations, leading to a diffuse posterior and poor predictions.
Ambiguous prompts yield uncertain outputs. When demonstrations could plausibly come from multiple tasks, the model's predictions are less confident, exactly as a diffuse posterior would predict.
More examples help even when redundant. Providing 16 similar examples rather than 4 can improve performance, because each example sharpens the posterior even if it doesn't add new information.

Synthesizing the TheoriesLink Copied

These three hypotheses are not competing explanations but rather complementary perspectives on the same phenomenon. Each answers a different question:

The three ICL hypotheses address complementary questions about the phenomenon.
Hypothesis	Question Answered
Task Location	What capabilities can ICL access?
Implicit Gradient Descent	How does attention implement adaptation?
Bayesian Inference	How does uncertainty reduce with evidence?

A unified view might work like this: During pretraining, the model develops specialized circuits for different tasks (task location). When demonstrations are provided, the attention mechanism adapts the query representation toward these demonstrations (implicit gradient descent), which has the effect of putting probability on task interpretations consistent with the evidence (Bayesian inference).

Recent theoretical work has made this unified view more concrete. Researchers have shown that transformers, through their architecture, learn to implement meta-learning algorithms during pretraining. When presented with demonstrations at inference time, they execute these algorithms to infer the task and generate appropriate predictions. The specific algorithm implemented may vary by layer, attention head, and task type, with different components of the network contributing different aspects of the adaptation process.