Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

In-Context Learning › Improving ICL Performance

Michael BrenndoerferUpdated July 27, 20256 min read

Language AI HandbookAll chapters

Part 28 of 67GPT Architecture

Chapter 200In-Context Learning › Improving ICL Performance

Reading section 7 of 11 · Improving ICL Performance · 6 minSection 7 of 11 · 6 min

Section 7 of 11 · 6 min read

In-Context Learning › Improving ICL PerformanceOne section · Sections in this chapter7 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Researchers have developed numerous techniques to enhance in-context learning beyond simple few-shot prompting.

Chain-of-Thought PromptingLink Copied

For reasoning tasks, showing intermediate steps significantly improves performance. Instead of just input-output pairs, demonstrations include the reasoning process:

In[25]:

Code

# Standard few-shot
standard_prompt = """
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 balls. How many tennis balls does he have now?
A: 11

Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many do they have?
A:"""

# Chain-of-thought few-shot
cot_prompt = """
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 balls. How many tennis balls does he have now?
A: Roger started with 5 balls. He bought 2 cans with 3 balls each, so 2 * 3 = 6 new balls. Total: 5 + 6 = 11.

Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many do they have?
A:"""

Out[26]:

Console

Standard Few-Shot:
----------------------------------------

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 balls. How many tennis balls does he have now?
A: 11

Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many do they have?
A:

Chain-of-Thought Few-Shot:
----------------------------------------

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 balls. How many tennis balls does he have now?
A: Roger started with 5 balls. He bought 2 cans with 3 balls each, so 2 * 3 = 6 new balls. Total: 5 + 6 = 11.

Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many do they have?
A:

Chain-of-thought (CoT) prompting guides the model to show its work. The reasoning trace helps the model:

Break complex problems into steps
Catch and correct errors mid-reasoning
Align intermediate computations with the final answer

CoT is particularly effective for arithmetic, logical reasoning, and multi-step problems where the direct answer is hard to compute in a single step.

Self-ConsistencyLink Copied

Rather than generating a single answer, self-consistency samples multiple reasoning paths and aggregates them through majority voting. The procedure works as follows:

Generate $N$ different chain-of-thought responses using temperature $T > 0$ to introduce diversity
Extract the final answer $a_i$ from each response $i$
Return the most common answer (majority vote)

Formally, the self-consistency prediction is:

\hat{a} = \arg\max_{a} \sum_{i=1}^{N} \mathbf{1}[a_i = a]

where:

$\hat{a}$ : the final predicted answer
$N$ : the number of sampled reasoning paths
$a_i$ : the answer extracted from the $i$ -th reasoning path
$\mathbf{1}[a_i = a]$ : an indicator function that equals 1 if answer $a_i$ matches $a$ , and 0 otherwise
$\arg\max_{a}$ : selects the answer $a$ that maximizes the vote count

This approach uses the intuition that correct reasoning paths, while potentially different in details, converge on the same answer. Incorrect paths are more likely to diverge, creating different wrong answers that split the vote. With enough samples, the correct answer accumulates more votes than any single incorrect answer.

Out[27]:

Visualization

Diagram showing multiple reasoning paths converging through majority vote. — Self-consistency improves reliability by sampling multiple reasoning paths. While individual paths may contain errors, the majority vote typically converges on the correct answer. This technique is especially valuable for complex reasoning tasks.

Out[28]:

Visualization

Bar chart showing vote counts for different answer values from self-consistency sampling. — Self-consistency vote distribution from 10 sampled reasoning paths. The correct answer (42) receives the most votes despite individual reasoning errors in some paths. Wrong answers scatter across different values, diluting their vote share. This demonstrates how aggregation improves reliability.

Calibration and ConfidenceLink Copied

ICL predictions can be overconfident or poorly calibrated. Several techniques address this:

Probability calibration: Adjust output probabilities using held-out validation data
Verbalizing uncertainty: Ask the model to express confidence in its answer
Null prompts: Compare predictions with and without demonstrations to detect when the model is guessing

Retrieval-Augmented ICLLink Copied

Instead of using a fixed set of demonstrations, retrieval-augmented approaches dynamically select examples for each test input:

Embed the test input
Retrieve similar examples from a large pool
Use retrieved examples as demonstrations

This combines the benefits of similarity-based selection with the scalability of large example pools. The model effectively has access to thousands of potential demonstrations, selecting the most relevant ones at inference time.

Out[29]:

Visualization

Grouped bar chart comparing accuracy of random vs retrieval-based example selection. — Performance comparison of random vs. retrieval-based example selection across tasks. Retrieval-augmented ICL (using semantic similarity to select demonstrations) consistently outperforms random selection, with the largest gains on tasks where relevant examples significantly differ from irrelevant ones.