Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

In-Context Learning › Example Selection Strategies

Michael BrenndoerferUpdated July 27, 202512 min read

Language AI HandbookAll chapters

Part 28 of 67GPT Architecture

Chapter 200In-Context Learning › Example Selection Strategies

Reading section 4 of 11 · Example Selection Strategies · 12 minSection 4 of 11 · 12 min

Section 4 of 11 · 12 min read

In-Context Learning › Example Selection StrategiesOne section · Sections in this chapter4 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Not all demonstrations are equally effective. Research has shown that the choice of examples can swing performance by 20-30 percentage points. Understanding what makes examples effective has become a necessary skill in prompt engineering.

Factors That Influence Example QualityLink Copied

Several properties affect how useful an example is as a demonstration:

Relevance: Examples similar to the test input help more than dissimilar ones
Diversity: Examples should cover the range of possible inputs and outputs
Clarity: Unambiguous examples with clear input-output relationships work best
Correctness: Erroneous labels can mislead the model
Ordering: Examples at the end of the prompt have stronger influence

Let's examine each factor and how to optimize for it.

Semantic Similarity SelectionLink Copied

One of the most effective strategies is selecting examples that are semantically similar to the test input. The intuition is that nearby examples in embedding space share relevant features that help the model generalize.

Similarity is typically measured using cosine similarity between embedding vectors:

\text{sim}(a, b) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \|\mathbf{b}\|}

where:

$\mathbf{a}$ and $\mathbf{b}$ : embedding vectors for two text inputs
$\mathbf{a} \cdot \mathbf{b}$ : the dot product of the two vectors
$\|\mathbf{a}\|$ and $\|\mathbf{b}\|$ : the Euclidean norms (lengths) of each vector

Cosine similarity ranges from -1 (opposite directions) to 1 (identical directions), with 0 showing orthogonal (unrelated) vectors. For text embeddings, higher similarity indicates semantically related content.

In[8]:

Code

from sklearn.metrics.pairwise import cosine_similarity


def select_similar_examples(test_input, candidate_pool, embedder, k=4):
    """
    Select k examples most similar to the test input.

    Args:
        test_input: The input to find similar examples for
        candidate_pool: List of (input, output) example tuples
        embedder: Function that maps text to embedding vectors
        k: Number of examples to select

    Returns:
        List of k most similar examples
    """
    # Embed the test input
    test_embedding = embedder([test_input])

    # Embed all candidates
    candidate_inputs = [inp for inp, _ in candidate_pool]
    candidate_embeddings = embedder(candidate_inputs)

    # Compute similarities
    similarities = cosine_similarity(test_embedding, candidate_embeddings)[0]

    # Select top-k
    top_indices = np.argsort(similarities)[-k:][::-1]

    return [candidate_pool[i] for i in top_indices]


# Simulate with mock embeddings for demonstration
def mock_embedder(texts):
    """Mock embedder that returns random but consistent vectors."""
    digest = hashlib.sha256(repr(texts).encode("utf-8")).digest()
    rng = np.random.default_rng(int.from_bytes(digest[:8], "big"))
    return rng.standard_normal((len(texts), 64))


# Example pool
example_pool = [
    ("Great food and amazing service!", "Positive"),
    ("Waited an hour for cold food.", "Negative"),
    ("The steak was cooked perfectly.", "Positive"),
    ("Overpriced for what you get.", "Negative"),
    ("Best Italian restaurant in town!", "Positive"),
    ("Never coming back here again.", "Negative"),
    ("Friendly staff and cozy atmosphere.", "Positive"),
    ("Found a hair in my soup.", "Negative"),
]

test_input = "The pasta was absolutely delicious."

Out[9]:

Console

Test input: "The pasta was absolutely delicious."

Selecting 4 examples by semantic similarity...

Selected demonstrations:
  1. "Never coming back here again." → Negative
  2. "Found a hair in my soup." → Negative
  3. "The steak was cooked perfectly." → Positive
  4. "Great food and amazing service!" → Positive

The selection algorithm identifies examples that are semantically closest to the test input. In this case, the mock embedder produces consistent but randomized vectors, so the selected examples demonstrate the general approach. In practice, using a real sentence embedding model like sentence-transformers would select examples with similar meaning and vocabulary.

Similarity-based selection consistently outperforms random selection. The most similar examples share vocabulary and semantic features with the test input, helping the model identify the kind of input it is processing.

Out[10]:

Visualization

Scatter plot showing test input and candidate examples in 2D embedding space with connections to selected neighbors. — Semantic similarity selection in embedding space. The test input (star) is compared against candidate examples (circles). The algorithm selects the k nearest neighbors (connected by dashed lines). Positive and negative examples naturally cluster based on semantic content.

Diversity Through CoverageLink Copied

While similarity helps, relying solely on it can create blind spots. If all selected examples are too similar, the model may not learn to handle edge cases or the full range of outputs.

A balanced approach selects examples that are both relevant and diverse:

In[11]:

Code

def select_diverse_examples(
    test_input, candidate_pool, embedder, k=4, diversity_weight=0.3
):
    """
    Select examples balancing similarity and diversity.

    Uses maximal marginal relevance (MMR) to avoid redundant examples.

    Args:
        test_input: Input to select examples for
        candidate_pool: Available (input, output) examples
        embedder: Text to embedding function
        k: Number of examples to select
        diversity_weight: Balance between relevance (0) and diversity (1)

    Returns:
        List of k balanced examples
    """
    # Get embeddings
    test_emb = embedder([test_input])
    candidate_inputs = [inp for inp, _ in candidate_pool]
    candidate_embs = embedder(candidate_inputs)

    # Compute relevance scores
    relevance = cosine_similarity(test_emb, candidate_embs)[0]

    selected = []
    selected_embs = []
    remaining = list(range(len(candidate_pool)))

    for _ in range(k):
        if not remaining:
            break

        best_score = -float("inf")
        best_idx = None

        for idx in remaining:
            # Relevance term
            rel_score = relevance[idx]

            # Diversity term: max similarity to already selected
            if selected_embs:
                div_score = max(
                    cosine_similarity([candidate_embs[idx]], selected_embs)[0]
                )
            else:
                div_score = 0

            # MMR score
            score = (
                1 - diversity_weight
            ) * rel_score - diversity_weight * div_score

            if score > best_score:
                best_score = score
                best_idx = idx

        selected.append(candidate_pool[best_idx])
        selected_embs.append(candidate_embs[best_idx])
        remaining.remove(best_idx)

    return selected

Out[12]:

Console

Maximal Marginal Relevance (MMR) Selection
==================================================
Test: "The pasta was absolutely delicious."

With diversity_weight=0.3, selected examples:
  1. "Never coming back here again." → Negative
  2. "Found a hair in my soup." → Negative
  3. "The steak was cooked perfectly." → Positive
  4. "Great food and amazing service!" → Positive

Class distribution: 2 Positive, 2 Negative

Unlike pure similarity-based selection, MMR actively penalizes redundancy. The diversity weight of 0.3 means the algorithm balances 70% relevance with 30% diversity. This typically results in better class coverage, as selecting multiple very similar examples provides diminishing returns.

The maximal marginal relevance (MMR) algorithm iteratively selects examples that are relevant to the query but dissimilar to already-selected examples. At each step, MMR scores each candidate using:

\text{MMR}(d) = (1 - \lambda) \cdot \text{sim}(d, q) - \lambda \cdot \max_{s \in S} \text{sim}(d, s)

where:

$d$ : a candidate demonstration being evaluated
$q$ : the query (test input) we want to find examples for
$S$ : the set of already-selected demonstrations
$\text{sim}(d, q)$ : similarity between candidate $d$ and query $q$ (relevance term)
$\max_{s \in S} \text{sim}(d, s)$ : maximum similarity between candidate $d$ and any already-selected example (redundancy term)
$\lambda$ : diversity weight balancing relevance versus diversity (typically 0.3-0.5)

The first term rewards candidates similar to the query. The second term penalizes candidates similar to already-selected examples. By subtracting the redundancy term, MMR prevents selecting demonstrations that are too similar to each other. This keeps coverage of diverse aspects of the task.

Out[13]:

Visualization

Scatter plot showing pure relevance selection at lambda=0. — λ=0.0 (Pure Relevance): Selects examples closest to test input, may be redundant.

Scatter plot showing balanced selection at lambda=0.3. — λ=0.3 (Balanced): Balances relevance and diversity for better coverage.

Scatter plot showing high diversity selection at lambda=0.7. — λ=0.7 (High Diversity): Prioritizes spreading selections across the space.

Class-Balanced SelectionLink Copied

For classification tasks. This keeps balanced representation across classes is necessary. If your demonstrations are skewed toward one class, the model may be biased toward predicting that class.

Out[14]:

Visualization

Bar chart comparing accuracy with balanced vs imbalanced few-shot examples across classes. — Impact of class balance in few-shot demonstrations. Imbalanced examples bias the model toward the majority class, while balanced selection improves overall accuracy and calibration.

The effect of class imbalance can be dramatic. In extreme cases. This provides examples from only one class causes the model to predict that class for all inputs, regardless of the actual content.

Ordering and Recency EffectsLink Copied

The order of examples in the prompt matters more than you might expect. Examples closer to the test input (at the end of the prompt) tend to have stronger influence on the prediction. This "recency effect" emerges from the attention mechanism, where nearby tokens naturally receive more attention.

Out[15]:

Visualization

Bar chart showing increasing influence weight for examples from first to last position. — Recency effect in few-shot prompts. Examples at the end of the prompt exert stronger influence on predictions. Placing the most relevant or representative examples last can improve performance.

Practical implications of the recency effect:

Place the most representative examples at the end of the prompt
When class-balancing, ensure the final examples aren't all from one class
For tasks with a default or majority class, counterbalance by ending with minority examples