GPT-3 › Worked Example: Analyzing In-Context Learning Behavior

Let's work through a concrete example that illustrates several key aspects of in-context learning: the effect of example quality, the importance of label balance, and the degradation of performance when demonstrations conflict.

Suppose you want to use GPT-3 for a three-way sentiment classification task (Positive, Neutral, Negative). You have a set of test reviews and you want to construct the best possible few-shot prompt. Here is how different choices affect the setup:

Scenario 1: Unbalanced labels. You select 6 examples, but 4 are Positive, 1 is Neutral, and 1 is Negative. The model will be biased toward predicting Positive because it has seen that label most recently and most frequently. For a well-calibrated classifier, balance matters, and you should aim for roughly equal representation of all classes in your demonstrations.

Scenario 2: Label-text mismatch. You accidentally swap some labels while preparing your examples: a clearly positive review is labeled Neutral, and a neutral review is labeled Positive. For this simple classification task, the model may perform nearly as well as with correct labels (consistent with the Min et al. findings), because it is mainly using format information. But if the task is more complex, the mismatched labels will introduce label noise.

Scenario 3: Semantically distant examples. Your test inputs are about restaurant experiences, but your examples are about software products. The input distribution mismatch means the model has less relevant context for making predictions. Performance may be better than zero-shot (the format is still informative) but worse than if you used restaurant-focused examples.

Scenario 4: Optimal construction. You select 9 examples: 3 Positive, 3 Neutral, 3 Negative, all about restaurants, ordered so that the example most similar to your typical test input is last. This configuration maximizes the format signal, provides balanced label information, minimizes distribution mismatch, and exploits the recency effect. It represents best-practice few-shot construction for this task.

The contrast between these scenarios illustrates that in-context learning, while powerful, requires thoughtful prompt construction. It is not a magic black box where any examples produce good results. Systematic thinking about label balance and semantic relevance, along with ordering and format, pays off in practice.