Chain-of-Thought › Few-Shot Chain-of-Thought

The original CoT formulation is a few-shot approach. You provide between four and eight exemplar question-answer pairs, each with an explicit reasoning chain, before the question you want answered. The model learns from these exemplars to produce similar reasoning chains when generating its own answer.

The choice of four to eight exemplars reflects a practical balance. Too few exemplars, and the model may not clearly understand the desired reasoning format. Too many, and the prompt grows unwieldy and begins to eat into the space available for the target question and its answer. For most tasks, four exemplars reliably establish the format, and diminishing returns kick in beyond eight.

The quality of CoT exemplars significantly affects performance. Several principles guide their construction.

The reasoning chains should be correct. Incorrect exemplars teach the model wrong reasoning patterns and often degrade performance below even standard prompting. This seems obvious but matters practically: when building exemplars by hand, small arithmetic errors or logical slips are easy to introduce. Always verify every step in every exemplar numerically, not just by inspection.

The reasoning should be atomic. Each step should represent one conceptual operation: a single arithmetic calculation, one logical deduction, one fact retrieval. Long compound steps that secretly perform multiple operations at once reduce the benefit of writing steps out, since the model still has to perform all the work in one generation step to produce that compound step. A step like "She earns $12 per hour, working 8 hours per day for 5 days over 3 weeks, so she earns 12 × 8 × 5 × 3 = $1440" defeats the purpose. Better to write "She earns 12 × 8 = $96 per day. Over 5 days: 96 × 5 = $480 per week. Over 3 weeks: 480 × 3 = $1440."

The exemplars should cover the reasoning patterns expected in the test set. If the test questions involve percentage calculations, some exemplars should demonstrate percentage reasoning. If the test questions involve temporal ordering, some exemplars should demonstrate that. Diverse exemplars generalize better than redundant ones that all demonstrate the same pattern. If all four of your exemplars involve simple addition and subtraction, the model will not learn to apply CoT to questions requiring multiplication or division.

Exemplar annotation is expensive. For a complex domain, constructing eight exemplars with high-quality reasoning chains requires significant expert effort. This limitation motivated research into reducing the annotation burden, ultimately leading to zero-shot CoT and automated exemplar generation.

Not all exemplars are equally useful. Automatic exemplar selection methods choose questions most likely to activate the relevant reasoning skills. Active prompting (Diao et al., 2023) identifies the questions where a model is most uncertain, measured by variance across multiple sampled answers, and solicits human annotations for those. The intuition is that these uncertainty points correspond to reasoning patterns the model needs the most help with. You are using the model's own uncertainty to guide where human annotation effort goes.

Complexity-based selection (Fu et al., 2023) prefers exemplars with longer, more detailed reasoning chains, on the theory that more complex exemplars teach more thorough reasoning. Empirically, selecting for chain complexity often outperforms random selection. When given a pool of candidate exemplars varying in step count from two to eight, selecting the four with the longest chains typically outperforms randomly selecting four. The model learns to be more thorough when it sees thorough exemplars.

Diverse selection, which explicitly picks exemplars covering different reasoning patterns and problem types, consistently outperforms naive selection. If you have a budget of four exemplars for a math word problem task, choosing one that demonstrates rate calculations, one that requires unit conversion, one involving sequential accumulation, and one involving distribution tends to outperform choosing four problems all of the same type.

The order in which exemplars appear in the prompt has a modest but measurable effect on performance. Models tend to be somewhat recency-biased, meaning the last exemplar can have slightly more influence on the output format. For most tasks, the effect is small enough to ignore, but when fine-tuning prompts for critical applications, experimenting with exemplar order is worthwhile. Placing the most relevant or most complex exemplar last is a reasonable heuristic.