Chain-of-Thought Emergence › Chain-of-Thought Elicitation

Researchers have developed multiple techniques for eliciting chain-of-thought reasoning from language models. Each approach balances reasoning quality against generality and the prompt engineering it requires. Understanding these techniques is needed for practitioners who want to apply CoT reasoning effectively, as the choice of elicitation method can materially impact both the quality of reasoning and the practical costs of deployment.

The original approach involves giving exemplars that show the reasoning process. This builds directly on the in-context learning capabilities we discussed in the previous chapter, using the model's ability to recognize and continue patterns it observes in its input context. The prompt structure typically follows this pattern:

The model learns from the demonstrations that it should generate intermediate steps before the final answer. This learning happens entirely through pattern recognition. The model observes that questions in this context are followed by explanatory text, not just bare answers, and it generalizes this pattern to new questions. The quality of the reasoning chains in the exemplars matters materially. Demonstrations with clear, logical steps tend to produce better results than those with convoluted or incorrect reasoning. If the examples contain sloppy reasoning, skipped steps, or errors, the model may reproduce these flaws in its own generations.

Few-shot CoT requires careful exemplar selection. This is both its main strength and its primary burden. The demonstrations should cover the types of reasoning required for the target task, but they need not be from the same domain. Research has shown that exemplars from math problems can improve reasoning on logical puzzles, and vice versa. This cross-domain transfer suggests that CoT elicitation activates general reasoning capabilities rather than task-specific knowledge. The model isn't learning "how to solve math problems" from the examples. It's learning "when presented with questions in this format, produce step-by-step reasoning rather than jumping to conclusions."

The generality of few-shot CoT comes at the cost of prompt engineering effort. Crafting high-quality exemplars requires understanding both the task and what constitutes good reasoning for that task. Poor exemplar choices can undermine performance, and the optimal number and type of examples varies by task and model.

Kojima et al. (2022) discovered that a remarkably simple prompt addition could elicit chain-of-thought reasoning without any demonstrations whatsoever. Simply appending "Let's think step by step" to a question causes models to generate reasoning chains. This discovery was surprising because it suggested that the reasoning capability existed within the model and merely needed the right trigger to activate, requiring no examples.

This zero-shot approach works because models have seen phrases like "let's think step by step" in their training data. These phrases are often followed by detailed explanations. Academic papers, textbooks, tutorial videos, and forum posts frequently use such phrases as preambles to methodical explanations. During pretraining, the model learned that this phrase predicts a particular style of subsequent text: careful, sequential, explanatory. The prompt acts as a trigger, activating the model's learned association between this phrase and systematic reasoning.

The effectiveness of zero-shot CoT varies by task and model. For some problems, it approaches the performance of carefully crafted few-shot prompts, achieving results that seem remarkable given the minimal prompt engineering required. For others, particularly those requiring domain-specific reasoning patterns or unusual problem structures, few-shot demonstrations remain superior because they can communicate task-specific conventions that the generic trigger cannot.

The advantage of zero-shot CoT lies in its simplicity and universal applicability: no exemplar construction is required, which makes it easy to apply across diverse tasks without domain expertise. A practitioner can add "Let's think step by step" to any question and potentially improve reasoning quality, without needing to understand the task deeply enough to craft good examples.

Other trigger phrases have been explored:

• "Take a deep breath and work through this step by step"
• "Let me work through this carefully"
• "I need to break this down" These phrases all draw on patterns from the model's training on human problem-solving discourse. Such phrases commonly precede methodical analysis. The variety of effective triggers supports the view that CoT activation is about shifting the model into an "explanation mode" rather than about any specific phrase's magical properties.

Wang et al. (2022) proposed an enhancement to chain-of-thought prompting called self-consistency that addresses a basic limitation of single-chain generation. Rather than generating a single reasoning chain, the model produces multiple independent chains by sampling with temperature. The final answer is determined by majority voting across all generated answers.

The intuition behind self-consistency is straightforward. Correct reasoning processes are more likely to converge on the same answer because truth is constrained: there's typically one right answer, and valid reasoning paths must arrive at it. Errors, by contrast, introduce variance and idiosyncrasy. A model might add instead of multiply in one chain, misread a number in another, or skip a step in a third, leading to different wrong answers. If five reasoning chains produce the answer "42" and two produce "38," we can be more confident that "42" is correct because the agreement suggests these chains found the constrained path to the true answer.

Self-consistency works because different reasoning paths can arrive at the same correct answer through different approaches. A math problem might be solved by working forward from the given information or backward from the goal; an arithmetic calculation might be decomposed in multiple valid orders. These alternative paths give independent verification. Meanwhile, errors tend to be idiosyncratic because they depend on which specific mistake was made. A model might make an arithmetic mistake in one chain but not another, or choose different valid approaches that nevertheless converge on the same answer.

This approach materially improves performance but increases inference cost linearly with the number of samples. Generating five chains costs five times as much as generating one. This tradeoff must be evaluated in context. For high-stakes applications where accuracy is critical, the additional cost may be well justified; for high-volume, low-stakes applications, single-chain generation may be more appropriate.

For problems requiring extended reasoning with many steps, least-to-most prompting decomposes the task hierarchically rather than asking the model to tackle everything at once. First, the model breaks the problem into subproblems, explicitly identifying what intermediate questions need to be answered. Then it solves each subproblem in order, with solutions to earlier subproblems included in the context for later ones.

This two-phase approach addresses a key limitation of standard CoT: for very complex problems, even large models struggle to maintain coherent reasoning across many steps when they must plan and execute simultaneously. By separating the planning phase (identifying subproblems) from the execution phase (solving them), least-to-most prompting reduces the cognitive load at each step.

This approach is particularly effective for tasks with compositional structure, where the answer to the full problem can be built systematically from answers to simpler components. Mathematical proof naturally decomposes into lemmas; multi-hop question answering requires establishing intermediate facts; program synthesis builds complex functions from simpler ones. In all these cases, least-to-most prompting gives a scaffold that guides the model through the problem's inherent structure.