Chain-of-Thought › Zero-Shot Chain-of-Thought
Language AI HandbookAll chapters Chain-of-Thought › Zero-Shot Chain-of-ThoughtOne section · Sections in this chapter6 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Few-shot CoT requires constructing exemplars for each new domain or task. Zero-shot CoT, introduced by Kojima et al. (2022), eliminates this requirement entirely with a single addition to the prompt: the phrase "Let's think step by step."
Appending this phrase before the model's response elicits reasoning chains without any demonstration examples. The model has encountered this phrase in training data in contexts where it introduces structured reasoning, so it generalizes to produce similar reasoning structures in new contexts. The mechanism is essentially retrieval: by pattern-matching to contexts in pretraining data where careful step-by-step explanation follows, the model activates the register of careful explanation rather than the register of quick-answer generation.
The Two-Stage Pipeline
Zero-shot CoT is implemented as a two-stage process. In the first stage, the reasoning prompt asks the model to think through the problem:
Q: If you have 3 liters of water and drink half of it, then add 0.5
liters, how much water do you have?
A: Let's think step by step.
The model generates a reasoning chain in response. In the second stage, a separate extraction prompt retrieves the final answer:
[original question]
[generated reasoning chain]
Therefore, the answer (arabic numerals) is
The two-stage design matters because language models asked to reason and answer simultaneously sometimes produce reasoning that drifts from the eventual answer. By separating reasoning from answer extraction, the pipeline ensures the final answer is grounded in the completed chain rather than generated in parallel with it. The extraction step forces the model to commit to a specific answer based on what the reasoning concluded, rather than interpolating between the reasoning output and whatever the model would have produced without reasoning.
In practice, when you call an LLM API, you implement this as two sequential API calls. The first call returns a reasoning chain. You concatenate the original question, the reasoning chain, and a prompt like "Therefore, the answer is" and send that as the second call. The second call's first generated tokens constitute the answer.
Why "Let's Think Step by Step" Works
The phrase "Let's think step by step" appears in training data in a specific kind of context: tutorial explanations, worked examples, educational content, collaborative problem-solving sessions, and instructional writing of all kinds. By activating these contexts, it shifts the model toward a reasoning-oriented generation mode, one associated with carefulness, thoroughness, and structured exposition.
Kojima et al. (2022) tested numerous other trigger phrases and found significant variation in effectiveness. "Let's think step by step" consistently performed best across tasks, though phrases like "Let's work this out in a few steps" and "Let's think about this carefully" also improved performance over no trigger. The specific phrasing matters because different phrases activate different distributions in the model's training data. "Let's think step by step" is particularly common in educational and instructional writing, which tends to contain exactly the kind of careful, sequential reasoning that helps with multi-step problems. "Think about this question" activates a broader distribution that includes casual conversation and opinion-sharing, which is less helpful.
The gap between zero-shot and few-shot CoT reveals an important asymmetry. Zero-shot CoT gives the model full latitude to decide what "step by step" means for a given problem. Few-shot CoT constrains the format through demonstration, which is helpful when the appropriate reasoning pattern is specific and non-obvious, but adds overhead. For well-structured mathematical problems, few-shot CoT with good exemplars consistently outperforms zero-shot CoT. For open-ended reasoning tasks where the relevant steps vary unpredictably, zero-shot CoT can match or exceed few-shot CoT by allowing the model to adapt its reasoning format to the specific question.
Zero-shot CoT is remarkably effective given that it adds just a few words to the prompt. On the GSM8K grade school math benchmark, it improved GPT-3 performance from around 10% to around 40% accuracy. It performs somewhat below well-constructed few-shot CoT on the same benchmarks, but the trade-off in annotation effort is dramatic: zero annotation versus expert construction of 8 high-quality exemplars.