Chain-of-Thought › The Historical Context Behind Chain-of-Thought

To appreciate why CoT was such a breakthrough, it helps to understand what came before it. Prior to 2022, the prevailing approach to few-shot in-context learning was simple: give the model a small number of question-answer pairs and hope it learned the task from those examples. For factual retrieval, classification, and simple pattern-matching tasks, this worked reasonably well. For multi-step reasoning, it failed almost uniformly.

Researchers had tried various approaches to patch this gap. Some added a scratchpad area to the prompt, allowing the model to write intermediate work before committing to an answer. Others used output-parsing strategies that broke down a single complex question into multiple simpler subquestions. Still others relied on careful problem decomposition by humans before passing anything to the model. Each approach had merit, but none combined the simplicity, generality, and effectiveness that Wei et al. would achieve.

The Wei et al. (2022) paper made several specific observations that helped explain the gap. First, the improvement from CoT was not uniform across model sizes. Small models did not benefit, and in some cases degraded with CoT. The benefit emerged sharply around 100 billion parameters and grew with scale. This emergent property suggested that CoT was eliciting a latent capability that only manifested in sufficiently large models. Second, the improvement was especially pronounced for tasks with a natural step-by-step structure: arithmetic word problems, commonsense causality questions, and symbolic manipulation tasks. For tasks that did not require sequential reasoning, such as extractive question answering, the benefit was smaller.

Simultaneously, Kojima et al. (2022) published a companion paper showing that you did not need carefully crafted exemplars at all. The single phrase "Let's think step by step," appended to the question before the model's response, was enough to elicit coherent reasoning chains. This zero-shot finding was arguably even more surprising than the few-shot result. It demonstrated that large language models already possessed latent reasoning capabilities and that eliciting them required only a minimal cue.

Together, these two papers marked a watershed moment. Reasoning capability was no longer an innate property of a model that either existed or did not. It was a behavior that prompt designers could reliably elicit, shape, and improve.