Chain-of-Thought › Why Chain-of-Thought Works

The improvement from CoT is not just behavioral. It reflects something real about how autoregressive language models compute, and understanding the mechanism helps you apply and extend CoT more effectively.

Transformers process input in parallel within each forward pass, but they generate output sequentially, one token at a time. Each generated token extends the context window and becomes part of the input for the next token. This means the model's effective computation per output token is bounded by how much the attention mechanism can integrate within a single forward pass.

When intermediate reasoning steps are written out explicitly, the model does not need to compress all necessary computation into a single prediction. Instead, it distributes the work across multiple generation steps. Each step is a new forward pass with more context, including the reasoning produced so far.

Consider a problem that requires four sequential arithmetic operations. Without CoT, the model must implicitly perform all four in the course of predicting one answer token. The residual stream at that final token must somehow carry the result of four cascaded computations, each dependent on the previous. This places severe demands on the model's internal representations. With CoT, the first operation is performed and written out, then the second operation can take the written result as input at the token level, at both the token and representation levels. The problem is decomposed into four separate computation steps.

This decomposition has formal parallels to circuit complexity theory. A problem that requires $k$ sequential computations, where each computation depends on the result of the previous one, has a serial depth of $k$. A model with fixed depth (say, $L$ transformer layers) can handle computations of depth up to $L$ in a single forward pass. For problems with depth exceeding $L$, the model cannot perform the computation in one pass. CoT effectively increases the usable computation depth by breaking the problem across multiple passes: each generation step is a fresh $L$-layer computation over the context that includes all previous reasoning. The effective depth is $L \times T$, where $T$ is the number of reasoning steps. This analysis, formalized by Feng et al. (2023) among others, provides a theoretical basis for why CoT dramatically improves performance on problems that require many sequential steps.

From the in-context learning perspective, CoT prompting teaches the model which program to run by example. Standard few-shot examples teach the model: "respond to a question with a short answer." CoT examples teach it: "respond to a question with a reasoning chain followed by an answer."

Language models trained on large text corpora have seen millions of examples of human reasoning, from math textbook solutions to forum explanations to scientific papers. CoT prompting activates this latent capability by demonstrating the desired output format. The model generalizes from the few-shot examples to apply the same reasoning pattern to new questions.

This framing helps explain why the improvement is not uniform across model sizes. Smaller models have not seen enough training data, have insufficient capacity to reliably identify and apply reasoning formats from few-shot examples, or lack the representations necessary to perform multi-step inference even when prompted. The capability is absent rather than hidden behind a prompt barrier. At sufficiently large scale, the capability exists and the prompt elicits it. This is why CoT's effectiveness is considered an emergent property: it appears abruptly and qualitatively differently above a scale threshold rather than improving gradually.

Explicit reasoning chains have a practical benefit that pure answer generation does not: they are interpretable. When a model produces an incorrect answer in standard prompting, it is hard to know where the reasoning went wrong, or even that there was reasoning to inspect. With CoT, you can read the chain and identify exactly where the model made an error, whether in problem decomposition, arithmetic, logical inference, or fact retrieval.

This interpretability also matters during development. Researchers can examine failing CoT traces to understand failure modes, identify categories of errors, and design better prompting strategies or training procedures accordingly. If you see that a model consistently fails on CoT chains that require percentage calculations, you know to add exemplars that demonstrate percentage arithmetic. If you see errors only in chains where a variable's value must be tracked across four or more steps, you know to add an explicit bookkeeping step to your exemplars.

Error localization also enables human-in-the-loop correction. A human reviewer can spot the error in a CoT chain, correct it, and either use the corrected chain directly or incorporate it as a new exemplar. This creates a feedback loop that is much more tractable than trying to correct output by adjusting prompts blindly.