Chain-of-Thought Emergence › Mechanistic Theories

Understanding why chain-of-thought works requires moving beyond behavioral observations to examine what's happening inside the model. This investigation connects observable improvements in reasoning to underlying computational mechanisms. Several theoretical frameworks attempt to explain the phenomenon, each giving different insights into the nature of reasoning in neural networks.

One explanation views the reasoning chain as extending the model's computation by giving additional processing steps. A transformer processes its input through a fixed number of layers, typically dozens of attention and feedforward layers applied sequentially. Each forward pass through the network is a fixed amount of computation, regardless of problem complexity. When the model generates each output token, it performs another forward pass, effectively increasing its computational depth.

By creating intermediate tokens, the model gains additional forward passes, with the results of earlier computations made available via the context for subsequent steps. Consider a problem that requires 10 sequential reasoning steps, meaning 10 distinct pieces of computation that must happen in order, each depending on the results of the previous. With direct answering, the model must compress all 10 steps into a single forward pass through its fixed architecture. This is like asking someone to solve a problem entirely in their head without writing anything down. With chain-of-thought, each step can be computed in a separate forward pass, with the result externalized as text and made available for the next step.

This hypothesis explains why longer, more detailed reasoning chains sometimes improve performance: they give more intermediate computation, more opportunities for the model to process and refine its understanding. It also explains why CoT helps more for complex problems requiring many steps than for simple problems solvable in few steps. The benefit of additional computation scales with the amount of computation the problem requires.

However, this explanation has important limitations. Not all additional tokens improve reasoning. Generating irrelevant steps or padding (text that does not advance the solution) does not help and may hurt by consuming context window space and potentially confusing subsequent processing. The model must generate useful intermediate steps, which requires knowing what computation to perform at each step. The computation extension hypothesis explains why intermediate steps can help but not how the model knows which steps to generate.

Related to computation extension, the scratchpad hypothesis stresses that intermediate tokens act as external memory. This creates a written record that persists and can be referenced. The model's internal state, including its hidden representations and attention patterns, is limited in capacity and transient. But text in the context window persists and can be attended to in later steps. This gives a form of external memory.

When a model writes "2 cans at 3 balls each is 6 balls," it has offloaded the result "6" to the context where it remains accessible throughout all subsequent generation. Without this externalization, the result would need to be maintained in internal representations, which may be unreliable or capacity-limited. The model's internal memory for carrying numerical results across many tokens of generation is imperfect; writing the result down sidesteps this limitation.

This view suggests that chain-of-thought helps most when problems require maintaining intermediate results that would otherwise be lost. This includes problems where you need to compute something, hold onto it, compute something else, and then combine the results. Empirically, CoT gives larger gains on problems with more intermediate values to track, supporting this hypothesis. A problem requiring tracking five separate quantities benefits more from CoT than one requiring tracking only a single value.

The scratchpad hypothesis also explains why certain formats of chain-of-thought are more effective than others. Explicitly labeling intermediate results ("total cost = $15") makes them easier to retrieve later than burying them in prose. The scratchpad is only useful if its contents are organized for easy access.

Another perspective focuses on training data and the patterns the model has learned to associate with certain contexts. During pretraining, models encounter many examples of step-by-step reasoning: mathematics textbooks working through problems, programming tutorials explaining code line by line, and explanatory articles building up to conclusions. When prompted with chain-of-thought examples, models may simply be doing what they've learned: continuing the pattern of reasoning discourse they've seen many times in training.

This hypothesis explains why CoT prompting can be elicited with phrases like "let's think step by step." These phrases frequently precede detailed reasoning in the training data, creating a learned association. When the model encounters such a phrase, it recognizes a context where reasoning is appropriate and likely. Under this view, the model isn't discovering how to reason at inference time. It's recognizing a context where reasoning behavior is expected based on patterns in its training.

The format matching view implies that the quality of CoT reasoning is bounded by the quality of reasoning in training data. If the training corpus contains incorrect or confused reasoning (which it certainly does, alongside correct reasoning), models may reproduce those patterns. Indeed, CoT reasoning in current models often exhibits systematic errors that reflect common human mistakes: confusing correlation with causation, making sign errors in arithmetic, or drawing conclusions that don't fully follow from premises. These errors appear not because the model lacks knowledge, but because it learned from human text that contains such errors.

A more speculative view proposes that sufficiently large transformers can learn to implement general algorithms through their weights: statistical patterns that amount to real computational procedures. Specific patterns of reasoning may be encoded as learned subroutines that activate when appropriate prompts are given. These patterns include decomposition of problems into parts, case analysis for handling different conditions, and recursive problem-solving for nested structures.

Under this view, CoT prompting doesn't create reasoning capability but accesses already-learned algorithmic modules that exist within the model's parameters. These reasoning algorithms develop during pretraining on large text corpora, which contain implicit problem-solving strategies in worked examples and explanatory text. Through exposure to millions of instances of human reasoning, the model distilled something like general reasoning procedures.

Recent mechanistic interpretability work has identified specific attention patterns and MLP activations associated with particular reasoning operations. This gives some support for this view. Researchers have found "induction heads" that copy patterns, circuits that perform comparison operations, and specialized processing for numerical quantities. However, a complete characterization of "reasoning circuits" in transformers remains an open research challenge. We can identify pieces of the puzzle, but the full picture of how transformers implement reasoning algorithmically remains elusive.