Chain-of-Thought Emergence › Scaling Behavior of Chain-of-Thought

The relationship between model scale and chain-of-thought effectiveness reveals important insights about how reasoning capabilities develop in language models. Understanding this relationship is important both for practitioners deciding when to use CoT and for researchers seeking to understand the nature of reasoning in neural networks.

Empirical studies consistently find that chain-of-thought prompting gives no benefit, and often hurts performance, for models below approximately 10 billion parameters. This threshold isn't sharp (representing a region where CoT effectiveness transitions from absent to present), but it's remarkably consistent across different model families, training approaches, and benchmark tasks. The existence of this threshold suggests something basic about what's required for coherent multi-step reasoning.

Below the threshold, smaller models prompted with chain-of-thought examples often produce malformed reasoning chains that reveal the absence of real reasoning capability. They might repeat steps mechanically without advancing toward a solution. They might generate non sequiturs, statements that have no logical connection to what came before. Or they might produce text that resembles reasoning in surface form but lacks logical coherence, using mathematical symbols and reasoning words without applying them correctly. The model has learned to mimic the format of step-by-step reasoning from its training data without having developed the underlying capability to reason correctly.

At and above the threshold, models begin creating coherent chains with valid intermediate steps. The transition is qualitative, not just quantitative. It's not that larger models reason slightly better; it's that they reason in a fundamentally different way. The quality of reasoning continues to improve with scale even above the threshold; larger models make fewer logical errors, handle more complex reasoning patterns, and generalize better to novel problem types. But the important transition is from incoherent to coherent, and this happens within a relatively narrow band of model sizes.

The scaling behavior of CoT can be understood by comparing performance curves across model sizes for different prompting strategies. These curves reveal the distinctive nature of chain-of-thought as an emergent capability.

For standard (non-CoT) prompting, performance typically increases smoothly with log model size, following patterns consistent with the scaling laws we discussed earlier. The curve rises gradually and shows no sudden transitions. A model twice as large is somewhat better, and a model ten times as large is noticeably better, but the improvement is continuous and predictable.

With chain-of-thought prompting, the curve has a sharply different shape. Below the threshold, CoT performance matches or falls below standard prompting. The additional reasoning steps give no benefit and may introduce confusion or errors. Above the threshold, CoT performance rises steeply, often surpassing what would be predicted by simple extrapolation of the standard prompting curve. The gap between standard and CoT performance widens as models grow larger, meaning that CoT becomes increasingly useful as models scale up.

This divergence suggests that CoT prompting doesn't just amplify existing capabilities in a straightforward way; it makes qualitatively different processing available at scale. The reasoning circuits activated by CoT appear to develop only in sufficiently large models, and once present, they enable a different mode of problem-solving that continues to improve with further scaling.

Several hypotheses explain why chain-of-thought reasoning requires scale, each showing different aspects of what makes multi-step reasoning computationally demanding.

Knowledge integration:

Multi-step reasoning requires combining multiple pieces of knowledge (mathematical facts, logical rules, and world knowledge) in sequence. A model solving a word problem must know arithmetic operations, understand the semantics of words like "each" and "total," recognize the structure of the problem, and apply all of this knowledge in the correct order. Larger models have greater capacity to store and integrate knowledge, letting them to bring more needed information to bear on each step. A model must know both that multiplication precedes addition and the specific multiplication facts needed, then apply both correctly in sequence, a coordination problem that requires substantial representational capacity.

Context tracking:

Reasoning chains require maintaining state across many tokens. The model must remember earlier computed values (like "the total cost is $15"), track what has and hasn't been established ("we know the price per item but haven't yet calculated the quantity"), and maintain coherence over extended generation. This is in effect a working memory task, and the representational capacity in larger models may let more reliable context tracking. With more parameters devoted to processing the context, models can maintain more accurate representations of what has been computed and what remains to be done.

Compositional generalization:

Reasoning often requires applying known operations in novel combinations. A model might have seen problems involving division and problems involving subtraction, but never a problem requiring subtraction followed by division with the result. Larger models appear to develop more modular, composable representations that support this generalization. They can chain operations they've seen separately into sequences they haven't encountered in training. This compositional capability seems to require scale to develop reliably.

Error accumulation:

Each step in a reasoning chain can introduce errors, whether in arithmetic, logic, or interpretation. Longer chains accumulate more errors unless each step is highly accurate. If each step has 95% accuracy, a ten-step chain has only about 60% accuracy ($0.95^{10} \approx 0.60$). Larger models make fewer per-step errors, letting them to maintain accuracy across longer chains. This error reduction compounds multiplicatively over the length of the reasoning chain, making scale increasingly useful for complex problems.