Mathematical Reasoning › Chain-of-Thought and Mathematical Problem Solving

The single most impactful technique for improving LLM mathematical reasoning is chain-of-thought prompting, which we explored in detail in the Chain-of-Thought chapter. For mathematical reasoning, the mechanism is particularly clear: by generating intermediate steps, the model creates a scratchpad that makes its reasoning explicit and provides computational structure that constrains subsequent steps.

For a problem like "If x + 3 = 7, what is 2x?", a model without chain-of-thought might generate "8" directly. A model with chain-of-thought might generate: "First, I need to find x. From x + 3 = 7, I subtract 3 from both sides to get x = 4. Then 2x = 2 times 4 = 8." The final answers are the same, but the generation paths are very different.

The difference is not just pedagogical. Generating explicit intermediate steps serves several computational functions that directly affect accuracy.

Error detection. Errors in intermediate steps are visible and can potentially be caught by verification mechanisms, rather than being hidden inside opaque computation. A verifier, whether human or automated, can check "does x = 4 follow from x + 3 = 7?" independently.

Step-by-step constraint. Each intermediate step constrains what comes next. A model that has written "x = 4" is now committed to that value in subsequent steps, reducing the probability of inconsistent reasoning. In information-theoretic terms, the conditioning context at each step includes the correct intermediate result, which narrows the token distribution toward steps consistent with that result.

Length-accuracy trade-off. Longer chain-of-thought reasoning correlates with better accuracy on hard problems. More tokens allow more computation, consistent with the idea that models are using sequential generation as a form of working memory. A problem that requires 8 reasoning steps cannot be solved reliably in a single autoregressive step but can often be solved correctly if the model is allowed to generate those 8 steps explicitly.

Decomposition forcing. Chain-of-thought naturally forces decomposition of complex problems into sub-problems. A multi-step algebra problem becomes a sequence of single-step transformations, each of which is easier than the whole. This mirrors how humans solve hard problems: break them into manageable pieces, solve each piece, and combine the results.

The relationship between chain-of-thought length and accuracy is not linear: at some point, very long chains introduce their own errors. When a model generates 2,000 tokens of reasoning for a problem that requires 200, the extra tokens are not all useful. Some steps may contradict earlier steps; some may introduce new variables that get confused with existing ones. Research on process reward models specifically addresses this by providing feedback on final answers and on the quality of individual reasoning steps, allowing the model to learn which intermediate steps are productive.

One practical implication is that prompting strategy matters enormously for mathematical reasoning. Few-shot chain-of-thought examples should demonstrate the level of detail appropriate for the problem type. For arithmetic word problems, showing 3 to 5 steps is usually sufficient. For competition problems, showing 10 to 15 steps with explicit case analysis and key insights may be necessary. The examples themselves should model good mathematical reasoning practices: defining variables clearly, stating equations before solving them, and verifying answers against original constraints.