Chain-of-Thought › Limitations of Chain-of-Thought

Understanding CoT's limitations is as important as understanding its capabilities. CoT is a powerful technique, but it has several failure modes that constrain where and how reliably it can be applied.

The most fundamental limitation is that CoT reasoning chains are not guaranteed to be faithful. A chain is faithful if it accurately represents the causal process by which the model arrived at its answer. A chain is unfaithful if it is a post-hoc rationalization: a plausible-sounding explanation constructed after the answer has already been determined by other means.

Research on CoT faithfulness (Turpin et al., 2023; Lanham et al., 2023) has found evidence of substantial unfaithfulness. Models sometimes produce chains that argue convincingly for the correct answer even when that answer is wrong. Conversely, chains can contain errors in intermediate steps but still arrive at the correct answer, suggesting the final token is influenced by factors beyond the explicitly written reasoning.

One striking demonstration involves biasing the prompt. If the question is presented alongside an answer suggestion ("I think the answer is X, but what do you think?"), models often produce reasoning chains that conclude with X, even when X is incorrect, while the chains are internally coherent and do not reveal that they are accommodating the suggestion. The chain justifies the biased conclusion rather than reaching an independent one.

This unfaithfulness is concerning for any application where the reasoning chain is used as an explanation of model behavior. Seeing a well-structured chain does not guarantee that the chain drove the answer. The chain may be an articulate rationalization of a pattern-matched output. Treating CoT chains as reliable audit trails of model reasoning is therefore unwarranted without further evidence.

Turpin et al. (2023) found that adding a subtle bias to the input, such as telling the model which answer choice is most common among other test-takers, systematically shifted conclusions toward that bias while producing fluent, seemingly independent reasoning chains. This suggests the chain generation and the answer determination are at least partially independent processes, with the chain adapting to justify the answer rather than deriving it.

CoT can amplify hallucination. When generating long reasoning chains, models sometimes introduce facts that are not in the question and are not true. A chain that invents an intermediate quantity, misremembers a factual relationship, or conflates two similar concepts can lead subsequent steps astray even if the reasoning structure is otherwise sound.

The problem is compounded by the sequential nature of generation. An error in step 2 of a 6-step chain affects steps 3 through 6, as each step takes the previous output as input. A small factual error early in the chain can propagate and amplify into a large error in the final answer, particularly for complex problems. This error amplification is the reverse of the benefit that CoT provides: just as correct intermediate results scaffold correct subsequent steps, incorrect intermediate results scaffold incorrect ones.

The risk of hallucinated facts is highest for questions that require drawing on world knowledge, not just performing computation on quantities given in the question. If the question asks "How long does it take to fly from New York to London at cruising speed?", the chain must supply the distance between the cities and the cruising speed of a commercial aircraft. If either fact is wrong, the answer will be wrong regardless of whether the arithmetic is correct.

As seen in the performance plots, CoT prompting is largely ineffective for models below approximately 10 billion parameters. For small models, generating reasoning chains does not produce better outcomes, and sometimes produces worse ones, because the model is not capable of reliably using intermediate reasoning steps as scaffolding.

This scale dependency constrains deployment. Many production applications use smaller, faster models for efficiency. These models cannot reliably benefit from CoT prompting, limiting the technique's applicability in latency-sensitive or cost-constrained settings. CoT fine-tuning partially addresses this, since fine-tuning changes the weights rather than just the context, but even fine-tuned small models typically fall short of large models using few-shot CoT.

The scale dependency also means that the impressive benchmark results in CoT papers are not uniformly reproducible with smaller open-source models. A practitioner evaluating CoT on a 7B parameter model may see little to no improvement and conclude the technique is overrated, when in fact they are below the effective scale threshold.

CoT performance is sensitive to the exact phrasing of exemplars and trigger phrases. Small changes in how exemplars are written, which exemplars are selected, or how the trigger phrase is worded can produce significant accuracy changes. This brittleness makes CoT prompting harder to apply reliably than it first appears.

The sensitivity also makes evaluation difficult. A CoT prompt that achieves high accuracy on a benchmark may do so partly because the specific benchmark questions happen to match the implicit assumptions of the chosen exemplars. Evaluating CoT robustly requires testing across multiple prompt formulations and reporting variance, not just peak performance. Published benchmark results almost always report peak performance with the best-found prompt, which substantially overstates the reliability a practitioner should expect when applying the technique to new problems.

Mitigating prompt brittleness requires treating prompt design as an engineering problem: define a held-out validation set, test multiple phrasings and exemplar sets, report mean and variance across these, and avoid reporting only the best result. This standard is rarely followed in practice, but adhering to it produces more reliable prompts.

Self-consistency and other multi-chain approaches multiply inference cost by the number of samples. At the scale of large models, inference is expensive. Generating 40 reasoning chains per query is 40 times as expensive as single-pass generation, which is often prohibitive in production.

CoT reasoning chains are also typically longer than direct answer generation, adding per-query latency even in the single-chain setting. A direct answer might be 5-10 tokens. A reasoning chain might be 100-200 tokens. This extra generation time adds latency that may be unacceptable for real-time applications with tight latency budgets.

Token cost matters at scale. If a system handles a million queries per day and CoT triples the average token count per query, that is three times the token generation cost. For applications where reasoning quality justifies the cost, this is acceptable. For applications where speed and volume are the priority, it is not.

CoT is well-suited to tasks that decompose into sequential steps where each step builds directly on the previous. It is less well-suited to tasks requiring broad search, where many branches must be explored simultaneously, global reasoning, where the answer depends on integrating many loosely related facts, or tasks where intermediate steps do not have natural language expressions.

Tree-of-thought prompting extends CoT to non-linear reasoning by explicitly maintaining multiple candidate chains and using a search procedure to navigate among them. This improves performance on tasks requiring exploration but at significantly greater computational cost. The right tool depends on the task structure: if the problem is sequential, CoT is appropriate; if it requires exploration, ToT or program-aided approaches may be necessary.

CoT also struggles with tasks that require very precise symbolic manipulation, such as multi-step algebraic transformations or formal logic derivations. For these tasks, hybrid approaches that combine natural language CoT with a symbolic solver, analogous to PAL but for symbolic rather than numeric computation, tend to outperform pure textual CoT.

Given these limitations, it is worth being explicit about when CoT is likely to help. CoT is most valuable when:

• The task requires at least two to three sequential steps where each depends on the previous.
• The model you are using has at least 10-15 billion parameters.
• Accuracy is more important than latency or cost.
• You need interpretable outputs that can be audited or debugged.

CoT is unlikely to help when:

• The model is small (below about 7B parameters).
• The task is simple one-step retrieval or classification.
• Latency or token cost is a binding constraint.
• The task requires precise symbolic manipulation better handled by code.

For the code case, PAL or code-interpreter approaches are nearly always better than pure CoT for arithmetic-heavy problems. For the scale case, CoT fine-tuning on a small model is worth exploring, since it partially circumvents the scale threshold by baking the reasoning behavior into the weights.