Reasoning Foundations › Reasoning Failure Modes

LLMs fail at reasoning in ways that are distinct from human error. Understanding these failure modes is essential for knowing when to trust model output, how to prompt effectively, and what future improvements need to address. The failures are not random: they cluster in predictable patterns that reflect the underlying mechanism of statistical pattern matching over text.

Training on text means that LLMs absorb statistical associations that happen to be present in the training corpus, regardless of whether those associations reflect logical relationships. A model might learn that questions involving "always" or "never" tend to have certain answer patterns, or that a particular problem setup tends to have a particular answer in training data. These spurious correlations can cause the model to give the right answer on familiar problem types even when it has not reasoned through the problem, and to give wrong answers on problem types that look familiar but differ in ways that matter.

A well-documented example involves the natural language inference task, where a model must determine whether a premise entails, contradicts, or is neutral with respect to a hypothesis. Early models achieved high accuracy by exploiting artifacts in the dataset: negation words in the hypothesis tended to predict contradiction, so a model that learned this heuristic without understanding the semantic relationship could score well on the original benchmark while failing on carefully constructed adversarial examples where negation does not signal contradiction.

Spurious correlations are especially insidious in evaluation. If a benchmark has systematic biases in how correct and incorrect options are phrased, a model can achieve high scores by exploiting those biases rather than reasoning. Several well-known NLP benchmarks have been shown to suffer from annotation artifacts of exactly this kind. Gururangan et al. (2018) found that many NLP models exploit these artifacts to achieve high benchmark scores without learning the intended task. This finding drove the creation of adversarial evaluation sets and more careful annotation processes, but the fundamental challenge of separating spurious correlations from systematic reasoning remains.

The practical implication is that benchmark scores on reasoning tasks should always be interpreted with caution. High performance on a static benchmark may reflect successful pattern matching on that benchmark's specific distribution, not a reasoning capability that will transfer to novel inputs. This is why the field has moved toward dynamic benchmarks, holdout sets with post-training cutoff dates, and human-crafted adversarial examples.

LLM reasoning is surprisingly brittle in response to changes in how a problem is stated. The same logical problem, expressed in different words or with different surface formatting, can lead to dramatically different responses. Researchers have shown that small changes to arithmetic word problems, such as changing names, units, or irrelevant context, can flip a model from correct to incorrect on problems it would otherwise solve.

This sensitivity is difficult to reconcile with robust reasoning. A human mathematician solving a proof would not be tripped up by whether the proof uses "therefore" or "thus" or whether the variables are named $x$ and $y$ or $u$ and $v$. LLMs, by contrast, have learned from text that happens to use certain conventions, and deviations from those conventions can disrupt performance.

The practical implication is that the phrasing of prompts matters substantially for LLM reasoning. Prompt engineering is partly an exercise in finding formulations that align with the statistical patterns the model learned during training. A problem phrased in an unusual way may perform much worse than the same problem phrased conventionally, not because the model lacks the underlying capability but because the unusual phrasing does not trigger the right pattern-matching behavior.

Shi et al. (2023) showed a particularly striking version of this: adding irrelevant information to arithmetic word problems causes significant performance drops in LLMs. Humans, understanding the underlying logical structure, can ignore irrelevant information. LLMs are distracted by it, suggesting that they are attending to surface features of the problem rather than identifying the logically relevant quantities.

Closely related to spurious correlations is the failure to generalize. LLMs can solve reasoning problems that resemble problems in their training data but struggle with structurally similar problems when the surface form is unfamiliar. This is most apparent in mathematical reasoning: a model may solve a standard algebraic word problem but fail at an isomorphic problem presented in an unfamiliar domain or with unusual notation.

A clean illustration comes from studies of compositional generalization: the ability to combine familiar elements in novel arrangements. Humans can understand a sentence they have never encountered before because they understand its component words and the grammatical rules for combining them. LLMs, which learn from statistical co-occurrence, struggle more when novel compositions appear in unfamiliar surface forms, even if all component elements are familiar.

The implication is that LLM reasoning generalizes within training distribution but may not generalize outside it. This is a fundamental distinction from the ideal of systematic reasoning, which would produce the same correct answer regardless of surface presentation, because the underlying logical structure is what matters. When you need a model to reason reliably about novel problems, you cannot rely on benchmark performance on familiar problem types as evidence that the model will succeed.

Reasoning chains that require many steps are vulnerable to error accumulation. Each step in a multi-step chain has some probability of error. Even if each individual step is highly reliable, the overall chain fails if any step fails. For a ten-step chain where each step is 95% accurate, the probability of the full chain being correct is only about $0.95^{10} \approx 0.60$. Longer chains compound this problem severely.

The mathematical relationship is straightforward but worth internalizing. If each of $n$ steps in a reasoning chain is independently correct with probability $p$, the probability that the entire chain is correct is:

$$P(\text{chain correct}) = p^n$$

For $p = 0.99$ and $n = 20$, this gives $0.99^{20} \approx 0.82$. For $p = 0.90$ and $n = 10$, it gives $0.90^{10} \approx 0.35$. The chain accuracy falls quickly, even when individual steps are quite reliable. At $p = 0.80$ and $n = 10$, only about $11\%$ of chains will be fully correct. These numbers help explain why long reasoning chains are much harder than short ones, independently of whether individual steps are easy.

There is also a structural issue beyond the independence assumption: errors early in a chain can cascade. If step three produces an incorrect intermediate conclusion, all subsequent steps that depend on step three are now operating on a false premise. Even if the model's conditional reasoning (given this premise, what follows?) is perfect, the final answer will be wrong because the premise was wrong. This cascading failure is particularly dangerous in chains where a single critical inference enables all subsequent reasoning.

This is why verification, checking the outputs of reasoning chains for consistency and correctness, is an important complement to generation. We will cover verification strategies in detail in the Reasoning Verification chapter. The key insight is that generating a reasoning chain and verifying it require different capabilities: generation requires proposing plausible steps, while verification requires checking whether those steps hold. By separating these two functions, we can catch the errors that generation introduces before they propagate.

LLMs sometimes generate reasoning chains that look coherent but contain fabricated steps. The model may cite a theorem that does not exist, perform an algebraic manipulation that is incorrect, or introduce an assumption not warranted by the problem statement. Because the output looks superficially like valid reasoning, these hallucinations can be difficult to catch without careful verification.

This failure mode arises from the same mechanism that causes factual hallucination in general: the model generates plausible-looking text, and in the context of a reasoning problem, plausible-looking reasoning steps are ones that follow the surface form of correct reasoning. The model can generate something that looks like a valid step without any guarantee that it is one. A fake theorem citation looks the same in the surface text as a real one; an incorrect algebraic step has the same format as a correct one.

The insidiousness of hallucinated reasoning is that it often occurs in precisely the situations where verification is hardest: novel problems, specialized domains, or steps that require rare knowledge. When a model hallucinates a step in a domain where you lack the expertise to evaluate it, you may accept the reasoning simply because you cannot check it. This creates a dangerous dynamic where the model is most likely to hallucinate when you are least able to catch it.

Research on mathematical reasoning has found that models can produce chains where all steps look plausible but the argument as a whole is logically invalid, because intermediate steps introduce false assumptions or apply rules incorrectly. Lightman et al. (2023) showed that process reward models, which evaluate each step in a reasoning chain rather than just the final answer, can substantially improve the reliability of multi-step reasoning by providing feedback at each step rather than only at the end.

A specific and recurring failure mode is reversing the direction of a logical relationship. Models sometimes infer that if "A implies B" then "B implies A" (the converse fallacy), or that if "not A" then "not B" (the inverse fallacy). These are among the most common logical fallacies in human reasoning too, which may explain why they appear in model outputs: the training data contains many examples of these fallacies alongside correct reasoning.

The converse fallacy is especially common because natural language often uses implication language loosely. "If it's raining, I'll carry an umbrella" is usually interpreted as a biconditional (I carry an umbrella if and only if it's raining) in everyday conversation, even though logically it only asserts the one direction. An LLM trained on everyday text will naturally learn this loose interpretation and may apply it in formal reasoning contexts where precision matters.

Research has shown that LLMs are particularly prone to reversals when relationships are stated in complex syntactic forms or when there are many plausible confounders. A model might correctly understand that "if the patient has condition X, administer treatment Y" without correctly understanding that "not administering treatment Y does not imply the patient lacks condition X." The asymmetry of the implication is logically obvious but easy to miss in practice, and training data will contain many instances where the two directions were treated as equivalent.

LLMs tend to express incorrect reasoning with the same confident tone as correct reasoning. They rarely hedge appropriately when they are in fact uncertain or when the problem is at the edge of their capability. This makes it hard for users to know when to trust the model's reasoning and when to verify it independently.

Calibration, the alignment between a model's expressed confidence and its actual accuracy, is a significant ongoing research challenge. A well-calibrated model that says it is 80% confident should be correct about 80% of the time. Many LLMs have been found to be overconfident: they express high certainty on claims that are in fact incorrect at substantial rates. Some newer models have been trained to express uncertainty more accurately, using explicit uncertainty language and self-consistency checks, but the problem is not solved.

The overconfidence problem is compounded by the social dynamics of human-AI interaction. Users often treat confident-sounding assertions as more reliable than hedged ones, and a model that never says "I'm not sure" provides less signal for when to seek verification. The ideal would be a model that accurately communicates its own uncertainty, allowing users to invest verification effort where it is most needed. Building reliable uncertainty communication into LLMs is an active area of research.