Reasoning Foundations: Types, LLMs, Failure Modes

When a language model is asked "If all mammals are warm-blooded, and whales are mammals, are whales warm-blooded?", it can answer correctly. When asked to solve a multi-step arithmetic problem, it can often get the right answer. When given a logic puzzle, it can reason toward a solution. But how does this happen? What does "reasoning" even mean for a system that was trained to predict the next token? And where does it break down?

These questions are not philosophical abstractions. If you are building applications that depend on model reasoning, understanding the mechanism matters. It determines which problems you can trust the model to handle, which prompting strategies help, and what failure patterns to watch for in production. A model that looks like it reasons well on a benchmark might be exploiting statistical patterns in the test data rather than performing inference, and the two behaviors have very different implications for reliability.

This chapter establishes the conceptual foundation for reasoning in AI systems. Before diving into specific prompting strategies or evaluation benchmarks, you need a clear picture of what reasoning is, what forms it takes, how large language models (LLMs) approximate it, and where that approximation fails. This foundation will inform everything that follows in the reasoning section of the handbook, from chain-of-thought prompting through mathematical reasoning and verification.

Reasoning is the process of drawing conclusions from available information. It is not the same as remembering facts: recall is retrieval, whereas reasoning is inference. When you know that Paris is in France and that France is in Europe, you can infer that Paris is in Europe even if no one told you that directly. That inference step, from premises to a new conclusion, is reasoning.

The distinction matters deeply for how we evaluate and use AI systems. A system that has memorized that Paris is in Europe will answer the question correctly, but it would fail the moment you asked about a newly created country or a hypothetical political reorganization. A system that can reason from the premises, working through the containment relationship, would handle novel cases correctly because the reasoning process is not tied to any specific memorized fact.

The study of reasoning has deep roots in philosophy and logic. Aristotle codified deductive inference through the theory of syllogisms in the fourth century BCE, and logicians from Leibniz through Frege and Russell spent centuries formalizing the rules by which valid conclusions follow from premises. This work eventually led to formal logic, the mathematical theory of proof, and the computational theory of inference that underlies automated theorem proving and formal verification. Over the centuries, researchers identified several distinct forms of reasoning. Understanding these forms matters because LLMs engage with some of them more reliably than others, and knowing which is which helps you understand why a model might succeed at one problem type while failing at another.

Deductive reasoning moves from general premises to a specific, necessary conclusion. The classic form is the syllogism:

• All men are mortal.
• Socrates is a man.
• Therefore, Socrates is mortal.

The key feature of deduction is that if the premises are true and the argument is valid, the conclusion must be true. Deductive reasoning is truth-preserving: you cannot have true premises and a valid argument leading to a false conclusion. This makes deduction the gold standard for mathematical proof and formal logic.

It is worth pausing on why truth-preservation is such a powerful property. In everyday reasoning, we often accept conclusions tentatively, knowing they might be revised. Deduction gives us something stronger: a guarantee. If we accept the premises and accept that the form of the argument is valid, we have no choice but to accept the conclusion. This is why mathematics proceeds by proof rather than by experiment: a mathematical theorem, once proved, is necessarily true given the axioms, not merely probable.

In an AI context, deductive reasoning appears in tasks like logical entailment (given premises, what necessarily follows?), constraint satisfaction, and theorem proving. These tasks have crisp right answers, which makes them well-suited for automated evaluation. They also make the strongest demands on a reasoning system: there is no room for "probably" or "usually" in a deductively valid inference.

A particularly challenging variant for LLMs is reasoning about negation and universal quantification. "All A are B" and "No A is B" have very different logical content, but they share surface-level similarity that can mislead a model relying on statistical patterns. Similarly, "some A are B" and "all A are B" have overlapping implications that are easy to confuse. Human logicians use formal notation precisely to avoid these surface-level traps; LLMs must recover the underlying logic through the fuzzy medium of natural language.

Inductive reasoning moves from specific observations to a general rule. If you observe that the sun has risen every morning of your life, you inductively conclude that the sun rises every morning. Unlike deduction, inductive conclusions are not guaranteed: they are probabilistic, not necessary. Induction is the engine of scientific discovery, but the conclusions it produces are always defeasible, meaning they can be overturned by new evidence.

The philosopher David Hume identified what he called the "problem of induction" in the eighteenth century: no finite number of confirming observations can logically guarantee a universal generalization. You might observe a million white swans, but the inductive conclusion "all swans are white" turns out to be false once you encounter Australian black swans. Induction is epistemically risky, but it is also indispensable. Science would be impossible without it.

In AI, inductive reasoning underlies learning itself. When a model generalizes from training examples to unseen inputs, it is performing a form of inductive inference. Tasks like pattern recognition, analogy completion, and next-step prediction in a sequence all have an inductive character. The training process can be understood as large-scale induction: the model sees many examples of text and induces statistical regularities that allow it to predict new text.

This means there is a deep sense in which any LLM is a product of inductive reasoning: the parameters encode inductive generalizations from the training corpus. The challenge is that the model has no explicit mechanism to represent the uncertainty inherent in inductive conclusions. It generates outputs as if they were certain, even when the underlying generalization is fragile or the test case is near the edge of the distribution.

Abductive reasoning, sometimes called "inference to the best explanation," works differently from both deduction and induction. Given an observation, abduction asks: what hypothesis best explains it? If you come home to find the door open and muddy footprints on the floor, you might abductively infer that someone entered while you were away. This is the most likely explanation, but it is not the only one: perhaps a pet knocked the door open, or the wind blew it open after rain splashed mud inside.

The philosopher Charles Sanders Peirce introduced the term "abduction" in the late nineteenth century to describe this form of inference. He recognized that abduction is how we generate hypotheses in the first place: we observe something surprising, and we reason backward to what might have caused it. Deduction then tells us what would follow if the hypothesis were true, and induction tests that hypothesis against further observations. Together, these three forms of inference constitute the cycle of scientific inquiry.

Abduction is the reasoning mode that underlies diagnosis, detective work, and common-sense interpretation of ambiguous situations. When a physician looks at a patient's symptoms and reasons toward a diagnosis, they are performing abduction: finding the disease that best explains the presenting symptoms. When a detective names a suspect, they are abducing: choosing the explanation that best fits the evidence. Sherlock Holmes's famous "deductions" are, technically, abductions, since they generate the most plausible explanation of observations rather than deriving necessary conclusions.

It is also the most vulnerable to error: the "best" explanation is only as good as the set of hypotheses under consideration, and important alternatives are easily missed. A physician who has not considered a rare disease will not diagnose it, even if it is the true explanation. A detective who fixates on a prime suspect may not generate the alternative hypothesis that would exonerate them.

LLMs engage in something resembling abduction when they interpret ambiguous prompts, complete stories, or answer "why" questions. The model generates the most plausible explanation given its training distribution. This can work remarkably well when the question has a common answer in the training data, and can fail badly when the true explanation is rare or when the question is designed to probe explanations that differ from the statistical mode.

Analogical reasoning transfers knowledge from a familiar domain to a new one by identifying structural similarity. When physicists in the early twentieth century used planetary orbital models to reason about electron behavior in atoms, they were reasoning by analogy. The Bohr model of the atom, with electrons orbiting a nucleus the way planets orbit the sun, was explicitly analogical. The analogy guided intuition and hypothesis formation, even though the underlying physics eventually proved different: electrons obey quantum mechanics, not Newtonian gravity.

The history of science is full of such productive analogies. James Clerk Maxwell used analogies between electromagnetic and mechanical phenomena to develop his equations of electromagnetism. Darwin's mechanism of natural selection was partly inspired by an analogy with selective breeding. The mathematician George Polya argued that analogy is one of the primary tools of mathematical discovery, allowing mathematicians to transfer proof strategies from known results to new ones.

Analogies are powerful because they let you apply previously acquired reasoning patterns to new situations without starting from scratch. They are also dangerous because structural similarity can be superficial: two situations that look analogous may differ in ways that matter substantially. The analogy between electromagnetic waves and mechanical waves correctly predicted many of electromagnetism's properties, but it also misled physicists into searching for a luminiferous ether that did not exist.

LLMs are remarkably good at surface-level analogies and quite capable of deeper structural analogies when the relevant structure appears frequently in training data. They can complete "doctor is to hospital as teacher is to ___" with ease, and can often solve analogical reasoning problems from IQ-style tests. Where they struggle is with novel domains where the structural mapping requires abstract reasoning rather than pattern matching over familiar analogical pairs.

Causal reasoning identifies cause-and-effect relationships. It is distinct from correlation: just because two events co-occur does not mean one causes the other. Causal reasoning requires understanding interventions (what would happen if X were changed?) and counterfactuals (what would have happened if X had been different?). These questions cannot be answered from observed data alone without additional assumptions about the causal structure of the system.

The statistician Judea Pearl, who received the Turing Award in 2011 for his work on probabilistic reasoning, formalized the mathematics of causal inference through directed acyclic graphs and the do-calculus. Pearl's framework distinguishes sharply between observational claims ("patients who take aspirin have lower rates of heart disease") and interventional claims ("giving patients aspirin reduces their risk of heart disease"). The first can be established from observational data; the second requires either a randomized experiment or strong causal assumptions. Confounding, where a third variable causally affects both the apparent cause and the apparent effect, is the central challenge.

Causal reasoning is arguably the hardest form for current LLMs. The challenge is that statistical associations in training data capture correlations, not causal mechanisms. A model trained on medical literature might learn that patients who receive chemotherapy have increased inflammation markers, but this does not tell it whether chemotherapy causes inflammation or whether a confounding factor (the cancer itself) is responsible. Disentangling cause from correlation requires interventional data or explicit causal modeling, neither of which is straightforwardly available from text.

There are subtler failures too. LLMs sometimes reverse causal direction: given that "rain causes wet roads," they might affirm that "wet roads cause rain" in certain surface contexts. They may treat temporal precedence as sufficient for causality, concluding that because A happened before B, A caused B (the post hoc fallacy). And they may fail to reason about counterfactuals reliably: "would the road be wet if it hadn't rained?" is a counterfactual question that requires explicitly imagining a world different from the observed one.

LLMs are trained to predict the next token given a sequence of prior tokens. Nothing in this objective explicitly trains the model to reason. And yet these systems display behavior that looks like reasoning across a wide range of tasks. How should we understand this?

The question is not academic. If LLMs reason through systematic inference, their outputs in novel situations should be as reliable as their outputs in familiar ones. If they pattern-match over reasoning-shaped text, their reliability degrades precisely where you most need it: in novel situations, at the edge of the training distribution, in domains underrepresented in the training data. The mechanism matters for trust.

The most parsimonious explanation is that LLMs have learned reasoning as a pattern. Training corpora contain enormous quantities of text in which people reason explicitly: mathematical proofs, logical arguments, scientific explanations, philosophical treatises, tutoring dialogues, and informal arguments in online discussions. When a model is trained on this text, it learns the statistical patterns of how reasoning steps follow from one another.

From this perspective, LLM reasoning is a form of sophisticated pattern matching over reasoning-shaped text. The model has seen many chains of the form "if A then B; we have A; therefore B" and has learned to continue such chains in a way that looks correct. This is not nothing: the resulting behavior can be useful and remarkably reliable on familiar problem types.

But this view also explains a key vulnerability. If the model is reasoning by pattern, then novel problem structures that do not match anything in training distribution will trip it up. The model may generate a confident-looking but incorrect continuation because the surface structure resembles something it has seen, even though the underlying logic is different. A model trained on many examples of standard arithmetic word problems may fail at an isomorphic problem where the surface format differs, not because the underlying reasoning is harder, but because the pattern is less familiar.

This perspective also reframes what "emergent reasoning" means. When a model suddenly performs much better on a reasoning task at a certain scale, it may be because it has enough capacity to match reasoning patterns at a finer level of granularity, not because it has crossed a qualitative threshold into systematic inference. The debate between these interpretations is ongoing and not yet resolved.

As LLMs scaled in size and training compute, certain reasoning capabilities appeared that were not present, or not reliably present, in smaller models. This phenomenon, discussed in the Emergent Capabilities chapter, creates an interesting situation: researchers observed that models below a certain scale would essentially fail at multi-step reasoning tasks, while models above that scale would perform significantly better.

The most studied example is multi-step arithmetic. Small models fail badly at problems requiring more than one or two arithmetic operations. Larger models solve them with much higher reliability. Whether this reflects a phase transition or a smooth improvement that appears sharp only because evaluation metrics are coarse-grained is still debated. Wei et al. (2022) documented many such emergent capabilities and proposed that they arise from the combinatorial nature of reasoning: many sub-skills are each individually weak in small models but combine to produce functional reasoning only when all are present at sufficient strength.

The emergence of chain-of-thought reasoning is particularly interesting. Smaller models do not benefit from being prompted to show intermediate steps: they generate intermediate steps that are wrong, which then leads to wrong final answers. Larger models benefit substantially from the same prompting strategy. This suggests that the model must first acquire reliable intermediate reasoning before the scaffolding of showing work can help.

Scaling also affects the breadth of the failure patterns. Smaller models fail at a wide range of reasoning tasks and in consistent ways that reflect limited capacity. Larger models fail more sporadically and in ways that are harder to predict from surface features of the problem. This makes larger models harder to characterize but also harder to probe systematically, which is part of why evaluation methodology has become increasingly sophisticated alongside the models themselves.

The transformer architecture supports reasoning through several mechanisms that are worth understanding in detail, because they directly determine what kinds of reasoning the architecture can and cannot do well.

The attention mechanism allows the model to reference any prior token when generating the next one. In a reasoning chain, earlier steps provide the premises and intermediate conclusions that later steps depend on. Attention lets the model retrieve those earlier steps precisely, regardless of how far back in the sequence they appear. This is qualitatively different from recurrent architectures, which had to compress the entire prior context into a fixed-size state vector. Attention gives transformers essentially direct access to prior reasoning steps, which is why transformer-based models outperform recurrent models so dramatically on tasks that require integrating information across long contexts.

To understand what "direct access" means here in practice. The attention mechanism computes, for each query position, a weighted sum of all value vectors in the context, where the weights are determined by the similarity between the query and each key. This means the model can attend strongly to a specific earlier token if its key matches the current query, effectively reading from that position. In reasoning tasks, this mechanism allows a step that references an earlier intermediate conclusion to reliably retrieve that conclusion's content, rather than relying on it having been compressed into the hidden state.

Residual connections and layer depth allow for iterative refinement. Information can pass through many layers of processing, with each layer potentially contributing additional abstraction or inference. While it is an oversimplification to say each layer performs a distinct reasoning step, the depth of transformer networks does allow more complex transformations than shallow models. Empirical studies of transformer internals have shown that different layers tend to specialize in different types of computation, with earlier layers handling local syntactic patterns and later layers handling more abstract semantic relationships.

The token-by-token generation process means that reasoning is externalized into the model's own output. When a model generates "Let's think step by step" and then produces a chain of intermediate conclusions, it is using its own output as working memory. Each token the model generates becomes part of the context for generating the next token. This means the model can, in principle, perform multi-step reasoning by generating intermediate steps that it then attends to when computing the final answer.

This last point underlies chain-of-thought prompting. It is the basis for chain-of-thought prompting (covered in the next chapter) and explains why prompting the model to show its work often improves accuracy: the intermediate steps are not just for human readability, they extend what the model can compute by extending the effective context. The intermediate tokens function as a kind of working memory, allowing the model to store and retrieve partial results in a way that the fixed parameter space alone cannot support.

The feed-forward layers within each transformer block also contribute to reasoning by acting as key-value stores that retrieve factual associations and arithmetic operations. Research by Geva et al. (2021) showed that feed-forward layers can be understood as storing factual associations that the model queries during forward passes. This helps explain why factual retrieval often works well, while it also suggests a limitation: the stored associations are fixed at training time and cannot be updated dynamically during reasoning.

Reasoning problems often require maintaining and manipulating multiple pieces of information simultaneously. Context length directly constrains what information is available when any given token is generated. A model with a short context window cannot hold long reasoning chains or reference problems with many premises.

Consider a legal reasoning problem where you must integrate ten clauses of a contract to determine whether a particular action is permitted. If the contract exceeds the context window, the model cannot see all relevant clauses simultaneously. It must either reason from partial information, which introduces errors, or rely on retrieval mechanisms to fetch relevant clauses, which introduces its own failure modes. The same limitation applies to mathematical proofs with many lemmas, to strategic games with long histories, and to any domain where understanding requires integrating many pieces of information.

The expansion of context windows in modern LLMs, from a few hundred tokens to tens or hundreds of thousands, has significantly expanded the complexity of reasoning tasks these models can tackle. The GPT-3 model in 2020 worked with 2,048 tokens; by 2024, models like Claude and Gemini were processing contexts of hundreds of thousands of tokens. This quantitative increase enables new reasoning tasks become possible when the model can hold a full textbook chapter, a complete codebase, or an entire conversation history in context simultaneously.

But longer context is not automatically better: there is evidence that LLM attention degrades for information in the middle of very long contexts. Liu et al. (2023) documented the "lost in the middle" phenomenon: when the same fact is placed at the beginning, middle, or end of a long context, models perform significantly better when it is at the beginning or end, with performance dropping for middle positions. This suggests that even with long context windows, models do not attend uniformly to all positions, and information in the middle of a long context is at risk of being underweighted. Very long reasoning chains also accumulate errors. We will examine these failure modes in detail later in this chapter.

LLMs fail at reasoning in ways that are distinct from human error. Understanding these failure modes is essential for knowing when to trust model output, how to prompt effectively, and what future improvements need to address. The failures are not random: they cluster in predictable patterns that reflect the underlying mechanism of statistical pattern matching over text.

Training on text means that LLMs absorb statistical associations that happen to be present in the training corpus, regardless of whether those associations reflect logical relationships. A model might learn that questions involving "always" or "never" tend to have certain answer patterns, or that a particular problem setup tends to have a particular answer in training data. These spurious correlations can cause the model to give the right answer on familiar problem types even when it has not reasoned through the problem, and to give wrong answers on problem types that look familiar but differ in ways that matter.

A well-documented example involves the natural language inference task, where a model must determine whether a premise entails, contradicts, or is neutral with respect to a hypothesis. Early models achieved high accuracy by exploiting artifacts in the dataset: negation words in the hypothesis tended to predict contradiction, so a model that learned this heuristic without understanding the semantic relationship could score well on the original benchmark while failing on carefully constructed adversarial examples where negation does not signal contradiction.

Spurious correlations are especially insidious in evaluation. If a benchmark has systematic biases in how correct and incorrect options are phrased, a model can achieve high scores by exploiting those biases rather than reasoning. Several well-known NLP benchmarks have been shown to suffer from annotation artifacts of exactly this kind. Gururangan et al. (2018) found that many NLP models exploit these artifacts to achieve high benchmark scores without learning the intended task. This finding drove the creation of adversarial evaluation sets and more careful annotation processes, but the fundamental challenge of separating spurious correlations from systematic reasoning remains.

The practical implication is that benchmark scores on reasoning tasks should always be interpreted with caution. High performance on a static benchmark may reflect successful pattern matching on that benchmark's specific distribution, not a reasoning capability that will transfer to novel inputs. This is why the field has moved toward dynamic benchmarks, holdout sets with post-training cutoff dates, and human-crafted adversarial examples.

LLM reasoning is surprisingly brittle in response to changes in how a problem is stated. The same logical problem, expressed in different words or with different surface formatting, can lead to dramatically different responses. Researchers have shown that small changes to arithmetic word problems, such as changing names, units, or irrelevant context, can flip a model from correct to incorrect on problems it would otherwise solve.

This sensitivity is difficult to reconcile with robust reasoning. A human mathematician solving a proof would not be tripped up by whether the proof uses "therefore" or "thus" or whether the variables are named $x$ and $y$ or $u$ and $v$. LLMs, by contrast, have learned from text that happens to use certain conventions, and deviations from those conventions can disrupt performance.

The practical implication is that the phrasing of prompts matters substantially for LLM reasoning. Prompt engineering is partly an exercise in finding formulations that align with the statistical patterns the model learned during training. A problem phrased in an unusual way may perform much worse than the same problem phrased conventionally, not because the model lacks the underlying capability but because the unusual phrasing does not trigger the right pattern-matching behavior.

Shi et al. (2023) showed a particularly striking version of this: adding irrelevant information to arithmetic word problems causes significant performance drops in LLMs. Humans, understanding the underlying logical structure, can ignore irrelevant information. LLMs are distracted by it, suggesting that they are attending to surface features of the problem rather than identifying the logically relevant quantities.

Closely related to spurious correlations is the failure to generalize. LLMs can solve reasoning problems that resemble problems in their training data but struggle with structurally similar problems when the surface form is unfamiliar. This is most apparent in mathematical reasoning: a model may solve a standard algebraic word problem but fail at an isomorphic problem presented in an unfamiliar domain or with unusual notation.

A clean illustration comes from studies of compositional generalization: the ability to combine familiar elements in novel arrangements. Humans can understand a sentence they have never encountered before because they understand its component words and the grammatical rules for combining them. LLMs, which learn from statistical co-occurrence, struggle more when novel compositions appear in unfamiliar surface forms, even if all component elements are familiar.

The implication is that LLM reasoning generalizes within training distribution but may not generalize outside it. This is a fundamental distinction from the ideal of systematic reasoning, which would produce the same correct answer regardless of surface presentation, because the underlying logical structure is what matters. When you need a model to reason reliably about novel problems, you cannot rely on benchmark performance on familiar problem types as evidence that the model will succeed.

Reasoning chains that require many steps are vulnerable to error accumulation. Each step in a multi-step chain has some probability of error. Even if each individual step is highly reliable, the overall chain fails if any step fails. For a ten-step chain where each step is 95% accurate, the probability of the full chain being correct is only about $0.95^{10} \approx 0.60$. Longer chains compound this problem severely.

The mathematical relationship is straightforward but worth internalizing. If each of $n$ steps in a reasoning chain is independently correct with probability $p$, the probability that the entire chain is correct is:

For $p = 0.99$ and $n = 20$, this gives $0.99^{20} \approx 0.82$. For $p = 0.90$ and $n = 10$, it gives $0.90^{10} \approx 0.35$. The chain accuracy falls quickly, even when individual steps are quite reliable. At $p = 0.80$ and $n = 10$, only about $11\%$ of chains will be fully correct. These numbers help explain why long reasoning chains are much harder than short ones, independently of whether individual steps are easy.

There is also a structural issue beyond the independence assumption: errors early in a chain can cascade. If step three produces an incorrect intermediate conclusion, all subsequent steps that depend on step three are now operating on a false premise. Even if the model's conditional reasoning (given this premise, what follows?) is perfect, the final answer will be wrong because the premise was wrong. This cascading failure is particularly dangerous in chains where a single critical inference enables all subsequent reasoning.

This is why verification, checking the outputs of reasoning chains for consistency and correctness, is an important complement to generation. We will cover verification strategies in detail in the Reasoning Verification chapter. The key insight is that generating a reasoning chain and verifying it require different capabilities: generation requires proposing plausible steps, while verification requires checking whether those steps hold. By separating these two functions, we can catch the errors that generation introduces before they propagate.

LLMs sometimes generate reasoning chains that look coherent but contain fabricated steps. The model may cite a theorem that does not exist, perform an algebraic manipulation that is incorrect, or introduce an assumption not warranted by the problem statement. Because the output looks superficially like valid reasoning, these hallucinations can be difficult to catch without careful verification.

This failure mode arises from the same mechanism that causes factual hallucination in general: the model generates plausible-looking text, and in the context of a reasoning problem, plausible-looking reasoning steps are ones that follow the surface form of correct reasoning. The model can generate something that looks like a valid step without any guarantee that it is one. A fake theorem citation looks the same in the surface text as a real one; an incorrect algebraic step has the same format as a correct one.

The insidiousness of hallucinated reasoning is that it often occurs in precisely the situations where verification is hardest: novel problems, specialized domains, or steps that require rare knowledge. When a model hallucinates a step in a domain where you lack the expertise to evaluate it, you may accept the reasoning simply because you cannot check it. This creates a dangerous dynamic where the model is most likely to hallucinate when you are least able to catch it.

Research on mathematical reasoning has found that models can produce chains where all steps look plausible but the argument as a whole is logically invalid, because intermediate steps introduce false assumptions or apply rules incorrectly. Lightman et al. (2023) showed that process reward models, which evaluate each step in a reasoning chain rather than just the final answer, can substantially improve the reliability of multi-step reasoning by providing feedback at each step rather than only at the end.

A specific and recurring failure mode is reversing the direction of a logical relationship. Models sometimes infer that if "A implies B" then "B implies A" (the converse fallacy), or that if "not A" then "not B" (the inverse fallacy). These are among the most common logical fallacies in human reasoning too, which may explain why they appear in model outputs: the training data contains many examples of these fallacies alongside correct reasoning.

The converse fallacy is especially common because natural language often uses implication language loosely. "If it's raining, I'll carry an umbrella" is usually interpreted as a biconditional (I carry an umbrella if and only if it's raining) in everyday conversation, even though logically it only asserts the one direction. An LLM trained on everyday text will naturally learn this loose interpretation and may apply it in formal reasoning contexts where precision matters.

Research has shown that LLMs are particularly prone to reversals when relationships are stated in complex syntactic forms or when there are many plausible confounders. A model might correctly understand that "if the patient has condition X, administer treatment Y" without correctly understanding that "not administering treatment Y does not imply the patient lacks condition X." The asymmetry of the implication is logically obvious but easy to miss in practice, and training data will contain many instances where the two directions were treated as equivalent.

LLMs tend to express incorrect reasoning with the same confident tone as correct reasoning. They rarely hedge appropriately when they are in fact uncertain or when the problem is at the edge of their capability. This makes it hard for users to know when to trust the model's reasoning and when to verify it independently.

Calibration, the alignment between a model's expressed confidence and its actual accuracy, is a significant ongoing research challenge. A well-calibrated model that says it is 80% confident should be correct about 80% of the time. Many LLMs have been found to be overconfident: they express high certainty on claims that are in fact incorrect at substantial rates. Some newer models have been trained to express uncertainty more accurately, using explicit uncertainty language and self-consistency checks, but the problem is not solved.

The overconfidence problem is compounded by the social dynamics of human-AI interaction. Users often treat confident-sounding assertions as more reliable than hedged ones, and a model that never says "I'm not sure" provides less signal for when to seek verification. The ideal would be a model that accurately communicates its own uncertainty, allowing users to invest verification effort where it is most needed. Building reliable uncertainty communication into LLMs is an active area of research.

How do we measure reasoning capability in LLMs? This is a harder problem than it might appear, and the field has developed increasingly sophisticated approaches in response to lessons learned from earlier benchmarks. The history of reasoning evaluation is partly a history of benchmarks being constructed, models achieving high scores on them, and then the community discovering that the high scores reflect something other than the intended capability.

The evaluation of reasoning in LLMs began with standard multiple-choice and short-answer benchmarks. These include datasets designed to test a broad range of knowledge and reasoning capabilities:

• MMLU (Massive Multitask Language Understanding): Tests knowledge and reasoning across 57 academic subjects including mathematics, science, law, and humanities. Each question is multiple-choice with four options. Introduced by Hendrycks et al. (2020), MMLU was originally a strong discriminator between model sizes. MMLU scores are now saturating as frontier models approach ceiling performance, with some models exceeding 90% accuracy on the full test.
• ARC (AI2 Reasoning Challenge): A dataset of grade-school science questions requiring reasoning about scientific concepts. Split into an Easy and a Challenge set, the challenge questions were specifically designed to be hard for retrieval-based systems, requiring application of knowledge rather than simple lookup.
• HellaSwag: Commonsense reasoning about how activities proceed. Models must choose the most plausible continuation of a scenario. Designed by Zellers et al. (2019) to be difficult for models that rely on statistical co-occurrence patterns; the wrong answers were generated by a language model and specifically filtered to be hard to distinguish from correct answers.
• WinoGrande: A large-scale Winograd schema challenge designed to require commonsense reasoning for pronoun resolution. Pronoun references are ambiguous without reasoning about context, and the dataset was constructed to minimize the effectiveness of simple co-occurrence statistics.
• BIG-Bench: A massive collaborative benchmark with over 200 tasks spanning diverse reasoning modalities, including logical deduction, analogical reasoning, mathematical reasoning, and more. Designed to be difficult enough to remain discriminative for several model generations.

Each of these benchmarks represents a significant community investment in building evaluations that resist spurious-correlation shortcuts. And yet each has also been found, in various studies, to admit partial exploitation through non-reasoning strategies, illustrating how difficult it is to construct reasoning benchmarks that cannot be gamed.

Mathematical reasoning deserves special treatment because math has clear right answers and requires explicit multi-step reasoning. Key benchmarks include:

• GSM8K (Grade School Math 8,000): A dataset of grade-school math word problems requiring multi-step arithmetic, released by Cobbe et al. (2021). Each problem has a human-written solution chain. Became a standard benchmark for evaluating multi-step arithmetic in LLMs. State-of-the-art models now approach near-perfect performance on GSM8K, prompting the development of harder alternatives.
• MATH: A dataset of competition mathematics problems at much higher difficulty, covering algebra, geometry, number theory, calculus, and combinatorics, released by Hendrycks et al. (2021). State-of-the-art models still fall short of human expert performance on the hardest problems, particularly those requiring novel proof strategies. MATH has remained a useful discriminator even as models have improved.
• MGSM: Multilingual versions of GSM8K, testing whether reasoning capability generalizes across languages. Multilingual evaluation is important because it probes whether the model has learned the underlying mathematical structure or merely the English-language conventions of arithmetic word problems.
• AIME and AMC: Problems from American high school mathematics competitions, which require substantially more sophisticated mathematical reasoning than GSM8K. A small number of problems from AIME are now used to evaluate frontier models' mathematical capabilities at the high end.

The trajectory of performance on mathematical benchmarks illustrates the scaling of reasoning capability clearly. GPT-3 achieved around 5% on GSM8K; GPT-4 achieved above 90%. The MATH benchmark saw similar progression over a shorter period. This rapid improvement has driven the construction of ever-harder benchmarks to maintain discriminative power.

• LogiQA: Logical reasoning questions drawn from Chinese civil service exam reading comprehension sections. Requires multi-step logical inference to determine correct answers among distractors designed to test for common fallacies.
• ReClor: Logical reasoning questions from standardized tests like GMAT and LSAT, which were designed to test the kind of logical reasoning required for graduate education. Provides a challenging and ecologically valid test of logical inference.
• FOLIO: A dataset requiring formal first-order logic reasoning, with premises and conclusions expressed in natural language and paired with formal logical annotations. Tests whether models can reason correctly in strictly formal contexts, not just informal ones.
• StrategyQA: Questions that require implicit multi-step reasoning strategies to answer, where the reasoning steps are not explicitly provided. The model must decompose the question, identify the sub-questions it needs to answer, and integrate the sub-answers correctly.

Each of these benchmarks targets a specific facet of reasoning capability. Using multiple benchmarks together provides a more complete picture of a model's strengths and weaknesses than any single benchmark can.

A recurring challenge in reasoning evaluation is that benchmarks get saturated. As models improve, they achieve ceiling-level performance on benchmarks that were originally quite difficult. Once a benchmark is saturated, it loses discriminative power: it cannot distinguish between models of different capability levels, and further improvement on that benchmark is not meaningful.

The saturation of MMLU and many initial reasoning benchmarks has driven the development of harder evaluation suites. MMLU-Pro, introduced in 2024, extends MMLU with harder questions and more answer choices to restore discriminative power. BIG-Bench Hard focuses on a subset of BIG-Bench tasks where models still perform poorly. The progression illustrates that the field faces a moving target: a good evaluation benchmark must be calibrated to the current frontier of model capability, but frontier capability is advancing rapidly.

This cycle, of creating benchmarks, watching them saturate, and creating harder ones, is itself informative: it suggests that the underlying capabilities are improving, but it also makes it difficult to track progress on a stable axis. Different research groups may report results on different benchmarks, making direct comparisons difficult and progress measurement noisy.

The saturation problem is also philosophically interesting. When frontier models approach ceiling on grade-school math benchmarks, this does not mean they have mastered grade-school math in the way a human understands mastery. A human who has mastered addition can apply it reliably in any context, in any format, under any surface variation. A model near ceiling on a specific benchmark may still fail at isomorphic problems presented unusually. Saturation of benchmark performance is not the same as complete, robust mastery.

A specific and serious challenge for reasoning benchmarks is data contamination: the possibility that benchmark examples, or examples very similar to them, appear in the model's training data. If the model has seen the benchmark problems (or problems with the same answers) during training, then high benchmark scores reflect memorization, not reasoning.

Contamination is difficult to detect because training datasets are large, poorly documented, and often not publicly inspectable. Several studies have found evidence of contamination in widely used benchmarks, casting doubt on some published results. Golchin and Surdeanu (2023) developed methods to detect contamination by testing whether models exhibit unusual memorization behavior on benchmark examples relative to paraphrased versions of the same problems. Their findings suggested nontrivial contamination in several widely evaluated models.

This has driven the creation of dynamic benchmarks (where new problems are generated at evaluation time) and of held-out test sets where the release date of problems post-dates the training cutoff of models being evaluated. The LiveBench project, for example, curates reasoning problems from recent competitions and publications, making sure that the problems could not have been in training data. The FrontierMath benchmark uses problems from ongoing mathematical research, which are even less likely to have appeared in training corpora.

Contamination concerns also affect the interpretation of capability improvements over model generations. If a newer model achieves higher scores partly because of contamination and partly because of real capability improvement, comparing raw scores across generations overstates the real improvement. Researchers have developed statistical tests for contamination, but none are definitive, and the field must live with this source of uncertainty.

A deeper challenge in evaluating reasoning is distinguishing between correct outcomes and correct processes. A model might arrive at the correct answer through a flawed reasoning process (for example, by making two mistakes that cancel out), or might have a logically valid reasoning process that happens to end at a wrong answer because of a factual error.

For many downstream applications, correct outcomes are what matter. But for understanding model capabilities and limitations, process evaluation is essential. A model that reliably produces correct answers through unreliable processes will fail unpredictably when conditions change slightly. A model that produces correct answers through reliable processes will generalize more predictably. Understanding the process is therefore important for engineering reliable systems, not just for scientific understanding.

This has led to the development of datasets that include step-by-step annotated solutions, against which model-generated reasoning chains can be compared. Evaluating whether each step in a chain is logically valid is, however, itself a challenging problem, one that current automated evaluation methods handle imperfectly. Manual evaluation of reasoning chains is expensive and slow, which limits the scale at which process evaluation can be conducted.

Uesato et al. (2022) developed outcome reward models and process reward models as complementary evaluation approaches. Outcome reward models simply check whether the final answer is correct. Process reward models evaluate each step in the chain, giving richer feedback but requiring annotated step-by-step solutions for training. The process reward approach has been shown to improve performance on multi-step reasoning tasks when used for training or inference-time selection, validating the intuition that step-by-step correctness is a useful training signal.

A recent development is using LLMs themselves as evaluators of reasoning quality. A judge model is given a reasoning problem, the reference solution, and a candidate solution, and asked to rate the candidate's reasoning quality. This approach scales better than human evaluation and can be applied to open-ended tasks where there is no single correct answer.

The challenge is that LLM-based evaluation inherits the same failure modes as LLM reasoning. A judge model may be fooled by confident-sounding but incorrect reasoning chains, may have systematic biases toward longer or more elaborate explanations, and may not reliably catch subtle logical errors. Calibration of LLM judges against human expert judgments is an active area of research. Zheng et al. (2023) showed that GPT-4 as a judge has substantial agreement with human preferences on many tasks but exhibits systematic biases, including a tendency to prefer longer responses and to be anchored by the order in which responses are presented.

A practical approach to mitigating these biases is to use multiple judges, present candidates in random order, and check for consistency across multiple evaluation runs. Even with these precautions, LLM-based evaluation is better treated as a fast, scalable approximation to human evaluation than as a substitute for it on high-stakes evaluation tasks.

The following code section demonstrates several of the reasoning concepts from this chapter. We build a set of experiments that illustrate how reasoning chains can improve accuracy, how multi-step errors accumulate, and how sensitivity to phrasing can be measured. The experiments use simple arithmetic reasoning and logical puzzles that we can evaluate exactly, avoiding the need for neural network inference and making every result interpretable.

We start by setting up the environment.

To illustrate error accumulation in multi-step reasoning chains, we simulate a reasoning process where each step has a fixed accuracy probability, and model the probability of getting the full chain correct as a function of chain length. This is a simplified but precise model: it treats each step as independent, which is optimistic compared to real reasoning where errors often cascade.

We now implement a simple checker for categorical syllogisms of the standard form. This illustrates what deductive reasoning requires: checking that the conclusion follows necessarily from the premises, not just that it sounds plausible. The function parses premises and conclusions in a structured natural language form and verifies validity by checking the structural requirements for a valid categorical syllogism.

Syllogisms 1 and 2 are valid: the conclusion follows necessarily from the premises. Syllogism 3 is invalid: it applies the rule about mammals to salmon, which are not mammals. The middle term does not match. Syllogism 4 is also invalid: the conclusion generalizes in a direction not warranted by the premises. Even though "all mammals are warm-blooded" is true, the syllogism cannot validly derive it from the specific premise that whales are mammals. This last example illustrates the reversal failure mode directly: a model might accept this conclusion because it sounds correct, without attending to the logical structure that makes it an invalid inference from the given premises.

A key failure mode is that LLMs are sensitive to surface phrasing. We illustrate this by taking logically equivalent problems and showing that a naive keyword-matching approach, which is a rough proxy for what a model relying on spurious correlations might do, produces different answers when the phrasing changes. Real LLMs are more sophisticated than this classifier, but they exhibit qualitatively similar sensitivity.

This simulation demonstrates how a system relying on surface keywords, rather than understanding the underlying logical structure, can be consistent or inconsistent with phrasing changes. The naive classifier gets some problems right when the phrasing triggers the right heuristic, and gets them wrong or inconsistently when phrasing deviates from expected patterns. Real LLMs are more sophisticated, but empirical research shows they exhibit qualitatively similar sensitivity: the same arithmetic problem phrased differently can change a correct answer to an incorrect one, especially when the phrasing includes unusual syntactic constructions or irrelevant contextual information.

We now create an illustrative visualization that summarizes reasoning types and their difficulty characteristics for current LLMs. The accuracy values combine approximate results from heterogeneous published benchmarks, while the brittleness scores are qualitative assessments based on the research literature. They show directional patterns rather than results from one controlled evaluation, so small differences between categories should not be overinterpreted.

The accuracy chart reveals a clear pattern: causal reasoning is far harder than other types, showing the fundamental difficulty of learning causal structure from text that records correlations. Abductive reasoning is also hard, likely because identifying the best explanation requires integrating many contextual factors. Analogical reasoning, by contrast, is relatively easy for LLMs, which have been trained on enormous quantities of text containing explicit analogical comparisons.

The brittleness chart shows a different pattern. Multi-step arithmetic is the most brittle reasoning type: the same arithmetic problem can be solved or failed depending on how it is phrased. Causal reasoning is also brittle, perhaps because the surface forms that signal causal claims in text are varied and the underlying causal structure is hard to identify from surface cues alone. Analogical reasoning is again the most robust, consistent with the hypothesis that LLMs have strong pattern-matching over structural similarity.

The final experiment illustrates benchmark saturation: the phenomenon where models improve over time and a fixed benchmark loses discriminative power. We model benchmark accuracy as a logistic growth curve, which captures the characteristic S-shape of capability improvement: slow at first, rapid in the middle, and then leveling off near ceiling.

The saturation curve shows why the field constantly needs new, harder benchmarks. Once a benchmark's accuracy curve flattens near ceiling, additional model improvement is invisible on that benchmark. The hard benchmark continues to discriminate longer, but eventually it too will saturate, driving the creation of yet harder evaluations. This dynamic has played out repeatedly in the history of NLP benchmarks: the GLUE benchmark was replaced by SuperGLUE when models approached human-level performance, and SuperGLUE was in turn succeeded by BIG-Bench and domain-specific professional exams as models saturated it as well.

Beyond the specific failure modes cataloged above, there are deeper architectural and conceptual limitations to the reasoning capabilities of current LLMs. Understanding these limitations helps you know what the techniques covered in subsequent chapters compensate for.

The most fundamental limitation is the lack of a persistent working memory separate from the context window. Human reasoning can maintain complex mental models, perform recursive subproblems, and return to prior reasoning states. You can start a proof, set it aside to prove a lemma needed for the main result, complete the lemma, and then return to the main proof with the lemma available. LLMs do none of this explicitly: the only "state" available is the context window, which is finite and grows with each generated token. When context overflows, earlier reasoning steps are lost. Recursive reasoning, which requires temporarily suspending a problem to solve a subproblem and then returning, requires careful prompt engineering to simulate in an LLM, and even then it is constrained by context length and the model's ability to maintain the structure of the suspended problem across long intermediate computations.

A second deep limitation is the absence of explicit symbolic manipulation. Formal reasoning systems represent logical statements as structured data and apply rules mechanically. A theorem prover manipulates syntax according to precisely specified rules, and each step is checkable by a much simpler verification procedure than the generation procedure. LLMs represent everything as continuous vectors and generate outputs by sampling from distributions. This works surprisingly well in practice, but it means that formally equivalent statements may be processed differently depending on their surface form, and that subtle errors in symbolic manipulation (like a sign error in an algebraic step) can occur without any mechanism to catch them. The model does not "know" that it has made a sign error because it has no explicit symbolic representation of the algebra; it only knows that the text it generated looks like plausible algebra.

Third, LLMs have no principled notion of ground truth. When reasoning, humans can query the world (run an experiment, look something up, ask an expert) to check whether their conclusions hold. LLMs generate from prior context alone. They cannot verify claims against reality during inference. A model might assert that a mathematical constant has a particular value, that a historical event happened in a particular year, or that a scientific law takes a particular form, with equal confidence regardless of whether the assertion is correct. This limitation is being addressed by tool use and retrieval augmentation (covered in later chapters of this handbook), but the base model reasoning pipeline remains unchecked by default.

A fourth limitation, less often discussed, is the lack of explicit goal representation. When a human solves a multi-step problem, they maintain an explicit goal state (what they are trying to prove or compute), track progress toward that goal, and can detect when a line of reasoning has gone in an unproductive direction and backtrack. LLMs generate the next token based on the prior context, with no explicit representation of the goal beyond what is encoded in the prompt and the generated reasoning chain so far. This means a model can generate a plausible-looking chain that gradually drifts away from solving the actual problem, without any mechanism to detect that it has wandered off-track.

Despite these limitations, LLM reasoning is useful. Many practical reasoning problems that users bring to LLMs are within the capability envelope: they involve familiar structures, moderate chain lengths, and problem types well-represented in training data. The failures tend to cluster at the edges: novel structures, long chains, causal questions, and problems that require systematic verification. The techniques in the chapters that follow, chain-of-thought prompting, self-consistency, least-to-most prompting, and process-level verification, are engineering responses to exactly these limitations. Each technique can be understood as compensating for one or more of the architectural limitations described above.

Understanding where the limits are is what this chapter has tried to give you. With that understanding, you can design applications that stay within the envelope, construct prompts that bring the relevant context into scope, choose verification strategies proportionate to the risk, and interpret benchmark results with appropriate skepticism.

Reasoning is the process of drawing conclusions from premises, and it takes multiple forms: deductive (necessary conclusions from general rules), inductive (general rules from specific observations), abductive (best explanations for observations), analogical (transfer from familiar to novel domains), and causal (identifying cause-and-effect relationships). Each form has distinct epistemological properties, and LLMs engage with them with different degrees of reliability.

LLMs acquire reasoning-like behavior through training on text that contains extensive examples of human reasoning. This makes them surprisingly capable on familiar problem types while leaving systematic vulnerabilities: spurious correlations from training data, brittleness to surface phrasing, failure to generalize to novel structures, error accumulation in long chains, hallucinated reasoning steps, reversal of logical polarity, and overconfidence. These failure modes are not random but reflect the underlying mechanism of statistical pattern matching over text.

Evaluating reasoning in LLMs requires careful attention to benchmark contamination, the distinction between correct outcomes and correct processes, and the saturation of benchmarks as models improve. Standard benchmarks like MMLU and GSM8K, mathematical benchmarks like MATH, and logical benchmarks like FOLIO and ReClor each probe different facets of reasoning capability. No single benchmark provides a definitive picture, and the field is actively developing harder, less contamination-prone evaluation frameworks, including dynamic benchmarks and competition-level evaluations with post-training-cutoff release dates.

The architectural properties of transformers, especially attention over the full context and the ability to generate intermediate steps as working memory, provide a foundation for reasoning capability. Feed-forward layers contribute factual associations; residual connections allow iterative refinement; and token-by-token generation externalizes reasoning into the context window. But fundamental limitations remain: no persistent working memory beyond the context window, no symbolic manipulation guarantee, no ability to verify claims against reality without external tools, and no explicit goal representation.

Understanding these foundations prepares you for the techniques explored in the chapters that follow. Chain-of-thought prompting exploits external reasoning traces to improve multi-step performance by extending effective working memory. Reasoning strategies like self-consistency and least-to-most prompting address specific failure modes. Verification methods address the error accumulation problem. Mathematical reasoning evaluation unpacks the hardest end of the reasoning capability spectrum. All of these techniques make more sense once you understand the architecture and failure patterns they are designed to address.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about reasoning foundations in language AI.