Reasoning Foundations › Reasoning in Large Language Models

LLMs are trained to predict the next token given a sequence of prior tokens. Nothing in this objective explicitly trains the model to reason. And yet these systems display behavior that looks like reasoning across a wide range of tasks. How should we understand this?

The question is not academic. If LLMs reason through systematic inference, their outputs in novel situations should be as reliable as their outputs in familiar ones. If they pattern-match over reasoning-shaped text, their reliability degrades precisely where you most need it: in novel situations, at the edge of the training distribution, in domains underrepresented in the training data. The mechanism matters for trust.

The most parsimonious explanation is that LLMs have learned reasoning as a pattern. Training corpora contain enormous quantities of text in which people reason explicitly: mathematical proofs, logical arguments, scientific explanations, philosophical treatises, tutoring dialogues, and informal arguments in online discussions. When a model is trained on this text, it learns the statistical patterns of how reasoning steps follow from one another.

From this perspective, LLM reasoning is a form of sophisticated pattern matching over reasoning-shaped text. The model has seen many chains of the form "if A then B; we have A; therefore B" and has learned to continue such chains in a way that looks correct. This is not nothing: the resulting behavior can be useful and remarkably reliable on familiar problem types.

But this view also explains a key vulnerability. If the model is reasoning by pattern, then novel problem structures that do not match anything in training distribution will trip it up. The model may generate a confident-looking but incorrect continuation because the surface structure resembles something it has seen, even though the underlying logic is different. A model trained on many examples of standard arithmetic word problems may fail at an isomorphic problem where the surface format differs, not because the underlying reasoning is harder, but because the pattern is less familiar.

This perspective also reframes what "emergent reasoning" means. When a model suddenly performs much better on a reasoning task at a certain scale, it may be because it has enough capacity to match reasoning patterns at a finer level of granularity, not because it has crossed a qualitative threshold into systematic inference. The debate between these interpretations is ongoing and not yet resolved.

As LLMs scaled in size and training compute, certain reasoning capabilities appeared that were not present, or not reliably present, in smaller models. This phenomenon, discussed in the Emergent Capabilities chapter, creates an interesting situation: researchers observed that models below a certain scale would essentially fail at multi-step reasoning tasks, while models above that scale would perform significantly better.

The most studied example is multi-step arithmetic. Small models fail badly at problems requiring more than one or two arithmetic operations. Larger models solve them with much higher reliability. Whether this reflects a phase transition or a smooth improvement that appears sharp only because evaluation metrics are coarse-grained is still debated. Wei et al. (2022) documented many such emergent capabilities and proposed that they arise from the combinatorial nature of reasoning: many sub-skills are each individually weak in small models but combine to produce functional reasoning only when all are present at sufficient strength.

The emergence of chain-of-thought reasoning is particularly interesting. Smaller models do not benefit from being prompted to show intermediate steps: they generate intermediate steps that are wrong, which then leads to wrong final answers. Larger models benefit substantially from the same prompting strategy. This suggests that the model must first acquire reliable intermediate reasoning before the scaffolding of showing work can help.

Scaling also affects the breadth of the failure patterns. Smaller models fail at a wide range of reasoning tasks and in consistent ways that reflect limited capacity. Larger models fail more sporadically and in ways that are harder to predict from surface features of the problem. This makes larger models harder to characterize but also harder to probe systematically, which is part of why evaluation methodology has become increasingly sophisticated alongside the models themselves.

The transformer architecture supports reasoning through several mechanisms that are worth understanding in detail, because they directly determine what kinds of reasoning the architecture can and cannot do well.

The attention mechanism allows the model to reference any prior token when generating the next one. In a reasoning chain, earlier steps provide the premises and intermediate conclusions that later steps depend on. Attention lets the model retrieve those earlier steps precisely, regardless of how far back in the sequence they appear. This is qualitatively different from recurrent architectures, which had to compress the entire prior context into a fixed-size state vector. Attention gives transformers essentially direct access to prior reasoning steps, which is why transformer-based models outperform recurrent models so dramatically on tasks that require integrating information across long contexts.

To understand what "direct access" means here in practice. The attention mechanism computes, for each query position, a weighted sum of all value vectors in the context, where the weights are determined by the similarity between the query and each key. This means the model can attend strongly to a specific earlier token if its key matches the current query, effectively reading from that position. In reasoning tasks, this mechanism allows a step that references an earlier intermediate conclusion to reliably retrieve that conclusion's content, rather than relying on it having been compressed into the hidden state.

Residual connections and layer depth allow for iterative refinement. Information can pass through many layers of processing, with each layer potentially contributing additional abstraction or inference. While it is an oversimplification to say each layer performs a distinct reasoning step, the depth of transformer networks does allow more complex transformations than shallow models. Empirical studies of transformer internals have shown that different layers tend to specialize in different types of computation, with earlier layers handling local syntactic patterns and later layers handling more abstract semantic relationships.

The token-by-token generation process means that reasoning is externalized into the model's own output. When a model generates "Let's think step by step" and then produces a chain of intermediate conclusions, it is using its own output as working memory. Each token the model generates becomes part of the context for generating the next token. This means the model can, in principle, perform multi-step reasoning by generating intermediate steps that it then attends to when computing the final answer.

This last point underlies chain-of-thought prompting. It is the basis for chain-of-thought prompting (covered in the next chapter) and explains why prompting the model to show its work often improves accuracy: the intermediate steps are not just for human readability, they extend what the model can compute by extending the effective context. The intermediate tokens function as a kind of working memory, allowing the model to store and retrieve partial results in a way that the fixed parameter space alone cannot support.

The feed-forward layers within each transformer block also contribute to reasoning by acting as key-value stores that retrieve factual associations and arithmetic operations. Research by Geva et al. (2021) showed that feed-forward layers can be understood as storing factual associations that the model queries during forward passes. This helps explain why factual retrieval often works well, while it also suggests a limitation: the stored associations are fixed at training time and cannot be updated dynamically during reasoning.

Reasoning problems often require maintaining and manipulating multiple pieces of information simultaneously. Context length directly constrains what information is available when any given token is generated. A model with a short context window cannot hold long reasoning chains or reference problems with many premises.

Consider a legal reasoning problem where you must integrate ten clauses of a contract to determine whether a particular action is permitted. If the contract exceeds the context window, the model cannot see all relevant clauses simultaneously. It must either reason from partial information, which introduces errors, or rely on retrieval mechanisms to fetch relevant clauses, which introduces its own failure modes. The same limitation applies to mathematical proofs with many lemmas, to strategic games with long histories, and to any domain where understanding requires integrating many pieces of information.

The expansion of context windows in modern LLMs, from a few hundred tokens to tens or hundreds of thousands, has significantly expanded the complexity of reasoning tasks these models can tackle. The GPT-3 model in 2020 worked with 2,048 tokens; by 2024, models like Claude and Gemini were processing contexts of hundreds of thousands of tokens. This quantitative increase enables new reasoning tasks become possible when the model can hold a full textbook chapter, a complete codebase, or an entire conversation history in context simultaneously.

But longer context is not automatically better: there is evidence that LLM attention degrades for information in the middle of very long contexts. Liu et al. (2023) documented the "lost in the middle" phenomenon: when the same fact is placed at the beginning, middle, or end of a long context, models perform significantly better when it is at the beginning or end, with performance dropping for middle positions. This suggests that even with long context windows, models do not attend uniformly to all positions, and information in the middle of a long context is at risk of being underweighted. Very long reasoning chains also accumulate errors. We will examine these failure modes in detail later in this chapter.