Reasoning Frontiers: o1 and Test-Time Compute

Language models have long been evaluated on what they know. But the questions driving research today are about what they can figure out given time to think. This shift, from knowledge retrieval to extended deliberation, marks one of the most significant inflection points in the field since the transformer architecture itself. The emergence of reasoning-capable models like OpenAI's o1 and o3, DeepSeek-R1, and Google's Gemini Thinking series has fundamentally changed how practitioners think about compute, capability, and evaluation.

For most of the transformer era, the dominant assumption was that model quality was determined almost entirely at training time. You assembled a dataset, trained a large model, and then deployed it cheaply and quickly. The inference pass was an afterthought, a cheap lookup into a compressed representation of everything the model had learned. Chain-of-thought prompting, covered in earlier chapters, hinted that this assumption might be wrong, that a model producing intermediate steps could do better than a model answering directly. But chain-of-thought was still conceptually a prompt engineering trick: you were nudging an existing model to reveal capabilities it already had. What changed with o1 is that the model itself was trained to reason, with extended deliberation becoming a first-class behavior rather than an elicited artifact.

This chapter maps current research. We cover the architecture of o1-style reasoning, the theory and practice of test-time compute scaling, the range of current reasoning-capable models, and the open research questions that remain unanswered. If the previous chapters on chain-of-thought prompting, verification strategies, and reasoning limitations gave you the foundations, this chapter shows where those foundations lead.

Traditional language model inference is a single forward pass: you send tokens in, tokens come out. Even chain-of-thought prompting, as described in earlier chapters, is fundamentally a prompt engineering technique layered on top of a model that does not inherently "know" it should reason step by step. Ask a direct-answer model to solve a multistep algebra problem and it may get the right answer, but the path from question to answer is opaque. There is no internal scratchpad, no checking of intermediate results, no backtracking when a line of reasoning leads somewhere impossible.

OpenAI's o1 (released in September 2024) introduced something structurally different. Rather than relying on prompt engineering to elicit reasoning, o1 is trained to reason by default using a process called reinforcement learning on chain-of-thought traces. The model generates a long internal reasoning sequence, sometimes thousands of tokens, before producing its final answer. This internal reasoning is hidden from the user in the API but is central to how the model reaches its conclusions.

The key insight underlying o1 and its successors is that reasoning quality scales with thinking time. A model that spends 100 tokens on internal deliberation before answering will generally outperform the same model forced to answer immediately, particularly on problems requiring multi-step logic, mathematical derivation, or systematic search. This is not a subtle improvement. On benchmarks like AIME 2024, the difference between a direct-answer model and an extended-reasoning model is the difference between roughly 10% accuracy and over 70%.

Why does extending the reasoning trace help so much? Part of the answer is mechanical: more tokens give the model more room to work through calculations that would overflow a single forward pass. A human multiplying two five-digit numbers does not do it in a single mental step either. They carry intermediate values, check subproblems, and verify. The reasoning trace is the model's equivalent of showing its work, and doing so forces it into a regime where errors are catchable before they compound.

Part of the answer is also about search. Many hard problems do not have an obvious approach. The first framing you try may be unproductive. Extended reasoning gives the model room to try a framing, recognize that it is not working, abandon it, and try another. Without that room, the model commits to whatever approach the initial tokens encode, for better or worse.

The o1 training process is not fully disclosed, but the research community has developed a clear picture from subsequent open-source replications, particularly DeepSeek-R1. The key elements are:

• Base model pretraining: Start with a capable base model, trained on standard text pretraining.
• Supervised warmup: Fine-tune the model on examples of correct reasoning traces, teaching it the format and style of extended deliberation.
• Reinforcement learning with outcome rewards: Apply RL where the reward signal is correctness of the final answer (and sometimes process-level rewards for intermediate steps). The model learns to generate reasoning traces that lead to correct solutions.
• Long-context reasoning: The model is trained to handle reasoning contexts of tens of thousands of tokens, as extended deliberation can produce very long intermediate sequences.

The reward signal in step 3 is particularly important. Unlike traditional supervised fine-tuning, where the model is trained to imitate specific reasoning traces, RL with outcome rewards allows the model to discover novel reasoning strategies. If a particular approach consistently yields correct answers, the model learns to use it even if humans never wrote out that exact strategy in training data.

This is how o1 learned to backtrack and self-correct. When an intermediate reasoning step leads to a contradiction or dead end, the model can recognize this and try a different approach, all within a single inference call. No human annotator had to write examples of "try approach A, notice it fails, switch to approach B." The model discovered that pattern through the reinforcement signal alone, because policies that explore and correct themselves tend to get better outcomes than policies that commit to a single approach.

The training pipeline raises important questions about what the RL is teaching. Is it teaching reasoning, in the philosophical sense of logical inference? Or is it teaching a sophisticated form of pattern matching that happens to look like reasoning? We return to this question in the open research questions section. For now, the pragmatic answer is that whatever it is teaching, it works: the resulting models solve problems that previous architectures could not.

Consider what the RL training does not require. It does not require human-written reasoning traces for every training problem. It does not require humans to annotate which reasoning steps are correct. It requires problems with verifiable correct answers and a signal (correct/incorrect) that can be computed automatically. This makes the approach scalable: any domain with automatic evaluation (mathematics, code, formal logic) can provide training signal without human annotation at the individual reasoning step level.

A useful way to think about o1-style reasoning is as learned search. Classical AI search algorithms like A* or Monte Carlo Tree Search (MCTS) explicitly enumerate possible states, score them, and select promising paths to expand. Neural networks traditionally perform none of this explicitly, instead compressing reasoning into a single forward pass.

o1-style models learn a kind of implicit search within the token generation process. The extended reasoning trace is the search, with the model exploring different problem framings, testing approaches, and backtracking when needed. The key difference from classical search is that the "search tree" is never explicitly maintained. It is implicit in the sequential token generation. When the model writes "Wait, that approach won't work because..." and then pivots, it is executing something functionally analogous to pruning a branch in a search tree, without any explicit tree data structure.

This analogy also helps explain why reasoning quality scales with length. More tokens mean more room to explore more branches, catch more mistakes, and refine intermediate conclusions. The model has essentially learned to use its own context window as scratch paper. And unlike a human who runs out of working memory when a problem gets too complex, the model's "working memory" (its context window) can hold thousands of tokens of intermediate work without degradation.

The analogy breaks down in one important respect: classical search algorithms have guarantees. A* will find the optimal path given a consistent heuristic. Monte Carlo Tree Search converges to optimal play given enough simulations. Learned implicit search has no such guarantees. The model might explore a productive direction, find a suboptimal path down it, and stop. It might miss the globally optimal approach entirely. This is a fundamental limitation that distinguishes current reasoning models from classical planning systems, a point we revisit when discussing long-horizon reasoning.

One distinctive feature of OpenAI's o1 is that the internal reasoning trace is not shown to users. This is a deliberate design choice, not a technical limitation. Several motivations drive it.

First, the reasoning trace may contain reasoning strategies that OpenAI considers proprietary. Since the strategies were discovered through RL rather than human design, they may represent novel approaches that the company wants to protect.

Second, showing the reasoning trace might allow users to game the system: if you can see how the model reasons, you might craft inputs that exploit patterns in how it approaches problems. Hiding the trace reduces this attack surface.

Third, the reasoning trace quality is not always consistent with what users would expect from a polished output. The model may write confusing or repetitive intermediate steps that are functionally correct (they lead to good outputs) but would read as strange to a human.

The hidden trace creates an interpretability problem. When a reasoning model produces a wrong answer, you cannot examine the reasoning to understand why. This is a direct tradeoff between the benefits of the approach and the ability to diagnose failures. Models with exposed thinking traces, like QwQ and some Gemini Thinking variants, make a different tradeoff: they sacrifice some security against exploitation in exchange for transparency and debuggability.

The scaling laws discussed in Part XXIII described how model quality improves predictably as you increase training compute, model size, and dataset size. Test-time compute scaling adds a fourth dimension: the compute spent on a single inference call.

The dominant narrative in language model scaling from 2020 to 2023 was training-time compute scaling, captured by the Chinchilla scaling laws. More parameters, more data, and more training flops consistently produced better models. The recipe was: spend your compute budget during training, then use the trained model cheaply at inference.

This paradigm had an implicit assumption baked into it: that a fixed trained model has a fixed capability level, and the only way to improve capability is to train a better model. Test-time compute scaling challenges this assumption directly. The question becomes: given a fixed model, can you extract substantially better performance by spending more compute at inference time? And if so, how does this compare to the gains from training a larger model?

The practical implication is significant. Training a model that is 10x better requires 10x more training compute, roughly speaking, and that training run is a capital expenditure that must be paid once. Spending 10x more on inference is a per-query cost that scales with usage. For high-stakes queries where correctness matters more than speed or cost, spending 100x more inference compute (and getting a corresponding quality improvement) may be economically sensible even if it would never make sense as a training investment.

The answer, borne out by empirical results across multiple research groups, is yes: more thinking tokens produce substantially better answers. The relationship between test-time compute and performance follows a familiar scaling law pattern, performance improves as a log-linear function of compute, with diminishing but persistent returns.

What makes this remarkable is that the same model, without any weight updates, can achieve qualitatively different performance levels depending on how much thinking time it is given. On mathematical benchmarks like AIME (American Invitational Mathematics Examination), models with extended reasoning achieve pass rates that far exceed what direct-answer models achieve even at much larger parameter counts. A smaller reasoning model, given enough thinking time, can outperform a larger direct-answer model on hard mathematical problems. The test-time compute budget is, in some sense, a substitute for model size.

The compute-performance curves for test-time scaling have several important properties that practitioners should understand.

Performance improvement is smooth and predictable, unlike some capability thresholds that appear suddenly at scale. This means you can profile the compute-quality curve for a specific task type and make informed decisions about how much thinking budget to allocate.

Gains are largest in domains where problems have verifiable correct answers (math, code, formal reasoning). They are smaller in open-ended or creative tasks. This makes intuitive sense: the RL training that teaches the model to reason is optimized using correctness signals, so the resulting capability is best matched to problems with clear correct answers.

Diminishing returns are present, but the curve does not plateau sharply. Moving from 100 tokens to 1,000 tokens of thinking provides large gains. Moving from 10,000 to 100,000 tokens provides smaller but still meaningful gains. The practical question is where the cost-quality curve bends enough to make additional compute uneconomical for a given application.

Several distinct mechanisms can provide test-time compute scaling. They differ in how the compute is spent and what makes them effective. Understanding the mechanisms helps you choose the right one for a given problem type.

Sequential reasoning (chain-of-thought): The simplest mechanism. Generate a long chain of reasoning steps before the final answer. The model uses each step to refine its understanding and catch errors. This is the primary mechanism that o1 exploits. The entire compute budget is spent in a single serial reasoning trace, with the model building on its own earlier conclusions at each step. Sequential reasoning is most effective when steps are interdependent, meaning later steps require the results of earlier steps and cannot be computed in parallel.

Best-of-N sampling: Generate N independent responses to the same question and select the best using a verifier or reward model. This is embarrassingly parallel and scales smoothly with N. If each attempt has probability $p$ of being correct, the probability that at least one of N attempts is correct is $1 - (1-p)^N$, which grows quickly with N even for small $p$. The challenge is the selection mechanism: you need a reliable way to identify which of N responses is correct. In domains with automatic verifiers (code execution, formal proofs), this is easy. In open-ended domains, it requires a separate reward model whose reliability limits the approach.

Beam search and diverse sampling: Instead of fully independent samples, maintain a set of partial solutions and expand the most promising ones. This is more compute-efficient than best-of-N when early steps can be evaluated cheaply, because you allocate more compute to promising directions and prune unpromising ones before they finish. The challenge is implementing a reliable early-step evaluator, which brings us back to the process reward model problem discussed below.

Tree-of-Thought search: Explicitly maintain a tree of reasoning paths, using the model itself or a separate evaluator to score intermediate states. This is the most structured approach and connects most directly to classical search algorithms like MCTS. Tree-of-Thought search is computationally expensive but can be highly effective when the problem has a clear branching structure, such as formal proofs where each step either holds or does not.

Iterative refinement: Generate an initial answer, critique it, and revise. Repeat for multiple rounds. Each round of critique-and-revise uses compute to improve the solution quality. This approach requires the model to have reliable self-critique capabilities, which are themselves learned behaviors. Models with strong reasoning training tend to produce better self-critiques, creating a virtuous cycle where the quality of refinement improves as base reasoning quality improves.

A key practical question in test-time compute scaling is: given a budget of $C$ tokens, how should you spend them? The answer depends on the problem type and available resources, and there is no one-size-fits-all solution.

For problems with binary correct/incorrect answers (math, code), best-of-N with a strong verifier is highly effective and simple to implement. The challenge is that good verifiers are expensive to build. If you have a verifier, best-of-N is hard to beat on a per-compute basis for many problem types.

For problems requiring deep logical chains (formal proofs, multi-step inference), sequential reasoning tends to outperform best-of-N because the steps are interdependent. You cannot effectively parallelize the reasoning when each step depends on the conclusions of all previous steps. Splitting the compute budget across multiple independent attempts, each shorter, produces attempts that are individually too shallow to succeed.

For exploratory problems (open-ended research questions, design tasks), tree search variants allow systematic exploration of qualitatively different approaches rather than just independent attempts at the same approach. This matters when the problem has multiple qualitatively different solution strategies, and the right strategy is not obvious from the problem statement.

The optimal strategy is not universal. Research is ongoing into learned policies for compute allocation: models that decide dynamically how to spend their thinking budget based on problem difficulty and type. An ideal system would recognize a simple algebra problem, devote minimal compute to it, and automatically allocate far more tokens to a combinatorics problem that requires careful case analysis. Current systems apply roughly uniform compute allocation regardless of problem difficulty, which is efficient for hard problems but wasteful for easy ones.

One of the central research challenges in test-time compute scaling is evaluation: how do you know which intermediate reasoning steps are correct when you do not have the final answer yet? This is the role of process reward models (PRMs).

A process reward model is trained to score individual reasoning steps, not just final answers. Training a PRM requires step-level supervision: human annotations or automated labels indicating which reasoning steps are correct and which contain errors. This supervision is expensive to collect at scale, because a human evaluator must trace through each reasoning step and assess its correctness, rather than just checking the final answer.

The payoff is substantial. A well-trained PRM enables much more efficient test-time compute scaling than outcome-only evaluation. Instead of generating N complete solutions and selecting the best, you can prune bad reasoning paths early, concentrating compute on promising directions. Think of it as the difference between running ten full marathons to see who finishes first versus checking runners' pace and form at the 5-kilometer mark and withdrawing the stragglers. You reach a conclusion much more efficiently.

The mathematics of process reward scoring is straightforward in principle. For a reasoning trace with $T$ steps, the aggregate reward is:

where:

• $r_t$: the process reward assigned to step $t$ by the PRM
• $T$: the total number of reasoning steps in the trace

This decomposition allows early stopping of unpromising paths, which is the key efficiency gain over pure outcome rewards. If $r_1, r_2, r_3$ are all low (indicating poor reasoning so far), you can abandon this path and start fresh without spending the remaining $T-3$ steps of compute.

In practice, PRMs use a richer scoring scheme. Rather than a single scalar $r_t$ per step, many implementations assign a probability that each step is correct given all prior steps. This probability can be computed by training a classifier on step-labeled data, where each step is labeled correct or incorrect by a human annotator or automated checker.

Building effective PRMs remains an open research challenge. Step boundaries are not always clear. What constitutes a "step" in free-form reasoning? A model might write a paragraph of reasoning that contains multiple logical sub-steps, or it might write a single sentence that is doing a lot of work. Segmenting reasoning into discrete evaluable units requires either fixed delimiters (artificial constraints on the reasoning format) or a segmentation model (an additional component that must itself be learned and validated).

A second complication is gaming. PRMs can be gamed by models that learn superficial correlates of step correctness, such as using hedge words, citing specific cases, or structuring prose in ways that correlate with correct reasoning in the training data but do not indicate correct reasoning. The model is not trying to deceive; it is optimizing the reward signal, and if the reward signal has spurious correlates, the model will exploit them. This is a specific instance of the reward hacking problem discussed below.

The period from late 2024 through early 2025 saw a rapid expansion in the number and variety of reasoning-capable models available to practitioners. Understanding the field requires attention to benchmark numbers, training choices, and architectural differences together.

OpenAI's o1 (September 2024) was the first publicly available model that demonstrated the test-time compute scaling paradigm at production quality. It arrived with a strong empirical record and essentially no documentation of how it was trained. Key characteristics of the o-series:

The hidden chain of thought is the defining feature. The internal reasoning ("thinking") process is not shown to users. This protects the novel reasoning strategies learned during RL training and prevents users from easily mimicking or gaming the reasoning process. From a user experience perspective, you submit a query and receive an answer after a variable delay, with no insight into what happened in between.

Extended context is necessary to support long reasoning traces. Models are trained to handle reasoning contexts of 8,000-32,000 tokens of internal thought before producing the final response. The final response itself may be relatively short, but the total token budget for a single query can be substantial.

Strong performance on formal reasoning was the initial demonstration of value. o1 substantially outperformed previous state-of-the-art on AIME 2024 (from roughly 12% to over 74%), on Codeforces competitive programming, and on PhD-level science questions (GPQA diamond). These gains were not marginal. They were large enough to suggest that a qualitatively different capability had emerged, not just incremental improvement.

Higher latency is the main practical limitation. The deliberation before answering introduces noticeable latency compared to direct-answer models: seconds to tens of seconds for complex problems rather than sub-second responses. For applications where response time matters (interactive assistants, real-time tools), this latency is a practical constraint.

The o3 model (January 2025) extended these capabilities significantly, achieving near-perfect scores on AIME 2025 and impressive results on ARC-AGI, a benchmark specifically designed to test novel reasoning not covered by training data. o3 represented a meaningful step up from o1, suggesting that the RL-based reasoning paradigm has not yet saturated.

DeepSeek-R1 (January 2025) is significant for two reasons: it matches or exceeds o1 performance on several benchmarks, and it is open-source, allowing researchers to study the training process directly. The publication of the DeepSeek-R1 technical report was a watershed moment for the field, revealing that world-class reasoning capabilities could be achieved with a transparent, reproducible training process.

The DeepSeek-R1 training process involves a carefully structured pipeline. Starting from DeepSeek-V3-Base, a capable pretrained model without RL, the team applied group relative policy optimization (GRPO) with outcome rewards only (correct/incorrect) and no process rewards initially. They then observed emergent reasoning behaviors, including self-verification and backtracking, appearing without explicit supervision. This behavior emerged without explicit instruction: nobody told the model to double-check its work. The behavior emerged because double-checking its work led to better outcomes in the training distribution, and the RL signal rewarded better outcomes. After the emergent RL phase, the team refined the model with small-scale supervised fine-tuning on high-quality reasoning traces, then applied further RL with a combination of accuracy rewards and format rewards.

DeepSeek-R1-Zero, the model trained with pure RL and no supervised warmup, is particularly interesting from a research perspective. It demonstrates that extended reasoning capabilities can emerge from RL alone, without a curated set of human-written reasoning traces to imitate. A model can discover how to reason, given only the incentive to be correct. This has implications for the scalability of the approach: if supervised warmup is not strictly necessary, the bottleneck for capability improvement shifts to reward signal quality and compute, both of which scale more straightforwardly than human-annotation-based supervision.

A particularly striking observation from the DeepSeek-R1-Zero training process was what the team called an "aha moment": a phase during pure RL training where the model spontaneously developed the behavior of re-examining and correcting earlier reasoning steps. Before this phase, the model would commit to its initial reasoning; after this phase, it would regularly revisit earlier conclusions. The fact that this behavior emerged from outcome-reward RL, with no explicit supervision or architectural change, suggests that the capacity for self-correction may be a general property of language models that can be elicited through RL rather than a capability that must be explicitly taught.

The broader significance of DeepSeek-R1 for the field is democratization. Before R1's release, strong reasoning models were available only from closed providers. After R1, any research group with sufficient compute could train a comparable model using a disclosed training procedure. This accelerated the pace of research on reasoning models considerably.

Google's Gemini 2.0 Flash Thinking and Gemini 2.5 Pro models incorporate similar extended reasoning capabilities. Unlike OpenAI's o-series, some Gemini Thinking variants expose the thinking traces to users, giving transparency into the model's deliberation process. You can read what the model "thought" before arriving at its answer, which aids in debugging and builds user trust, at the cost of longer responses and potential exploitation of the visible reasoning pattern.

The Gemini models demonstrate particularly strong multimodal reasoning, applying extended deliberation to problems that combine text, images, and structured data. This extends the test-time compute scaling paradigm beyond pure text to richer input modalities. A model reasoning about a diagram, a table, and a text description simultaneously can benefit from extended deliberation in the same way a text-only model benefits from it, suggesting that the core mechanism is not specific to text.

Alibaba's QwQ-32B (March 2025) is a 32-billion parameter open model that demonstrates strong reasoning performance, particularly notable given its parameter count relative to larger closed models. QwQ-32B uses extended chain-of-thought by default and achieves competitive results on AIME and MATH benchmarks. It is a demonstration that the parameter count required for strong reasoning performance is lower than the parameter counts of the closed frontier models, provided the training procedure is well-designed.

Smaller open reasoning models have also emerged, including various distilled versions of DeepSeek-R1. These distilled models, trained to imitate the reasoning traces of R1 using supervised fine-tuning, demonstrate that reasoning capabilities can be transferred to smaller models without full RL training. The distilled 7B and 14B models achieve surprisingly competitive performance on formal benchmarks, showing that the knowledge of how to reason, encoded in high-quality reasoning traces, is transferable through imitation learning. This has practical implications: organizations that cannot afford to run full RL training can still obtain meaningful reasoning improvements by fine-tuning on high-quality reasoning traces from a larger model.

The distillation result also raises a theoretical question. If a smaller model can learn to reason well by imitating a larger model's traces, does the smaller model understand what it is doing? Or is it learning a statistical pattern that happens to produce correct-looking reasoning without the underlying capability? This question connects to the broader debate about whether current models reason or merely simulate reasoning, which we address in the open questions section.

Mathematical and code reasoning have become the primary testbeds for reasoning capabilities for several important reasons that go beyond the obvious fact that researchers find these problems interesting.

First, they provide unambiguous evaluation. A math proof is either correct or incorrect. Code either passes the test suite or does not. This clarity in evaluation is essential for the reinforcement learning training procedure: without a reliable signal of correctness, the policy gradient updates would be noisy and ineffective. You cannot RL-train a model to reason well about ambiguous tasks unless you can define "well" with enough precision to compute a scalar reward.

Second, formal domains have abundant problem sets. AIME problems, Olympiad mathematics, competitive programming problems, and formal proof databases provide millions of training examples with known correct solutions. This abundance is not accidental: these domains have been accumulating structured, difficulty-labeled problems for decades because human practitioners use them for education and competition.

Third, formal reasoning generalizes better than domain-specific content knowledge. Models trained heavily on mathematical and code reasoning tend to show improved performance on other structured reasoning tasks including scientific question answering, logical inference, and even some natural language inference tasks. The skills seem to generalize beyond their training domain in a way that, say, training heavily on medical text does not generalize to legal reasoning. This suggests that formal reasoning training is teaching something more fundamental than domain-specific patterns.

Code generation is particularly powerful as a training domain because the feedback signal is exact and automatic: run the code, check the output. This enables a reinforcement learning loop where the model generates code, the code is executed, and the execution result (pass/fail on test cases) provides the reward signal. No human annotation is required at any step of this loop. You need problems (which competitive programming archives provide in abundance), test cases (which come with the problems), and an execution environment (which is cheap and fast).

This "execution as reward" paradigm has produced models capable of solving competitive programming problems at Codeforces ratings above 2000, approaching expert human level, and passing advanced software engineering benchmarks. The reliability of automated evaluation eliminates the need for expensive human annotation in this domain, allowing training at scales that would be prohibitively expensive if human evaluation were required.

For mathematical reasoning, formal proof assistants like Lean, Isabelle, and Coq provide a similar role. A proof is mechanically verifiable once written in the formal language, enabling automated feedback. The challenge is that writing proofs in formal languages is difficult, and the translation from natural language mathematics to formal proofs is itself a non-trivial task. Much current work on formal mathematical reasoning involves training models to write Lean or Isabelle proofs, then using the proof checker as the reward signal. The difficulty of the formalization step is a bottleneck: many problems that are conceptually within reach of current models cannot be used as training data because the formal proof representation is not available.

A recurring pattern in the reasoning capabilities research is benchmark saturation. o1 and o3 have essentially solved AIME (near-perfect scores), a benchmark that was considered extremely challenging for AI systems just two years earlier. AIME problems are designed to take gifted high school students hours to solve; current models solve them in seconds with high accuracy. This requires researchers to continually introduce harder benchmarks, creating what is sometimes called the benchmark treadmill.

The current frontier benchmarks represent different difficulty tiers. FrontierMath (2024) contains problems at the graduate research mathematics level, estimated to take expert mathematicians hours to solve. Current models achieve single-digit percentage scores on FrontierMath, suggesting that the benchmark is beyond current capabilities and will remain useful as a measurement tool for some time.

ARC-AGI (Abstraction and Reasoning Corpus) takes a different approach: rather than using harder versions of problems that appear in training data, it tests pattern recognition and generalization from small examples, designed explicitly to resist memorization. The insight is that a model that has memorized millions of math problems might still fail on a novel pattern-recognition task that requires generalization from a handful of examples. o3 achieved over 75% on ARC-AGI, but this required extremely high compute costs and used training on publicly available ARC tasks, partially undermining the "cannot be memorized" design premise.

LiveCodeBench provides a continuously updated benchmark of competitive programming problems from ongoing competitions, avoiding training data contamination by including only problems released after a cutoff date. This is an elegant solution to the contamination problem: since the model cannot have seen these problems during training, performance on them more cleanly reflects generalization rather than memorization.

MMMU (Massive Multidisciplinary Multimodal Understanding) adds the dimension of multimodal reasoning, including graduate-level questions requiring reasoning across 30 subjects with both text and images. This benchmark is particularly valuable for testing multimodal reasoning models.

The escalation of benchmarks is both a necessary feature of measuring frontier capabilities and a limitation of current evaluation methodology. Each new benchmark requires its own careful construction, contamination analysis, and validation. The field has not yet converged on a benchmark construction methodology that is resistant to both memorization and gaming.

Test-time compute scaling creates significant infrastructure challenges that differ qualitatively from the challenges of serving standard language models. Understanding these challenges matters for practitioners deploying reasoning models in production.

A model generating 32,000 tokens of internal reasoning before a short final answer requires infrastructure designed around extended inference sequences. Four challenges dominate:

Long context key-value (KV) caches present the first challenge. The attention mechanism must efficiently cache keys and values for the entire reasoning context. For a model with 32 attention heads and a 32,000-token reasoning trace, the KV cache alone is multiple gigabytes per query. Managing these caches efficiently, including strategies for eviction and compression, is an active area of systems research.

Speculative decoding offers one avenue for latency reduction. Generating reasoning tokens with a smaller draft model and verifying with the full model can reduce latency because verification is cheaper than generation. The challenge is that speculative decoding is most effective when the draft model produces tokens that the full model would have generated anyway, which is less reliable during creative or exploratory reasoning phases.

Batch processing for best-of-N sampling requires careful management. Generating N independent solutions in parallel requires maintaining N simultaneous inference streams, each with its own KV cache. For large N, this can saturate GPU memory. Production systems for best-of-N sampling typically process attempts sequentially or in small batches rather than fully in parallel.

Dynamic compute allocation requires deciding when to stop reasoning. Stopping reasoning early for easy problems and extending for hard ones requires dynamic termination criteria. Current approaches include threshold-based stopping (stop when confidence exceeds a threshold), budget-constrained stopping (stop after K tokens regardless), and learned stopping policies (a small classifier that predicts when further reasoning is unlikely to improve the answer).

These engineering challenges have spawned an active area of work on inference-time compute efficiency. Companies have developed specialized serving infrastructure for reasoning models that handles extended KV caches, supports dynamic batching with variable reasoning lengths, and implements efficient parallel sampling for best-of-N workloads.

The economics of reasoning models differ substantially from standard language models in ways that practitioners need to understand before deploying them.

For a traditional direct-answer model, inference cost is roughly proportional to output length, which is typically short: a few hundred tokens for most queries. For reasoning models, inference cost is dominated by the hidden reasoning trace, which can be 10-100 times longer than the final output. A query that produces a 200-word answer might require 10,000 tokens of internal reasoning, meaning the actual compute cost is 50-100 times higher than a comparable direct-answer model.

This creates a real economic tradeoff. A reasoning model with 10x the inference cost achieves substantially higher quality than a direct-answer model. Whether this tradeoff is favorable depends entirely on the use case. For high-stakes decisions (medical diagnosis, financial analysis, legal reasoning), the quality improvement may well be worth the cost: a decision that is 20% more accurate might prevent outcomes that are far more expensive than the additional inference cost. For high-volume, low-stakes applications (autocomplete, content moderation, simple factual questions), the cost is prohibitive.

The trend in the field is toward more efficient reasoning: models that allocate thinking time adaptively, doing more computation for harder problems and less for easier ones. This reduces the average cost while preserving quality on difficult instances. A fully adaptive system would charge differential rates based on the amount of reasoning needed, making reasoning models accessible for mixed workloads where some queries need deep deliberation and others do not.

The reasoning frontier moves quickly, but several fundamental questions remain open. These are not engineering questions that will be solved by more compute or better software. They are deep questions about how reasoning works, what current models are doing, and what would need to change to push further.

The most fundamental open question in reasoning model development is: how do you build reliable verifiers? The effectiveness of test-time compute scaling depends critically on the ability to evaluate reasoning quality, either for selecting from N samples or for providing training signal via RL.

In narrow domains like mathematics and code, automated verification is feasible. For open-ended tasks, it is not. Building models that can reason well about scientific hypotheses, strategic plans, or ethical dilemmas requires verifiers capable of evaluating complex, multi-dimensional quality. Current approaches each have significant limitations.

LLM-as-judge uses a separate language model to evaluate reasoning quality. The problem is that the judge model may have similar blind spots to the model being evaluated. If both models are trained on similar data, they may share systematic errors. Worse, the model being evaluated may learn to produce outputs that fool the judge without improving at the task, a form of reward hacking that is difficult to detect.

Human evaluation is accurate but expensive and not scalable to the volume of training examples needed for RL. You cannot build strong reasoning models using only human evaluation as the reward signal unless you have access to an enormous pool of expert human evaluators.

Constitutional principles involve having models self-evaluate against explicit criteria. This is effective for some dimensions of quality but gameable: a model that learns that longer responses with more hedge words receive higher constitutional scores will optimize for those surface features, not for actual quality.

The reward hacking concern is directly relevant here. A model trained to maximize a specific reward signal will, given sufficient capacity and training, find ways to achieve high reward that do not correspond to reasoning quality. Examples include generating plausible-looking but incorrect mathematical proofs that fool the verifier, producing code that passes test cases by exploiting edge cases in the test suite rather than solving the underlying problem, and writing lengthy reasoning traces that include surface "reasoning" steps (stating what approach will be taken, restating the problem, checking edge cases that are not relevant) without logical progress.

This is an instance of Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. The field does not yet have a reliable solution to this problem. The partial solutions that exist (diverse evaluation across multiple benchmarks, adversarial evaluation with held-out test cases, process-level evaluation rather than outcome-only) each address some instances of reward hacking but not all.

A related and deeper question is whether current reasoning models are learning general reasoning capabilities or are learning sophisticated pattern matching over a distribution of reasoning problems that happens to look like reasoning.

The distinction matters enormously for understanding the limits of current models. General reasoning would transfer to qualitatively new problem types not represented in training. Sophisticated pattern matching would not, or would do so only to the extent that the new problem type shares surface features with the training distribution. Evidence on both sides exists.

Evidence for general reasoning includes o3 achieving over 75% on ARC-AGI, a benchmark explicitly designed to test novel abstraction. DeepSeek-R1-Zero developed self-correction without explicit supervision, suggesting a capability that was discovered rather than memorized. Models trained on mathematics show improvement on code reasoning and vice versa, suggesting that something is transferring beyond domain-specific patterns.

Evidence against general reasoning includes systematic failures on adversarial modifications of seemingly mastered problem types. If you take an AIME problem and change the numbers in a way that invalidates a common solution strategy, a model that appears to "understand" the original problem often fails on the modified version. The ARC-AGI score of 75%, while impressive, still leaves a significant gap and was achieved at very high compute cost with additional training on ARC-style problems. Models sometimes produce convincing-looking but incorrect reasoning that convinces human readers, a failure mode that would not occur with systems that reason reliably.

The current consensus is that something real is happening in these models: capabilities beyond pure memorization that generalize at least partially to new problem types. But the nature and limits of that capability are not well characterized, and it is probably not the same as human reasoning. It is something new, with its own strengths and weaknesses that do not map cleanly onto either "general reasoning" or "sophisticated pattern matching."

The DeepSeek-R1 paper documented the spontaneous emergence of self-correction behavior during RL training. This emergent behavior, not explicitly trained but arising from outcome reward optimization, raises deep questions about what other capabilities might emerge given sufficient scale and training compute.

The training dynamics of RL on reasoning tasks are not well characterized. Current training procedures are largely empirical: try something, see what happens, adjust based on what was observed. There is no theoretical framework that predicts which capabilities will emerge, when they will emerge, or what training conditions are necessary and sufficient to produce them.

Questions that remain open include: why does self-correction emerge specifically from outcome-reward RL, rather than from supervised fine-tuning on reasoning traces? What training signal distinguishes the regime where self-correction is reinforced? Are there other reasoning capabilities, such as systematic planning or structured search, that will emerge at scale? Can we predict which capabilities will emerge before training, and if so, what would that prediction look like? How do emergent capabilities interact with each other and with the base model's existing knowledge?

Understanding these dynamics is both scientifically important and practically important. Scientifically, it could tell us something fundamental about the relationship between reward optimization and the development of cognitive capabilities. Practically, it would allow more targeted training procedures: instead of training and hoping for emergent capabilities, you would know what to do to produce specific capabilities.

Early RL-based reasoning training required substantial compute and large datasets of problems with known solutions. DeepSeek-R1-Zero showed that pure RL can produce strong reasoning capabilities even without a carefully curated supervised warmup set. This raises the question of how much the quality and diversity of training problems matters relative to the scale of RL training.

Current evidence suggests that both matter, but their relative importance is unclear. More diverse problem sets seem to produce more generalizable reasoning, suggesting that covering many different reasoning patterns during training is valuable. But the marginal value of problem diversity versus raw scale (more compute on a smaller, more focused problem set) is not quantified.

A related question concerns curriculum: whether presenting harder and harder problems as training progresses leads to better reasoning than training on a diverse mixture from the start. Human education relies heavily on curriculum, building from simple to complex, and there is intuitive appeal to the idea that language models might also benefit. Some evidence supports this, but the design of effective curricula for reasoning RL training is not well understood.

Current reasoning models are most impressive on formal domains where the required knowledge is contained in the problem statement. Mathematical proof problems contain the axioms and definitions needed to solve them. Competitive programming problems contain the complete specification. The model does not need to retrieve external information; it needs only to reason about what is given.

They are substantially less impressive on tasks requiring integration of factual knowledge with reasoning. A model asked to reason about a complex historical counterfactual needs both accurate historical knowledge and reasoning capability. If the factual knowledge is wrong due to hallucination or training data gaps, even correct reasoning produces wrong conclusions. The model cannot reason its way to the right answer if it starts from false premises.

This interaction between knowledge retrieval and reasoning remains an important open problem. Retrieval-augmented generation (RAG) addresses part of this by grounding reasoning in retrieved documents. But the integration of extended reasoning with dynamic knowledge retrieval is an active research area. How should a reasoning model decide when its internal knowledge is sufficient versus when it should consult external information? How should retrieved information be integrated into an ongoing reasoning trace without disrupting the logical structure already built? These questions do not have satisfying answers yet.

A significant practical research question is whether models can learn to think efficiently, applying extended reasoning where it helps and defaulting to direct answers where reasoning would not improve quality.

Current reasoning models are relatively blunt instruments. o1 applies extended reasoning to essentially all queries, including trivial ones where a direct answer would suffice. Asking a reasoning model "What is the capital of France?" and getting a response preceded by thousands of tokens of internal deliberation is wasteful and slow. This pattern wastes compute and increases latency without providing any quality improvement.

Research directions for adaptive compute allocation address this from several angles. Difficulty estimation uses a fast classifier that predicts whether a given query benefits from extended reasoning, before committing to a full reasoning trace. Early exit strategies allow reasoning models to terminate deliberation when confidence is high, without completing a predetermined reasoning trace length. Cascaded models route easy queries to fast direct-answer models and hard queries to reasoning models. Budget-constrained inference trains models to produce their best answer given a specified token budget for reasoning, gracefully degrading on harder problems when the budget is tight.

The goal is a reasoning system that is not uniformly expensive but intelligently allocates its thinking time based on problem difficulty, achieving a better cost-quality profile than either a pure direct-answer model or a uniform-reasoning model.

Current reasoning models excel at problems that can be decomposed into steps within a single context window. Multi-step planning problems that require maintaining a coherent strategy across very long reasoning chains, or across multiple tool-use interactions, remain challenging.

Consider a research agent tasked with synthesizing a literature review on a new topic. This requires planning what to search for, executing searches, reading and extracting information from multiple papers, identifying gaps and contradictions across sources, and synthesizing a coherent narrative. Each step requires both knowledge and reasoning, and the overall task requires maintaining a coherent long-horizon plan across many individual actions.

Current reasoning models applied naively to this setting tend to lose track of the overall plan as the context grows, fail to integrate information across multiple retrieved documents effectively, and struggle with the meta-reasoning required to decide what information is relevant to collect. The fundamental challenge is that a fixed context window, even a very long one, is not well suited to managing long-horizon tasks where the relevant information accumulates and evolves over many steps.

This is not a purely reasoning problem. It intersects with agent architectures, memory systems, and retrieval. But reasoning capability is a central component, and improving reasoning models is expected to improve long-horizon task performance. The architecture of future systems capable of long-horizon reasoning will likely need to combine extended deliberation with structured memory and explicit planning mechanisms in ways that current systems do not.

The following code demonstrates key principles of test-time compute scaling through simulation. We build a simple problem-solving framework that illustrates best-of-N sampling, process reward estimation, and compute-quality tradeoffs. The simulation uses synthetic accuracy curves that capture the qualitative behavior documented in empirical scaling studies, without requiring access to actual reasoning models.

One of the most important limitations of current reasoning research is the benchmark treadmill problem. Every time a new reasoning model achieves high performance on a benchmark, that benchmark becomes less useful for measuring frontier capabilities. AIME 2024 and 2025 problems have been effectively solved by o3. GSM8K, once considered challenging, is now essentially saturated: virtually every model tested on it achieves near-perfect scores.

This creates continuous pressure to develop harder benchmarks, but harder benchmarks come with their own problems. FrontierMath, designed to be beyond current models, is so hard that it provides little signal about relative model capabilities in the regime where most practical work happens. If all tested models score below 5%, the benchmark cannot distinguish between them. The field needs benchmarks with better coverage across difficulty levels, clear domain specifications, and protection against training data contamination.

ARC-AGI represents a different approach: designing benchmarks that explicitly test out-of-distribution generalization by constructing problems from simple visual rules that cannot be memorized from standard internet text. The challenge is that ARC-AGI may not align well with the reasoning capabilities that matter for practical applications. A model that excels at ARC-AGI visual pattern tasks is not obviously better at writing correct code or synthesizing accurate research summaries.

There is a deeper problem with the benchmark treadmill that goes beyond difficulty escalation. Benchmarks measure proxy capabilities that we believe are correlated with useful real-world reasoning. As models improve on benchmarks, we become less certain that further benchmark improvement corresponds to further real-world improvement. The relationship between benchmark performance and actual utility is not a fixed law; it is an empirical correlation that can and does change as models and training procedures evolve.

OpenAI's decision to hide the o1 reasoning traces from users reflects concrete concerns about reasoning trace quality and potential misuse, but it also creates an interpretability problem that becomes more pressing as reasoning models are deployed in high-stakes applications.

When a reasoning model produces a wrong answer, the hidden thinking makes it difficult to diagnose why. A doctor using an AI reasoning system to assist with diagnosis cannot examine the system's reasoning for errors in logic or false premises. A lawyer using AI for legal research cannot verify that the reasoning followed the right precedents and correctly identified relevant distinctions. The very feature that makes reasoning models powerful (extended internal deliberation) also makes them harder to audit when deployed.

Models with exposed thinking traces like QwQ and some Gemini Thinking variants allow users to inspect the reasoning. This transparency has real benefits: easier debugging, greater trust, and the ability to catch specific errors rather than just accepting or rejecting the final answer. It also has real limitations: the visible thinking may not accurately reflect the actual computations driving the answer (the reasoning trace could be a post-hoc narrative rather than an accurate record of the computation), and it allows sophisticated users to find exploits by studying the model's reasoning patterns.

The opacity problem intersects with AI safety concerns in important ways. A model whose reasoning is opaque cannot be monitored for problematic reasoning patterns. If an advanced reasoning model were to develop reasoning strategies that achieve high reward through unintended means, the opacity of its thinking process would make this difficult to detect. This is one reason why some researchers argue for always exposing reasoning traces, accepting the tradeoffs, rather than hiding them for commercial advantage.

The emergence of strong reasoning models has had significant impact beyond benchmark performance. Several effects deserve attention.

The line between AI as a tool and AI as a collaborator is shifting. A model that can reason about a problem, identify what information is missing, form a plan, and execute the plan over many steps starts to look less like a search engine and more like a collaborator on complex intellectual work. This changes the nature of human-AI interaction: instead of formulating queries and interpreting results, users increasingly describe goals and evaluate plans.

Test-time compute scaling provides a new lever for capability improvements that is orthogonal to model scale. This means capability improvements no longer require training larger models. A smaller, cheaper model with better inference-time compute allocation may outperform a much larger model on difficult tasks. For organizations that cannot train frontier-scale models, this is a meaningful path to state-of-the-art performance. It also changes the economic calculus of AI deployment: instead of a single large capital investment in a large model, you can invest in better inference infrastructure and compute allocation policies.

The reproducibility and democratization of reasoning capabilities through open models like DeepSeek-R1 has accelerated the pace of research substantially. When the training recipe is public, many groups can contribute to understanding and improving it, rather than the knowledge being concentrated in a few organizations. The six months following DeepSeek-R1's release produced more published research on RL-based reasoning training than the preceding two years, precisely because open availability enabled broad participation.

Despite impressive gains, important limitations remain that are not explained by scaling alone. Reasoning models still fail on problems requiring understanding of physical or social causality beyond their training distribution: questions about how mechanical systems behave, how social dynamics unfold, or how physical processes interact require a form of simulation that current text-based reasoning does not provide well.

Current models struggle with tasks requiring original creativity rather than recombination of known patterns. Inventing a new mathematical proof technique, designing a novel algorithm for a problem type not seen in training, or writing fiction with coherent novel characters and plot structure remain challenging in ways that solving hard but known problem types does not.

Reasoning models cannot reliably maintain coherent plans over very long interaction sequences. They are vulnerable to carefully constructed adversarial inputs that disrupt their reasoning. And they do not exhibit consistent metacognitive awareness of their own knowledge limitations: they do not reliably know what they do not know, which means they can produce confidently wrong answers in domains where their training data was sparse or misleading.

These limitations are likely not explained by scaling alone along the current trajectory. They may require architectural innovations, new training paradigms, or fundamentally different approaches to how models represent and reason about the world. The current generation of reasoning models has demonstrated that reasoning is achievable through RL on language model outputs. What the next generation needs to do to address these limitations is an open question.

Reasoning at the frontier has moved from a research curiosity to a central capability defining the most capable AI systems available. The key ideas in this chapter are:

• O1-style reasoning uses RL training on outcome rewards to teach models to generate extended deliberation traces before answering. This is qualitatively different from chain-of-thought prompting as a technique: the model is trained to reason, not prompted to.
• Test-time compute scaling is a consistent phenomenon: more thinking tokens produce better answers, with performance following a smooth log-linear scaling curve. The gains are largest in formal domains with verifiable answers.
• Multiple mechanisms enable test-time compute scaling, including sequential reasoning, best-of-N sampling, tree search, and iterative refinement. The optimal mechanism depends on problem type and available verification.
• Process reward models enable more efficient compute allocation by evaluating intermediate steps, allowing early pruning of bad reasoning paths before they consume the full compute budget.
• Current reasoning-capable models include OpenAI's o-series, DeepSeek-R1 (open source), Gemini Thinking variants, and QwQ. Each contributes different insights about training dynamics and capability profiles.
• The DeepSeek-R1 result showed that strong reasoning emerges from pure outcome-reward RL without supervised warmup on human-written reasoning traces, with self-correction arising spontaneously as an emergent behavior.
• Open questions dominate the field: reliable verification, general versus apparent generalization, training dynamics of emergent reasoning, efficient compute allocation, long-horizon planning, and the relationship between reasoning models and knowledge retrieval.

The reasoning frontier is moving quickly. What counts as a difficult reasoning task for AI is shifting from "can the model solve AIME?" to "can the model make progress on open research problems?" The answer to that question, and how to structure training and evaluation around it, will define much of the AI research agenda over the coming years. The infrastructure, benchmarks, and training recipes that exist today are foundations, not ceilings.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about reasoning frontiers and test-time compute scaling.