Reasoning Frontiers › The O1 Paradigm: Reasoning as Extended Computation

Traditional language model inference is a single forward pass: you send tokens in, tokens come out. Even chain-of-thought prompting, as described in earlier chapters, is fundamentally a prompt engineering technique layered on top of a model that does not inherently "know" it should reason step by step. Ask a direct-answer model to solve a multistep algebra problem and it may get the right answer, but the path from question to answer is opaque. There is no internal scratchpad, no checking of intermediate results, no backtracking when a line of reasoning leads somewhere impossible.

OpenAI's o1 (released in September 2024) introduced something structurally different. Rather than relying on prompt engineering to elicit reasoning, o1 is trained to reason by default using a process called reinforcement learning on chain-of-thought traces. The model generates a long internal reasoning sequence, sometimes thousands of tokens, before producing its final answer. This internal reasoning is hidden from the user in the API but is central to how the model reaches its conclusions.

The key insight underlying o1 and its successors is that reasoning quality scales with thinking time. A model that spends 100 tokens on internal deliberation before answering will generally outperform the same model forced to answer immediately, particularly on problems requiring multi-step logic, mathematical derivation, or systematic search. This is not a subtle improvement. On benchmarks like AIME 2024, the difference between a direct-answer model and an extended-reasoning model is the difference between roughly 10% accuracy and over 70%.

Why does extending the reasoning trace help so much? Part of the answer is mechanical: more tokens give the model more room to work through calculations that would overflow a single forward pass. A human multiplying two five-digit numbers does not do it in a single mental step either. They carry intermediate values, check subproblems, and verify. The reasoning trace is the model's equivalent of showing its work, and doing so forces it into a regime where errors are catchable before they compound.

Part of the answer is also about search. Many hard problems do not have an obvious approach. The first framing you try may be unproductive. Extended reasoning gives the model room to try a framing, recognize that it is not working, abandon it, and try another. Without that room, the model commits to whatever approach the initial tokens encode, for better or worse.

The o1 training process is not fully disclosed, but the research community has developed a clear picture from subsequent open-source replications, particularly DeepSeek-R1. The key elements are:

• Base model pretraining: Start with a capable base model, trained on standard text pretraining.
• Supervised warmup: Fine-tune the model on examples of correct reasoning traces, teaching it the format and style of extended deliberation.
• Reinforcement learning with outcome rewards: Apply RL where the reward signal is correctness of the final answer (and sometimes process-level rewards for intermediate steps). The model learns to generate reasoning traces that lead to correct solutions.
• Long-context reasoning: The model is trained to handle reasoning contexts of tens of thousands of tokens, as extended deliberation can produce very long intermediate sequences.

The reward signal in step 3 is particularly important. Unlike traditional supervised fine-tuning, where the model is trained to imitate specific reasoning traces, RL with outcome rewards allows the model to discover novel reasoning strategies. If a particular approach consistently yields correct answers, the model learns to use it even if humans never wrote out that exact strategy in training data.

This is how o1 learned to backtrack and self-correct. When an intermediate reasoning step leads to a contradiction or dead end, the model can recognize this and try a different approach, all within a single inference call. No human annotator had to write examples of "try approach A, notice it fails, switch to approach B." The model discovered that pattern through the reinforcement signal alone, because policies that explore and correct themselves tend to get better outcomes than policies that commit to a single approach.

The training pipeline raises important questions about what the RL is teaching. Is it teaching reasoning, in the philosophical sense of logical inference? Or is it teaching a sophisticated form of pattern matching that happens to look like reasoning? We return to this question in the open research questions section. For now, the pragmatic answer is that whatever it is teaching, it works: the resulting models solve problems that previous architectures could not.

Consider what the RL training does not require. It does not require human-written reasoning traces for every training problem. It does not require humans to annotate which reasoning steps are correct. It requires problems with verifiable correct answers and a signal (correct/incorrect) that can be computed automatically. This makes the approach scalable: any domain with automatic evaluation (mathematics, code, formal logic) can provide training signal without human annotation at the individual reasoning step level.

A useful way to think about o1-style reasoning is as learned search. Classical AI search algorithms like A* or Monte Carlo Tree Search (MCTS) explicitly enumerate possible states, score them, and select promising paths to expand. Neural networks traditionally perform none of this explicitly, instead compressing reasoning into a single forward pass.

o1-style models learn a kind of implicit search within the token generation process. The extended reasoning trace is the search, with the model exploring different problem framings, testing approaches, and backtracking when needed. The key difference from classical search is that the "search tree" is never explicitly maintained. It is implicit in the sequential token generation. When the model writes "Wait, that approach won't work because..." and then pivots, it is executing something functionally analogous to pruning a branch in a search tree, without any explicit tree data structure.

This analogy also helps explain why reasoning quality scales with length. More tokens mean more room to explore more branches, catch more mistakes, and refine intermediate conclusions. The model has essentially learned to use its own context window as scratch paper. And unlike a human who runs out of working memory when a problem gets too complex, the model's "working memory" (its context window) can hold thousands of tokens of intermediate work without degradation.

The analogy breaks down in one important respect: classical search algorithms have guarantees. A* will find the optimal path given a consistent heuristic. Monte Carlo Tree Search converges to optimal play given enough simulations. Learned implicit search has no such guarantees. The model might explore a productive direction, find a suboptimal path down it, and stop. It might miss the globally optimal approach entirely. This is a fundamental limitation that distinguishes current reasoning models from classical planning systems, a point we revisit when discussing long-horizon reasoning.

One distinctive feature of OpenAI's o1 is that the internal reasoning trace is not shown to users. This is a deliberate design choice, not a technical limitation. Several motivations drive it.

First, the reasoning trace may contain reasoning strategies that OpenAI considers proprietary. Since the strategies were discovered through RL rather than human design, they may represent novel approaches that the company wants to protect.

Second, showing the reasoning trace might allow users to game the system: if you can see how the model reasons, you might craft inputs that exploit patterns in how it approaches problems. Hiding the trace reduces this attack surface.

Third, the reasoning trace quality is not always consistent with what users would expect from a polished output. The model may write confusing or repetitive intermediate steps that are functionally correct (they lead to good outputs) but would read as strange to a human.

The hidden trace creates an interpretability problem. When a reasoning model produces a wrong answer, you cannot examine the reasoning to understand why. This is a direct tradeoff between the benefits of the approach and the ability to diagnose failures. Models with exposed thinking traces, like QwQ and some Gemini Thinking variants, make a different tradeoff: they sacrifice some security against exploitation in exchange for transparency and debuggability.