Process Reward Models › The Reasoning Problem in Language Models

Before diving into the mechanics of process reward models, it is worth spending time on the underlying problem they solve. Language models are trained to minimize prediction loss over large corpora of text. This objective works extraordinarily well for tasks that are essentially about pattern completion: language generation, translation, summarization, question answering from context. But it has a fundamental misalignment with tasks that require reliable, multi-step deductive reasoning.

Consider what happens when a large language model is asked to solve a competition-level algebra problem. During pretraining, the model has seen many algebraic manipulations. It has learned which symbols tend to appear together, which operations follow which others, and what a typical solution looks like. When it generates a response, it draws on these distributional patterns. The result often looks like a valid proof, uses correct notation, and arrives at a numerical answer. But the model has no internal representation of "logical necessity." It is generating tokens that have high probability given the context, and correct algebraic manipulations have high probability because they appeared frequently in the training data.

This distinction between distributional plausibility and logical validity becomes critical as problems become harder. For routine algebra, distributional plausibility and logical validity largely coincide, because the training distribution contains many examples of the same problem type. For novel or difficult problems, the correlation breaks down. The model can generate reasoning that looks fluent and structured but contains subtle errors: an incorrect application of a theorem, an invalid step in a derivation, a false assumption carried through multiple steps.

The research community recognized this mismatch early in the scaling era. Papers on chain-of-thought prompting showed that asking models to generate intermediate reasoning steps, rather than jumping directly to answers, substantially improved performance on complex tasks. But chain-of-thought prompting does not guarantee that those steps are correct. It simply invites the model to generate token sequences that resemble correct reasoning, which is much easier to fake than to achieve.

The question becomes: how do you train a model to reason correctly, not just to generate plausible-looking reasoning? This requires a training signal that distinguishes correct steps from incorrect ones, rather than only distinguishing correct final answers from incorrect ones. Process reward models provide exactly this signal.

The conceptual lineage of process reward models runs through reinforcement learning from human feedback. RLHF, as you have seen in prior chapters, trains a reward model to predict human preference between responses and then uses reinforcement learning to fine-tune the policy toward higher-scoring outputs. This approach works well when the quality of a response can be assessed as a whole: which of these two summaries is better? Which response is more helpful?

But for reasoning tasks, whole-response assessment misses the critical distinction between good reasoning and lucky reasoning. A response that gets the right answer via a flawed proof deserves a different reward than one that gets the right answer via a correct proof. Outcome-based RLHF cannot make this distinction; it rewards both equally if the final answer is correct.

The natural generalization is to assess reasoning at the level of individual steps. This is what process supervision does. The term "process supervision" was formally introduced and analyzed in work by Lightman et al. at OpenAI in 2023, though the underlying idea had appeared in various forms in earlier work on reward shaping and potential-based reward functions in reinforcement learning.

The Lightman et al. paper made the comparison concrete: they trained both outcome-supervised and process-supervised reward models on the MATH benchmark (a collection of competition-level mathematics problems) and showed that process supervision substantially outperformed outcome supervision, especially at higher difficulty levels where reasoning chains are longer and more complex. This empirical validation catalyzed significant research attention toward PRMs.

We will explore how they differ from simpler reward formulations, how they are trained, where they work best, and where they fall short.