Process Reward Models › Process Reward Models

Language models learn to generate text, but they do not inherently learn to reason well. When you ask a model to solve a multi-step math problem, write a proof, or debug a program, the model's objective during training is simply to predict the next token correctly. Nothing in that objective forces the model to check whether intermediate steps are logically valid. A model that produces a plausible-looking chain of reasoning can still arrive at a wrong answer through a sequence of subtle errors.

The challenge is producing correct final answers and correct reasoning. To train models that reason reliably, we need a way to evaluate whether the final answer is right and whether each step along the way was valid. This is the core motivation for process reward models.

A process reward model (PRM) is a model that assigns a score to individual steps in a reasoning chain, rather than evaluating only the final outcome. Instead of asking "did the model get the right answer?", a PRM asks "was this step logically sound?". This distinction matters because it enables training and search procedures that optimize the quality of reasoning at every stage, not just at the end.

Process reward models connect reinforcement learning from human feedback (RLHF) and structured reasoning. They extend the idea of a reward model, which you encountered in earlier chapters on RLHF, to the sequential, structured setting of multi-step problem solving. To understand why this extension is necessary, it helps to trace the history of the problem and appreciate exactly where simpler reward formulations break down.