Mathematical Reasoning › Process Reward Models

A main insight for mathematical reasoning training is that the standard approach of rewarding only correct final answers provides weak learning signal on hard problems. If a model gets a competition math problem wrong 95 percent of the time, almost every training example produces no gradient update toward better reasoning. The sparse reward problem is severe precisely at the problems where improvement is most useful.

Process Reward Models (PRMs) address this by providing dense supervision at each reasoning step. Rather than binary reward at the end, the model receives a score for each step in its solution. This makes the training signal much richer: even a solution that arrives at the wrong final answer can receive credit for steps that are mathematically valid. Conversely, a solution that happens to arrive at the correct answer through flawed reasoning receives lower credit at the step level.

The Let's Verify Step by Step paper (Lightman et al., 2023) demonstrated that PRMs substantially outperform outcome-reward models (ORMs) on GSM8K and MATH. The key experimental finding was that best-of-N selection guided by a PRM achieves higher accuracy than best-of-N guided by an ORM, especially on harder problems where final-answer reward is too sparse. At N=64 candidates, the PRM achieved roughly 78 percent on MATH level 4 and 5 problems, while the ORM achieved only about 62 percent on the same set.

Training a PRM requires annotations at the step level, which is more expensive than outcome annotations. Human annotators must read each step of a mathematical solution and judge whether it is correct, incorrect, or ambiguous. A step might be mathematically correct but pedagogically wrong (using a technique that complicates the solution), or it might contain a subtle error that does not yet affect the final answer. Annotators must distinguish these cases.

The Lightman et al. dataset included 800,000 step-level labels collected through Amazon Mechanical Turk, requiring annotators with verified mathematical competency. Each label indicates whether the step is correct, wrong, or neutral. The PRM is trained as a binary classifier at each step position, taking the partial solution up to that step as input and predicting step correctness.

Because human annotation at this scale is expensive, researchers have developed automated approaches using what is sometimes called "Monte Carlo estimation of step value." The intuition is that a step's quality can be estimated from the downstream success rate it enables.

Given a partial solution ending at step $k$, you sample a large number of completions from the current policy and measure how often they result in a correct final answer. The estimated value of step $k$ is:

$$V(s_k) = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\text{completion}_i \text{ is correct}]$$

where:

• $s_k$: the partial solution state after step $k$
• $N$: the number of sampled completions
• $\mathbf{1}[\cdot]$: the indicator function (1 if correct, 0 otherwise)

Steps that are consistently followed by correct completions receive high value estimates; steps that almost always lead to wrong answers receive low values, regardless of whether the step itself looks locally plausible. This captures the key insight: a step that seems mathematically sound but puts the solution on a wrong track should receive low credit.

While Monte Carlo estimation produces noisier labels than human annotation, it scales to much larger datasets and has been shown to produce PRMs that are competitive with human-annotated ones in practice. The quality of the resulting PRM scales roughly with the number of rollouts per step and the accuracy of the completion model. With a capable base model generating 64 completions per step, the estimated values are reliable enough to train a useful PRM without any human labeling.

The practical implication is that process reward models are no longer limited to well-resourced labs. Any practitioner with a dataset of mathematical problems and a reasonably capable base model can train a PRM by sampling large numbers of completions and using outcome correctness as a signal. The cost is compute, not annotation budget.

Once trained, a PRM can be applied in several ways during inference. The most straightforward is beam search guided by PRM scores: at each step, generate multiple candidate next-steps, score each with the PRM, and keep only the top-scoring candidates. This steers generation toward high-quality reasoning paths without retraining the generation model.

Best-of-N is simpler: generate $N$ complete solutions and select the one with the highest aggregate PRM score. This requires less online computation than beam search but does not steer generation during the process. The tradeoff depends on how well the base model already generates plausible solution paths: if most paths are plausible until the final steps, best-of-N works well. If paths diverge early, beam search with PRM guidance is more efficient.