Process Reward Models › PRM for Mathematical Reasoning

The strongest empirical results for process reward models have come from mathematical reasoning tasks. Mathematics provides several properties that make it an ideal testbed for developing and evaluating PRMs.

First, mathematical reasoning is inherently multi-step. Solving a competition problem requires applying a sequence of definitions, theorems, algebraic manipulations, and logical inferences. Each step can be independently judged for validity. This structure maps naturally onto the PRM evaluation paradigm.

Second, correctness is verifiable. Mathematical solutions either reach the correct numerical answer or they do not. This makes it possible to generate large amounts of training data by having a model produce candidate solutions and checking the final answer automatically. The MATH benchmark, AMC/AIME competitions, and GSM8K problems all provide ground-truth answers that enable automated verification.

Third, errors are localized. A mathematical error typically occurs at a specific step and produces a specific incorrect claim. This makes step-level labeling tractable: an annotator can identify the first step where the reasoning goes wrong, rather than needing to assess overall quality. In contrast, assessing the correctness of an argumentative essay at the step level would require much more subjective judgment.

Fourth, the error taxonomy is relatively well-defined. Mathematical errors fall into recognizable categories: arithmetic mistakes, incorrect theorem application, invalid algebraic manipulation, missing case analysis, and so on. Annotators with mathematical training can reliably classify steps into these categories, producing consistent labels.

These properties make mathematics the natural first domain for PRM development, and most foundational PRM research has used mathematical benchmarks. However, the principles generalize, and researchers have begun exploring PRMs for code generation, formal verification, and scientific reasoning.

One of the most direct applications of a PRM is in best-of-N search. Instead of training a new policy, you use the PRM to select the best solution from a set of $N$ candidate solutions generated by a base model.

With outcome reward, best-of-N selects the solution that the ORM scores highest based on the complete solution. This is equivalent to having the ORM rank all $N$ solutions and return the top one. With process reward, you have two natural strategies:

Step-level reranking. Score each candidate solution by taking the minimum step score across all steps. A solution with one very bad step receives a low overall score, even if most steps are excellent. This is conservative but effective at eliminating solutions with clear errors.

$$\text{score}(s) = \min_{t=1}^{T} \text{PRM}(p, s_1, \ldots, s_t)$$

Step-level aggregation. Score each solution by taking the product or geometric mean of step scores. This aggregates step quality across the entire solution.

$$\text{score}(s) = \left(\prod_{t=1}^{T} \text{PRM}(p, s_1, \ldots, s_t)\right)^{1/T}$$

Both strategies outperform selecting by final-step score alone, because they account for the quality of the entire reasoning path, rather than only the conclusion. The minimum-step strategy has shown stronger empirical performance on mathematical benchmarks, likely because a single incorrect step in mathematics typically invalidates the entire solution regardless of how well the other steps are executed.

The relationship between best-of-N performance and the size of $N$ reveals important properties of both the PRM and the base model. Plotting accuracy against $N$ (on a log scale) produces a curve that rises steeply at first and then levels off. The height at which the curve levels off reflects the ceiling imposed by the base model's capability: no amount of selection can yield a correct solution if none of the $N$ candidates contains one. The steepness of the initial rise reflects the quality of the PRM as a selector: a better PRM selects the correct solution from fewer candidates.

A more powerful application is using the PRM to guide beam search during generation. Instead of generating $N$ complete solutions and then reranking them, you use the PRM to prune bad steps during generation. This changes the character of the search from selection among complete solutions to active guidance of the generation process.

The procedure works as follows:

1. Begin with the problem $p$ as the initial context.
2. Generate $K$ candidate next steps from the current prefix using the policy model.
3. Score each candidate step using the PRM, conditioning on the problem and all preceding steps.
4. Keep the top $B$ candidates (beam width $B \leq K$) and discard the rest.
5. For each surviving beam, generate $K$ more candidate next steps and repeat.
6. Continue until all beams have reached a terminal step (end of solution).
7. Select the highest-scoring complete solution from among the surviving beams.

This is similar to how beam search works in sequence generation, but the scoring function is the PRM rather than the language model's own log probability. The PRM acts as an evaluator, the language model acts as a proposer, and together they search the space of reasoning chains more effectively than either could alone.

The advantage of beam search over best-of-N is computational efficiency and search quality. Beam search explores promising paths more deeply by catching bad steps early and not wasting computation on paths that have already gone wrong. If step 3 is clearly incorrect, beam search discards that path before generating steps 4 through 20, while best-of-N would generate all 20 steps and only then reject the solution. This early pruning makes beam search substantially more efficient for fixed compute budgets.

The tradeoff is that beam search can commit to a flawed path if the PRM makes an early incorrect judgment. If the PRM assigns a high score to a subtly incorrect step 3, beam search will build on that step and potentially waste many subsequent computation steps before the error manifests as a wrong answer. Best-of-N, by contrast, assesses complete solutions and is less susceptible to this early-commitment failure mode.

The optimal beam width $B$ depends on the reliability of the PRM. With a highly accurate PRM, small $B$ (even 2 or 3) may suffice. With a noisy PRM, larger $B$ provides a safety buffer but increases computational cost. In practice, $B$ is typically set to between 4 and 16 for mathematical reasoning tasks.

At the frontier of reasoning research, process reward models are being combined with Monte Carlo Tree Search (MCTS) to create even more powerful search procedures. MCTS was developed in the context of game-playing AI (most famously in AlphaGo and AlphaZero) and provides a principled framework for searching tree-structured spaces with uncertain value estimates.

In the reasoning setting:

• Each node in the search tree represents a partial reasoning chain (a problem plus some steps).
• Expanding a node means generating candidate next steps from that prefix.
• The PRM evaluates the quality of each newly added step, giving an immediate local signal.
• Monte Carlo rollouts from a node estimate the probability of reaching the correct answer from that point, giving a longer-range value estimate.
• The MCTS algorithm balances exploration (trying new paths) with exploitation (extending high-value paths already discovered) using the UCB (Upper Confidence Bound) selection criterion.

The PRM is a value function in the MCTS framework, estimating the "value" of a partial solution (how likely it is to lead to a correct answer). This combination addresses a key weakness of both pure PRM scoring and pure Monte Carlo estimation: PRM scoring without rollouts can miss long-range dependencies, while pure Monte Carlo estimation is computationally expensive because it requires running many full completions. MCTS with PRM-guided rollouts finds a middle ground, using the PRM to quickly assess local step quality while using sparse Monte Carlo rollouts to calibrate longer-range value estimates.

This combination has shown impressive results in recent work. Models that reason via MCTS with PRM guidance substantially outperform both greedy decoding and standard beam search at comparable compute budgets, especially on the hardest mathematical problems where the search space is largest and step-level guidance is most valuable.