Reasoning Verification › Process Reward Models

A process reward model (PRM) is a trained model that scores each step of a reasoning chain. Unlike outcome reward models (ORMs), which only evaluate the final answer, PRMs provide dense supervision: a score for every step. This dense signal is more informative and allows the training or search process to understand where in a chain of thought things go right or wrong.

The term "process reward model" comes from the reinforcement learning from human feedback (RLHF) framework, where reward models are used to provide training signal for policy optimization. In standard RLHF for language model alignment, a reward model provides a scalar reward for a complete response. A PRM extends this to provide rewards at each step, enabling process-level supervision in both search and training contexts.

Given a problem $x$ and a reasoning chain $s_1, s_2, \ldots, s_T$ where each $s_i$ is an individual reasoning step, a PRM assigns a score to each step:

$$r_i = \text{PRM}(x, s_1, \ldots, s_i) \in [0, 1]$$

where:

• $x$: the input problem
• $s_1, \ldots, s_i$: all steps up to and including step $i$
• $r_i$: the score for step $i$, showing the probability that this step is correct given the context

The full chain score is typically the minimum step score, showing the principle that a chain of reasoning is only as strong as its weakest step:

$$R(x, s_1, \ldots, s_T) = \min_{i=1}^{T} r_i$$

Alternatively, some systems use the product of step scores, treating each step as an independent event:

$$R(x, s_1, \ldots, s_T) = \prod_{i=1}^{T} r_i$$

The minimum is more conservative and emphasizes the single worst step. The product can be used as an approximation to joint probability under an independence assumption. In practice, the minimum and product often rank chains similarly, but the minimum tends to be more robust to PRM miscalibration on individual steps because it does not compound small errors in probability estimates across many steps.

A third aggregation method, used in some production systems, is a weighted average that gives more weight to later steps, since errors in early steps tend to propagate and early errors are often identified by lower step scores downstream. The choice of aggregation is an empirical decision typically validated against a held-out correctness dataset.

Training a process reward model requires step-level labels: for each step in each training example, a label indicating whether that step is correct. This is substantially more expensive to collect than outcome labels, which only require knowing whether the final answer is correct.

The OpenAI PRM800K dataset, released in 2023 alongside the "Let's Verify Step by Step" paper, provides human labels for 800,000 steps in mathematical reasoning chains. Each step is labeled as positive (the step is correct and valid), negative (the step contains an error), or neutral (the step is a restatement or formatting change that is not a substantive reasoning step). This dataset enabled training PRMs specifically for grade-school and competition mathematics and became the benchmark for evaluating PRM quality.

Collecting this dataset required significant effort. Annotators were domain experts in mathematics who could verify each step independently. The annotation interface presented each step in context, with the problem statement and all prior steps visible, and asked annotators to judge whether the current step followed correctly from what came before. Annotators were paid for their time, making the total cost of 800K annotations substantial. This cost is one of the primary barriers to deploying PRMs in new domains.

Given step-level labels, a PRM is trained as a classifier. The architecture uses a language model backbone, typically the same architecture as the generator model to allow knowledge transfer. For each step position, a special marker token (such as [STEP] placed at the end of each step) is used to extract a representation, and a classification head predicts the correctness probability:

$$r_i = \sigma(w^\top h_i + b)$$

where:

• $h_i \in \mathbb{R}^d$: the hidden state at the step marker token for step $i$
• $w \in \mathbb{R}^d$: learned weight vector for the classification head
• $b \in \mathbb{R}$: learned bias term
• $\sigma$: the sigmoid function, mapping the logit to a probability in $[0, 1]$

The model is trained with binary cross-entropy loss summed over all step positions:

$$\mathcal{L} = -\sum_{i=1}^{T} \left[ y_i \log r_i + (1 - y_i) \log(1 - r_i) \right]$$

where:

• $y_i \in \{0, 1\}$: the human-provided label for step $i$ (1 for correct, 0 for incorrect)
• $r_i$: the predicted probability that step $i$ is correct
• $T$: the total number of steps in the reasoning chain

The cross-entropy loss penalizes the model for being confident and wrong. If a step is labeled correct ($y_i = 1$) but the model assigns low probability ($r_i$ close to 0), the term $-y_i \log r_i$ becomes large, driving a large gradient update. Conversely, if a step is labeled incorrect but the model confidently calls it correct, $-(1-y_i)\log(1-r_i)$ is large. The model is trained to be well-calibrated: high confidence only when the step is correct.

An important implementation detail is that the PRM is trained to score each step given all previous steps, not in isolation. The hidden state $h_i$ is computed by processing the full sequence $x, s_1, \ldots, s_i$ through the language model, so the representation at position $i$ has attended to all prior context. This means the PRM can detect errors that depend on earlier steps, including reference errors and unjustified assumptions that can only be identified relative to what was established earlier.

The key difference between outcome reward models and process reward models is the granularity of supervision. Both approaches use a language model backbone and a classification head. Both are trained on examples drawn from model outputs. The distinction lies entirely in what gets labeled.

An ORM sees the full problem and full solution, and predicts whether the final answer is correct. It learns which overall solution styles tend to produce correct answers. An ORM cannot localize errors, because it has no step-level training signal. When an ORM scores a solution highly, it is saying "this kind of solution tends to be correct," not "each step in this specific solution is valid."

A PRM sees the problem and each step in sequence, and predicts whether each step is correct. It learns to identify local reasoning failures. PRMs are harder to train (requiring more annotation effort) but more informative. A PRM that scores step 3 low is providing specific, actionable information: this step is where the reasoning went wrong.

This difference in granularity has important practical consequences for search. When selecting among N candidate solutions, an ORM provides one score per solution. A PRM provides T scores per solution (one per step), allowing much finer discrimination. Two solutions may both reach a correct final answer, but one arrives there via a clean, tight chain of reasoning while the other includes several uncertain steps that happened to cancel out. A PRM can distinguish these cases; an ORM cannot.

Empirically, PRMs substantially outperform ORMs at selecting correct solutions from large candidate sets. In the "Let's Verify Step by Step" experiments, a PRM selecting the best of 1860 generated solutions achieved 78.2% accuracy on MATH benchmark problems, compared to 72.4% for an ORM with the same selection budget. The gap is even larger at intermediate selection budgets (best-of-100, best-of-200), where the PRM's step-level discrimination provides the most benefit.

The primary cost of PRMs is annotation. Human annotators must read each reasoning step and judge whether it is correct given the problem context and prior steps. This requires domain expertise, particularly for mathematical reasoning, which means annotation is expensive and slow.

Several strategies exist for reducing annotation cost.

Monte Carlo estimation provides a way to infer step labels from outcome labels. The idea is to estimate the probability that a correct final answer can be reached from a given step by sampling many completions. If you have a partial reasoning chain up to step $i$ and you sample 100 completions from that point, and 80% of those completions reach the correct final answer, then step $i$ is likely correct. If only 5% of completions from step $i$ reach the correct answer, then step $i$ is likely flawed. More precisely, the Monte Carlo estimate of step correctness is:

$$\hat{r}_i = \frac{1}{K} \sum_{k=1}^{K} \mathbf{1}[\text{completion}_k \text{ reaches correct answer}]$$

where $K$ completions are sampled from the language model conditioned on the problem and all steps up to $i$. This approach, used in systems like MCTS-guided PRMs and the Math-Shepherd dataset, avoids human annotation entirely but introduces noise. A step that uses an incorrect method but arrives at a correct intermediate value (or where the error has not yet manifested) may be assigned high Monte Carlo probability even though it is logically flawed. The method works best when errors have clear downstream consequences in the form of wrong final answers.

Automated verifiers for specific domains (such as symbolic math solvers or code executors) can provide exact correctness labels without human annotation. A code step that produces the right intermediate output is correct; one that does not is wrong. For mathematical reasoning, a computer algebra system can verify algebraic transformations exactly. This applies to structured domains but not to open-ended reasoning or natural language arguments.

Weak labeling from model confidence uses the model's own probability estimates as a proxy for correctness. While noisy, this can be used to create large training datasets cheaply, which can then be refined with a smaller amount of human annotation. This semi-supervised approach allows scaling the labeled dataset beyond what pure human annotation permits.

Consensus labeling has multiple models or multiple annotators label the same steps independently and uses agreement to identify high-confidence labels. Steps where all annotators agree are straightforward to include; steps with high disagreement may require more careful review or may be excluded from training.

In practice, most deployed PRMs use a combination: a large number of Monte Carlo estimated labels supplemented by a smaller number of high-quality human labels. The human labels ensure that the PRM learns from at least some examples where correctness has been definitively established, while the Monte Carlo labels provide coverage across a wide range of problem types and reasoning styles.

The PRM backbone is typically initialized from the same pretrained language model used for generation. This initialization is important because the PRM needs to understand the mathematical language and reasoning conventions present in the steps it is scoring. A randomly initialized scorer would need to learn the domain from scratch, which requires more training data and produces a less reliable scorer.

After the backbone is initialized, step marker tokens are inserted at the boundaries between steps in the training data. These markers serve as anchor points where the classification head reads off the hidden state. During training, the backbone weights are updated along with the classification head, allowing the backbone to adapt its representations specifically for step correctness judgment.

One practical challenge is that step labels are often imbalanced: in any large dataset of model-generated reasoning, most steps are correct, and incorrect steps are a minority. This imbalance can cause the model to be overly optimistic in its scoring. Addressing this requires either resampling (increasing the proportion of incorrect-step examples in training batches) or loss reweighting (applying higher loss weight to incorrect-step examples).

Calibration is another concern. A PRM whose probability outputs are well-calibrated is more useful than one that merely ranks steps correctly. A score of 0.8 should mean that approximately 80% of similarly-scored steps are correct. Calibration can be checked by collecting held-out step labels and comparing model confidence scores against empirical accuracy across score bins.