Process Reward Models › PRM Limitations

Process reward models represent a significant advance in training signal quality for reasoning tasks, but they come with substantial limitations that constrain their practical applicability.

The most fundamental limitation is the cost of step-level annotations. Human annotation of individual reasoning steps is significantly more expensive than annotating final answers. A human rater must read and understand each step, verify its logical validity given the preceding context, and make a correctness judgment. For complex mathematical proofs or multi-step code generation, this is cognitively demanding work that cannot be delegated to non-experts.

The OpenAI PRM800K dataset, one of the most widely used PRM training resources, required approximately 800,000 step-level annotations from human mathematicians to cover the MATH benchmark. Scaling this to other domains, longer reasoning chains, or more complex problems requires proportional increases in annotation cost. The economics of annotation become a fundamental bottleneck: you need expert annotations at scale, but expert time is both expensive and limited in supply.

The Monte Carlo estimation approach alleviates this somewhat by automating label generation, but introduces a different cost: computational overhead. Generating hundreds of continuations per reasoning prefix requires significant inference compute. The quality of Monte Carlo labels also depends on the quality of the base model: if the base model is weak, Monte Carlo rollouts are dominated by noise and produce unreliable labels. Worse, if the base model has systematic biases (always preferring a certain incorrect approach), the Monte Carlo estimates will reflect those biases rather than true step correctness. The Monte Carlo approach effectively bootstraps the quality of the PRM from the quality of the base model, which creates a chicken-and-egg problem early in development.

A PRM trained on one domain or problem type may not generalize well to others. A PRM trained on competition mathematics does not automatically become a useful evaluator for code generation, logical reasoning, or scientific problem solving. The step correctness criteria differ significantly across domains, and the surface patterns that correlate with correctness in one domain may not transfer to another.

This is a practical limitation because building a PRM requires both a large base policy (to generate training data) and a large annotated dataset (to train the PRM). Both requirements are domain-specific. As a result, PRMs have primarily been demonstrated in narrow, well-defined domains like mathematics and code generation. Extending to broader reasoning domains, or building a general-purpose PRM that works across many different types of problems, remains an open challenge.

Even within mathematics, generalization across subfields is non-trivial. A PRM trained predominantly on algebra may not perform well on combinatorics, number theory, or calculus, even though all are mathematical reasoning tasks. The notation conventions, typical error patterns, and valid proof strategies differ enough across subfields that domain-specific training may be necessary.

A PRM requires that the reasoning chain be decomposed into discrete steps that can be individually evaluated. This decomposition is not always natural. Mathematical proofs have clear step structure; but other forms of reasoning (creative writing planning, scientific hypothesis formation, open-ended question answering) are harder to decompose into steps with independent correctness criteria.

Even within mathematics, the granularity of steps affects PRM quality. A coarse decomposition (few, long steps) may bundle multiple reasoning operations into a single step, making it difficult to localize errors. A fine decomposition (many, short steps) increases annotation cost and may produce trivially correct micro-steps that provide little training signal.

A recurring challenge in reward model-based training is overoptimization: the policy learns to maximize the reward model's score without improving the underlying capability. This problem appears in standard RLHF (where it was studied in detail by Gao et al., 2023) and is equally present in PRM-guided training.

A policy trained against a PRM can learn to produce steps that the PRM predicts are correct without those steps being valid. If the PRM has systematic biases (for example, rewarding verbose steps, steps with a lot of mathematical notation, or steps that match surface patterns from the training distribution), the policy will exploit those biases. Over many training iterations, the policy and PRM diverge from the underlying ground truth, and performance can degrade despite high PRM scores.

This phenomenon is sometimes called Goodhart's Law in the ML context: when a measure becomes a target, it ceases to be a good measure. The PRM's score was correlated with correctness when it was trained, but once the policy actively optimizes against that score, the correlation degrades. The policy finds adversarial examples for the PRM, generating solutions that receive high step scores but are factually incorrect.

One mitigation is to periodically update the PRM using new data generated by the current policy. By continuously retraining the PRM on the distribution the policy produces, you reduce the gap between what the PRM was trained on and what the policy generates. Another mitigation is to combine PRM guidance with verifiable correctness checks where possible (for example, running code to verify that a generated program produces the correct output). Neither approach fully eliminates overoptimization, and this remains an active area of research.

A PRM assigns a low score to any incorrect step, even if a skilled solver could recover from that step and reach the correct answer. In some reasoning contexts, detours through incorrect intermediate claims are a natural part of exploration. A student solving a novel problem might try an approach that does not work out, recognize the failure, and backtrack. A PRM that penalizes these exploratory steps may discourage the kind of flexible reasoning that leads to creative solutions.

This tension between strict step correctness and exploratory reasoning is not fully resolved in current PRM research. Most PRMs are trained to evaluate steps in a forward-only manner, given a specific chain, rather than to evaluate the overall quality of a reasoning strategy.

Beyond the general limitations, PRMs face challenges specific to their primary application domain:

Symbolic ambiguity. Mathematical notation admits many equivalent forms. The PRM must recognize that different surface forms of the same expression are all valid.

Proof branching. Many mathematical results can be proven by multiple valid approaches. A PRM trained predominantly on one standard proof technique may incorrectly penalize steps from an alternative but equally valid proof strategy.

Implicit assumptions. Mathematical steps often rely on implicit assumptions (for example, that a variable is positive or that a function is continuous) that are not stated explicitly. Evaluating whether a step is valid requires understanding these implicit conditions from context, which is challenging for current language model-based PRMs. A step that applies a theorem whose conditions have not been verified should receive a low score, but detecting this requires the PRM to track which conditions have and have not been established throughout the chain.

Notation-sensitive errors. Some mathematical errors are only visible when you expand implicit operations. A PRM that processes mathematical text at the surface level may not detect an error that is only apparent when you work through the detailed arithmetic or algebraic manipulation behind a compact expression.

Beyond the conceptual limitations, PRMs face practical challenges when deployed in real systems.

Inference cost is significant. Each step evaluation requires a forward pass through the PRM backbone, which is typically a large language model. In beam search with a beam width of 8 and a 20-step reasoning chain, you run the PRM 160 times per problem. At scale, this inference cost must be weighed against the accuracy improvement it provides. For many applications, the compute budget required for PRM-guided search exceeds what is available at inference time.

Latency is a related concern. PRM-guided beam search has higher latency than greedy decoding because steps must be evaluated before the next step is generated. For interactive applications where users expect low-latency responses, this creates a tension between reasoning quality and response speed. Distilling the PRM's quality improvements into the policy model, training the policy to reason like PRM-guided search would but without running the PRM at inference time, is one approach to resolving this tension, and it is an active research direction.

Step boundary detection is also non-trivial in practice. The PRM must know where one step ends and the next begins. For well-formatted mathematical solutions, step boundaries are usually clear. For more conversational reasoning, the boundaries are ambiguous. Training a secondary model to detect step boundaries, or requiring the policy to format its reasoning in a structured way that makes boundaries explicit, adds complexity to the overall system.