Process Reward Models › Outcome Reward vs. Process Reward

To understand why process reward models are useful, start with the simpler approach: outcome reward models (ORMs).

An outcome reward model evaluates a complete solution and assigns a single score based on whether the final answer is correct. Given a problem and a full response, the ORM produces a scalar signal indicating solution quality. This approach is natural, simple to implement, and works well when you have access to ground-truth final answers.

The appeal of ORMs is their simplicity. If you can automatically verify the final answer (for example, by checking whether a mathematical answer matches a known solution, or whether generated code passes a test suite), you can generate a training signal without human annotation. This is how many early RLHF pipelines worked: generate a solution, check the answer, assign a reward, update the policy.

The problem with outcome reward is that it is sparse and delayed. Consider a 10-step mathematical derivation. If step 3 contains a subtle algebraic error that propagates through all subsequent steps, the model receives no signal about where the reasoning went wrong. It only knows the final answer was incorrect. This is analogous to telling a student "your answer is wrong" without any feedback on which part of their reasoning was flawed. The learning signal is weak because it is not connected to the individual decisions that caused the error.

Sparsity has a direct effect on sample efficiency. In reinforcement learning, sparse rewards are notoriously difficult to optimize because the signal is so infrequent and so far removed from the individual actions that produced it. Techniques like reward shaping, potential-based reward functions, and hierarchical RL exist precisely to address this problem by introducing intermediate reward signals. Process reward models are, in essence, a domain-specific form of reward shaping for multi-step reasoning.

Process reward models provide a dense reward signal instead. Each step in the chain receives its own score. If step 3 contains an error, the PRM assigns a low score to step 3, regardless of whether subsequent steps recover or the final answer happens to be correct by coincidence. This fine-grained feedback allows training procedures to distinguish between "right answer via wrong reasoning" and "right answer via correct reasoning."

The difference between outcome and process reward becomes particularly important in longer reasoning chains. In a 20-step derivation, the probability that all steps are correct given only a correct final answer is not high: the model might have made compensating errors, used a non-generalizable shortcut, or gotten lucky with an algebraic cancellation. Outcome reward cannot distinguish any of these from sound reasoning. Process reward can.

Credit assignment is the question of which actions deserve credit for an outcome. In sequential decision-making, this is one of the central challenges: when a sequence of decisions leads to a reward, how do you know which decisions mattered?

Temporal difference learning and Monte Carlo methods in reinforcement learning both address credit assignment through different means: TD methods propagate rewards backward through time using bootstrapped value estimates, while Monte Carlo methods average returns over entire trajectories. Both approaches work when rewards are reasonably frequent, but they degrade substantially as the gap between actions and rewards grows.

With outcome reward applied to long reasoning chains, credit assignment is left entirely to the learning algorithm. The optimizer must infer, from the signal "this solution was wrong," which of the many steps in the reasoning chain contributed to the failure. In a chain of 15 steps, this requires attributing the outcome correctly across a very large space of possible step combinations. The gradient signal becomes increasingly diluted, and the optimizer may converge to policies that avoid certain patterns for the wrong reasons or that produce superficially correct-looking reasoning without improving correctness.

Process reward solves credit assignment explicitly by labeling each step. Step 7 receives a high score if step 7 is correct, independent of what happens in steps 8 through 20. The credit assignment problem is reduced from a complex inference problem to a direct annotation: was this step valid? The optimizer can now learn, with a strong gradient signal, exactly which step patterns are associated with correctness.

This explicit credit assignment also has implications for interpretability. When a PRM assigns a low score to step 7 in a 15-step chain, you know step 7 is the problem. This makes PRM-based feedback substantially more useful as a debugging tool than ORM-based feedback, which only tells you that something went wrong somewhere in the chain.

There are specific patterns where outcome reward models fail systematically:

Coincidental correctness. A model might reach the correct final answer through erroneous intermediate steps that happen to cancel out. This is common in algebra, where two errors can offset each other. An ORM rewards this behavior, effectively training the model to use flawed reasoning that occasionally produces correct answers.

Reward hacking. Models trained with outcome reward sometimes discover shortcuts that satisfy the scoring criteria without developing reasoning ability. For example, a model might learn to produce a final numerical answer in the correct format and range without solving the problem, if the scoring function is not sufficiently discriminating.

Long-horizon degradation. As reasoning chains grow longer, the probability that a single error in an early step propagates and causes a wrong final answer increases. The outcome reward becomes harder to use as a learning signal because the signal is noisy: many correct paths share the label "wrong answer" simply because of one early mistake, and the optimizer receives contradictory feedback. The same step can receive positive reward in one trajectory (where it led to a correct final answer) and negative reward in another (where it led to an incorrect one), not because the step itself was different, but because later steps varied. Process reward eliminates this noise by evaluating each step independently.

Distributional shift during training. As the policy improves and generates different distributions of reasoning chains, the outcome reward signal shifts accordingly. Early in training, a step that was usually followed by correct reasoning might later appear in chains that have drifted toward different patterns. The outcome reward provides no signal to stabilize this: it only sees the final answer. A process reward model, updated to match the current policy's distribution, provides a more stable and consistent learning signal.