Process Reward Models › Worked Example: Step-Level Scoring

Let us trace through a concrete example to see how a PRM evaluates a multi-step solution. This example illustrates both the information a PRM provides and how it differs from outcome reward.

Consider the problem: Solve for $x$: $2x^2 - 8x + 6 = 0$.

Here is a candidate solution with five steps:

Step 1: Divide both sides by 2 to simplify: $x^2 - 4x + 3 = 0$.

Step 2: Factor the quadratic: $(x - 1)(x - 3) = 0$.

Step 3: Apply the zero product property: $x - 1 = 0$ or $x - 3 = 0$.

Step 4: Solve each equation: $x = 1$ or $x = 3$.

Step 5: Check: $2(1)^2 - 8(1) + 6 = 2 - 8 + 6 = 0$. And $2(3)^2 - 8(3) + 6 = 18 - 24 + 6 = 0$.

A well-trained PRM would assign high scores (say, 0.95 or higher) to all five steps. The derivation is mathematically correct at every stage. Now consider an alternative solution with an error:

Step 1: Divide both sides by 2: $x^2 - 4x + 3 = 0$. (Score: 0.94, correct)

Step 2: Factor incorrectly: $(x - 2)(x - 2) = 0$. Error: $(x-2)(x-2) = x^2 - 4x + 4 \neq x^2 - 4x + 3$. (Score: 0.08, incorrect)

Step 3: Apply the zero product property: $x - 2 = 0$. (Score: 0.71, conditionally correct given step 2)

Step 4: Solve: $x = 2$. (Score: 0.72, conditionally correct given step 3)

Step 5: Check: $2(4) - 16 + 6 = -2 \neq 0$. Inconsistency detected. (Score: 0.15, reveals error)

The PRM assigns a low score to Step 2 because the factorization $(x-2)(x-2) = x^2 - 4x + 4$ does not equal $x^2 - 4x + 3$. An ORM would also correctly reject this solution (the final answer is wrong), but it provides no information about where the error occurred. The PRM's signal is more informative: Step 2 is where the reasoning breaks down.

Notice also that Steps 3 and 4 receive relatively high scores despite being based on the incorrect Step 2. This is appropriate: given the (incorrect) factorization from Step 2, applying the zero product property and solving are valid operations. The PRM correctly attributes the error to Step 2 rather than penalizing Steps 3 and 4 for following logically from a flawed premise.

Now consider a third, more insidious scenario: a solution that reaches the correct numerical answer through an algebraically incorrect path. Suppose a student incorrectly applies the quadratic formula with a sign error but then makes a compensating arithmetic mistake, arriving at $x = 1$ and $x = 3$ despite the flawed derivation. An ORM would reward this solution identically to the correct solution, because the final answers match. A PRM would detect the error at the step where the sign mistake occurs and correctly identify this as a flawed derivation despite the correct answer. Training against an ORM that rewards such solutions teaches the model to produce compensating errors; training against a PRM discourages this pattern entirely.

To make the Monte Carlo label generation procedure concrete, consider how we would estimate the correctness probability for Step 2 in the above example. Starting from the prefix (problem + Step 1 + the incorrect Step 2), we generate $M = 50$ completions of steps 3 through 5 using the base policy model.

From a prefix that includes the incorrect factorization $(x-2)(x-2)$, almost all completions will arrive at $x = 2$ via valid subsequent steps. The final answer will consistently be $x = 2$, which is incorrect. The Monte Carlo estimate becomes:

$$\hat{y}_2 = \frac{\text{number of completions reaching } x \in \{1, 3\}}{50} \approx \frac{1 \text{ or } 2}{50} \approx 0.02$$

The estimate of approximately 0.02 correctly identifies Step 2 as a problematic step. The model can almost never recover from an incorrect factorization and arrive at the correct final answer.

Compare this to the Monte Carlo estimate for Step 1. Starting from the correct prefix (problem + Step 1), the base model can frequently complete the solution correctly:

$$\hat{y}_1 = \frac{\text{number of completions reaching } x \in \{1, 3\}}{50} \approx \frac{40}{50} = 0.80$$

A strong base model succeeds 80% of the time from this point. The Monte Carlo label for Step 1 is 0.80, correctly indicating that this prefix is in good shape. The gap between 0.80 and 0.02 across these two steps provides a training signal that the PRM can learn from: steps that preserve the ability to complete correctly receive high labels, steps that destroy that ability receive low labels.