GSM8K › Process Supervision vs. Outcome Supervision

An important conceptual distinction that GSM8K helped crystallize is the difference between process supervision and outcome supervision in training reasoning-capable models. This distinction has practical consequences for both training efficiency and the reliability of the resulting system.

In outcome supervision, the training signal rewards or penalizes the model based solely on whether it produced the correct final answer. The intermediate reasoning steps receive no direct feedback. This is the simplest setup to implement: collect problems with known correct answers, run the model, check if it got the right answer, and update accordingly. The model learns to produce correct outputs but has no explicit incentive to generate correct intermediate steps. It might arrive at the right answer through invalid reasoning, or fail at the last step despite correct intermediate work.

Outcome supervision is well aligned with the standard GSM8K evaluation metric, which is purely based on final answer correctness. However, it creates models whose reasoning chains may be unreliable as explanations. A model trained entirely on outcome supervision might learn to generate plausible-sounding but mathematically invalid reasoning chains that happen to terminate at the correct answer.

In process supervision, the training signal evaluates each individual step in the reasoning chain. Steps that advance toward the correct solution receive positive reward; steps that introduce errors receive negative reward. This requires either human annotation of intermediate steps or automated verification of mathematical claims at each step.

OpenAI's research on process-supervised reward models (PRMs) demonstrated that models trained with step-level supervision outperform those trained with only outcome supervision on GSM8K, particularly on harder problems. The PRM800K dataset, released alongside this research, contains 800,000 step-level annotations on GSM8K problems. Each annotation labels individual reasoning steps as correct, incorrect, or neutral. This gives a fine-grained supervision signal.

The intuition behind process supervision's advantage is that it explicitly trains models to maintain correct intermediate state. Rather than letting the model to discover any path to the correct answer, including shortcuts that do not generalize, process supervision enforces that each step must be logically valid. This produces models whose reasoning chains are more reliable as formal arguments, not just generators of correct final answers. The downstream benefit is improved calibration: models trained with process supervision are less likely to generate confidently wrong reasoning chains and are more likely to produce interpretable traces that humans can verify.

The tradeoff is annotation cost. Labeling 800,000 reasoning steps requires substantial human effort, while collecting outcome labels requires only verifying final answers. Researchers have explored automated approaches to process supervision, such as using symbolic verifiers or executing code generated at each step, to scale this supervision signal without proportional annotation cost.