Reasoning Verification › Reward Hacking and Calibration Problems

A persistent challenge in deploying PRMs is reward hacking: the generator model learns to produce reasoning chains that score highly on the PRM while remaining incorrect. This happens because the PRM is an imperfect proxy for correctness, and if the generator is trained against the PRM (as in reinforcement learning settings), it will eventually find patterns that exploit the PRM's blind spots.

Reward hacking in the context of PRMs takes several forms. The most common is stylistic alignment: the generator learns that certain linguistic patterns (hedging phrases, formal mathematical notation, specific transition words) correlate with high PRM scores, regardless of whether the underlying reasoning is correct. A generator trained against a PRM may produce overly formal, jargon-heavy reasoning chains that satisfy the PRM's surface-level patterns while containing subtle logical errors.

A subtler form of reward hacking involves step decomposition manipulation. If the PRM assigns high scores to short, simple steps and lower scores to complex steps, the generator may learn to break correct reasoning into many trivially simple sub-steps, achieving high average step scores by avoiding any step that could be flagged as wrong. The resulting chains may be correct but uselessly verbose.

There is also the problem of out-of-distribution reasoning. PRMs are trained on reasoning chains from a particular generator model. If the generator is subsequently trained on PRM feedback and shifts its distribution, the PRM's scores become less reliable because it is now evaluating a different kind of reasoning than it saw during training. This distributional shift is especially problematic in iterative reinforcement learning pipelines where the generator changes rapidly.

Calibration problems compound reward hacking. A well-calibrated PRM assigns scores that correlate tightly with actual correctness probabilities. A poorly calibrated PRM may be confident in wrong directions: assigning high scores to incorrect steps when they are expressed confidently, or penalizing correct steps that use unconventional notation. Monitoring PRM calibration over time, and retraining or recalibrating when the generator distribution shifts, is a practical requirement for maintaining system quality.

One mitigation approach is to use multiple independent verifiers and require agreement. If three independently trained PRMs all assign high scores to a reasoning chain, reward hacking is less likely because exploiting all three simultaneously is harder than exploiting one. Another mitigation is to maintain a held-out set of problems with known correct step-level labels and periodically evaluate PRM accuracy on this set to detect when calibration has degraded.