Reasoning Verification › Summary

Reasoning verification addresses the gap between generating plausible reasoning and generating correct reasoning. The key ideas from this chapter are:

• Step verification breaks reasoning into individual checkable claims rather than evaluating only the final answer. This enables error localization and finer-grained quality control, and is motivated by the observation that verification is often easier than generation.

• Process reward models formalize step verification by training a model to score each reasoning step. They require step-level labels (expensive to collect but highly informative) and provide substantially better selection signal than outcome reward models, as demonstrated by the PRM800K experiments showing a 78.2% versus 72.4% accuracy advantage at large selection budgets.

• Verification-guided search uses PRM scores to steer reasoning toward correct paths. Best-of-N selects the highest-scoring complete chain; beam search prunes low-scoring partial chains; MCTS adaptively allocates compute to promising reasoning branches using the same UCT-based exploration-exploitation trade-off pioneered in game-playing AI.

• Self-correction allows models to review and revise their own outputs without a separate verifier, but is most effective when backed by external feedback (code execution, constraint satisfaction) rather than relying solely on the model's own judgment. Naive self-correction without external grounding often fails to improve accuracy.

• Reward hacking and calibration are practical concerns when PRMs are deployed in production. The same distributional mismatch and calibration issues that affect reward models in RLHF apply here, and require ongoing monitoring and retraining.

• Together, these components form a reasoning pipeline that is substantially more reliable than single-pass generation, at the cost of additional inference compute. The compute overhead is controllable and can be balanced against accuracy requirements for different application settings.

The next chapter extends these ideas to mathematical reasoning specifically, where the step structure is particularly well-defined and automated verification is often possible through symbolic computation.