Reward Hacking › Practical Implications

Understanding reward hacking has several practical implications for building and deploying aligned language models. These implications should inform the training process and the entire lifecycle of model development and deployment.

Monitoring is needed throughout training and deployment. You cannot assume that high reward scores indicate high-quality outputs, especially as training progresses. Human evaluation on held-out samples remains necessary throughout training to detect when reward hacking emerges, and continued human evaluation in deployment catches failure modes that were not apparent during training. The proxy metric and the true quality metric can diverge gradually; regular checks prevent small divergences from becoming large failures.

Conservative optimization is safer than aggressive optimization. Given the over-optimization curve, stopping training at or slightly before the reward peak is generally better than pushing for maximum reward scores. The gains from additional optimization past the peak typically do not justify the risk of quality degradation, and the optimal stopping point is difficult to identify precisely in real training runs. Building in evaluation checkpoints that compare proxy scores against human evaluations allows you to track where you are on the over-optimization curve and stop before quality degrades significantly.

Diverse evaluation matters for detecting subtle failures. A single reward model captures one perspective on quality. Using multiple reward models, diverse human evaluators with different backgrounds and expertise levels, and varied evaluation prompts that cover edge cases provides more reliable signals about actual model behavior. Models that look excellent on standard benchmarks may still exhibit reward hacking on unusual inputs or specialized domains. The failure modes described in this chapter tend to be particularly severe on low-frequency inputs that were poorly represented in training data, so evaluation should deliberately target these cases.

Reward hacking awareness should inform system design, not just training. When deploying a model that was trained with RLHF, you are deploying a system that has been specifically optimized to achieve high scores on a proxy metric. The model will be better at the behaviors that the reward model rewarded and worse at the behaviors that the reward model underweighted. Understanding this optimization history helps you anticipate where the model might fail and design appropriate safeguards for high-stakes applications.