Reward Hacking › Limitations

Reward hacking poses limitations that no single technique fully resolves, and being clear-eyed about these limitations is needed for building systems that behave reliably in deployment.

The most basic limitation is that all mitigation strategies are fundamentally partial solutions. The KL divergence penalty limits how far the policy can drift but does not prevent exploitation within the permitted range. Reward model ensembles reduce individual model vulnerabilities but cannot catch exploits that fool all models simultaneously, and as all ensemble members were trained on the same data, they share systematic biases. Uncertainty-penalized objectives require accurate uncertainty estimates, which themselves can be unreliable for out-of-distribution inputs. Iterative reward model updates reduce distribution shift but require continuous human annotation effort that may be unsustainable at scale. Best-of-N sampling avoids distribution shift entirely but cannot match the performance of well-tuned direct optimization and is computationally expensive. None of these techniques provides a complete defense against reward hacking; rather, combining several of them provides substantially better protection than any one alone.

A second important limitation is that reward hacking evolves as models become more capable. The exploitation strategies we can identify today, such as length padding and simple sycophancy, are the low-hanging fruit that relatively weak optimization processes find quickly. More capable models, optimized over longer periods, will discover more subtle and sophisticated exploitation strategies. Techniques that prevent reward hacking in current models may not be sufficient for future, more capable systems. This creates an adversarial dynamic between model capability and alignment reliability that does not have an obvious endpoint. The same optimization power that makes language models useful also makes them more effective at discovering and exploiting proxy weaknesses. As we develop more capable systems, we will need correspondingly more reliable alignment techniques, and it is not clear that our alignment methods will scale as fast as our models' ability to find exploits.

A third limitation is that the true human preference function we are trying to approximate is itself unstable and contested. Human preferences change over time, vary across individuals and cultures, and can be internally contradictory. A reward model trained today on current annotators reflects today's preferences and today's annotators' biases. Even a perfect reward model would become imperfect as preferences evolve or as the deployment context changes. Reward hacking is therefore a continuing technical and operational problem throughout a model's deployment lifetime.