RLHF Foundations (2017) › Limitations

While preference-based learning addresses many challenges, it also introduces new limitations and considerations. One fundamental issue is that human preferences may be inconsistent, context-dependent, or difficult to express through pairwise comparisons. Different humans might have different preferences, and the same human might express different preferences at different times or in different contexts. The method assumes that preferences are reasonably stable and consistent enough to learn from, but in practice, this assumption may not always hold.

The quality of the learned reward model depends critically on the quality and quantity of human preference data. If human evaluators provide noisy, inconsistent, or biased comparisons, the reward model will learn to reflect those issues. Poor quality preference data can lead to reward models that don't accurately capture human values, which then guide agents or models toward misaligned behaviors. Collecting high-quality preference data requires careful attention to evaluation protocols, training for evaluators, and quality assurance measures.

The scalability of human evaluation remains a challenge. While preference-based learning reduces the need for humans to specify reward functions, it still requires significant human effort to provide comparisons. For large-scale applications, this can become expensive and time-consuming. The method improves efficiency by learning generalizable reward models from relatively small amounts of comparison data, but human evaluation remains a bottleneck that limits how quickly systems can be improved or adapted to new domains.

Reward hacking remains possible when an agent finds behavior that scores highly under the learned reward model but conflicts with evaluator preferences. The reward model is an approximation and may contain blind spots or systematic errors. Optimization can exploit those gaps, just as it can exploit a hand-written reward function, while the learned model may make the failure harder to diagnose.

The method also assumes that pairwise comparisons provide sufficient information to learn good reward models. However, some preferences might be inherently difficult to express through comparisons, especially when preferences involve multiple dimensions that can't be easily reduced to a single ranking. For example, a conversation might be preferred in some ways but not others, and forcing a single preference judgment might lose important nuance. More sophisticated preference elicitation methods might be needed for complex multi-dimensional preferences.

There are also questions about whose preferences are being learned. The method learns from the preferences of the humans who provide comparisons, but these humans might not be representative of all users or stakeholders. Preferences might vary across cultures, contexts, or individuals, and learning from one group's preferences might not produce behaviors that align with other groups' values. This concern becomes especially important for language models and other systems with broad user bases, where diverse preferences must be considered.

Finally, the iterative improvement process, while valuable, can be slow. Each cycle of generating behaviors, collecting human feedback, updating the reward model, and retraining the agent takes time and resources. For applications requiring rapid adaptation or frequent updates, this iterative process might not be fast enough. The method works well when there is time for careful alignment, but may be less suitable for scenarios requiring quick deployment or frequent changes.