RLHF Foundations (2017) › The Solution

The preference-based learning framework addresses these challenges by separating reward specification from reward learning. Instead of requiring engineers to design reward functions, the method learns a reward model from human feedback, then uses this learned reward model to train reinforcement learning agents. The key insight is that humans can more easily compare behaviors than they can specify exact reward values or functions. By collecting pairwise comparisons from human evaluators, the system can learn what humans value without requiring them to formalize their preferences mathematically.

The process works in three main stages. First, a reinforcement learning agent interacts with the environment, generating trajectories of behavior. These trajectories are presented to human evaluators in pairs, and the evaluators indicate which trajectory they prefer. Second, a reward model is trained to predict human preferences by learning to assign higher rewards to trajectories that humans prefer. This reward model learns from the pairwise comparison data, developing an internal representation of what makes behaviors desirable according to human judgment. Third, the reinforcement learning agent is trained using the learned reward model as the source of rewards, optimizing its policy to maximize expected reward according to the reward model's predictions.

The reward model is typically implemented as a neural network that takes a trajectory as input and outputs a scalar reward value. During training, the model learns to assign higher rewards to preferred trajectories and lower rewards to non-preferred ones. The training objective encourages the reward model to correctly rank trajectories according to human preferences. Specifically, the model is trained using a ranking loss that maximizes the difference in predicted rewards between preferred and non-preferred trajectories. This approach allows the reward model to generalize beyond the specific comparisons it was trained on, learning general principles about what makes behaviors desirable.

The key technical innovation involves how trajectories are compared. Rather than requiring humans to evaluate complete trajectories, which might be long or complex, the method allows comparisons of trajectory segments. Human evaluators might compare specific segments of behavior, focusing attention on the most relevant parts. This segmentation makes human evaluation more efficient and allows the reward model to learn fine-grained preferences about specific aspects of behavior. The learned reward model can then aggregate these segment-level preferences when evaluating complete trajectories.

Another important aspect is how the reward model handles uncertainty. When the reward model is uncertain about which trajectory is preferred, it should express this uncertainty rather than making confident but potentially incorrect predictions. The framework incorporates this through the reward model's training, which learns point estimates of reward together with confidence in those estimates. This uncertainty can be used to guide further human evaluation, focusing attention on cases where the reward model is least confident and where additional human feedback would be most informative.

The integration with reinforcement learning uses the learned reward model as a drop-in replacement for a traditional reward function. Standard reinforcement learning algorithms, such as policy gradient methods or actor-critic approaches, can be applied without modification. The agent receives rewards from the reward model rather than from a hand-coded function, but from the agent's perspective, the learning process is identical. This compatibility meant that existing reinforcement learning infrastructure and algorithms could be used with minimal changes, making the approach practical to implement and experiment with.

The method also supports iterative updates. As the policy changes, it generates trajectories that differ from those used to train the initial reward model. Evaluators can compare these new trajectories, supplying additional data for another reward-model update. Repeating this process updates both the learned reward and the policy against behaviors the current policy can produce.