RLHF Foundations (2017) › 2017: RLHF Foundations

In 2017, researchers at OpenAI and DeepMind studied a problem in reinforcement learning: how could agents learn complex behaviors when engineers could not specify an adequate reward function? For tasks involving natural language or robotics, desired behavior often depended on human preferences that were difficult to express mathematically. Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei developed a method that allowed reinforcement-learning agents to learn from human comparisons rather than only predefined rewards.

The paper, "Deep Reinforcement Learning from Human Preferences," described a framework in which evaluators compared pairs of agent behaviors. A reward model learned from those comparisons, then supplied the signal used for reinforcement learning. This replaced an explicitly programmed reward function with preferences expressed through comparative judgments. In the paper's experiments, agents learned behaviors that matched evaluator preferences even when those preferences were difficult to formalize.

The framework was later adapted beyond the reinforcement-learning tasks in the original paper. Language models presented a related problem: maximum-likelihood training could produce fluent text without directly optimizing for evaluator preferences about its content or behavior. Preference learning supplied one method for adding that signal during model fine-tuning.

The method reduced the need to define rewards for every possible scenario. Evaluators supplied a sample of comparisons, and the learned reward model estimated preferences for new trajectories. This property made the approach applicable to language models, whose possible outputs cannot be enumerated in advance, although collecting comparison data still required substantial human effort.