RLHF Foundations (2017) › Applications and Impact
History of Language AIAll chapters RLHF Foundations (2017) › Applications and ImpactOne section · Sections in this chapter4 / 7 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The initial experiments covered simulated robotic manipulation and Atari games. In the robotics tasks, agents learned behavior that evaluators judged safe and efficient. In Atari, comparison feedback could favor behavior evaluators considered interesting or skillful rather than behavior selected only by game score. These experiments tested preferences that were difficult to express through simple reward functions.
The same approach could be applied where user judgments mattered but were difficult to formalize. A recommendation system, for example, could learn from comparative user behavior rather than a single hand-written quality metric. An autonomous system could also learn preferences about safe operation and appropriate interaction without an exhaustive list of rules.
Several years later, researchers adapted the preference-learning framework to large language models. Standard language-model objectives optimized next-token prediction rather than evaluator judgments about a response. The resulting alignment problem resembled the earlier task of fitting agent behavior to human comparisons.
For language models, evaluators compared responses to the same prompt instead of trajectories in an environment. A reward model learned from those comparisons and guided reinforcement-learning fine-tuning, often with Proximal Policy Optimization. This pipeline became known as Reinforcement Learning from Human Feedback, or RLHF.
Developers have used RLHF or related preference-learning methods to adjust models such as GPT-3.5, GPT-4, and Claude toward evaluator-selected behavior. These methods supplement language-model training with judgments about response usefulness and acceptable conduct. They have become a common stage in post-training at major AI labs.
Preference learning also applies beyond language models. Image generators can learn from comparative judgments about outputs, while code generators can use programmer rankings of proposed code. The approach is relevant when direct human comparison captures desired behavior better than a fixed metric.