RLHF Foundations (2017) › The Problem

Traditional reinforcement learning relies on a reward function that provides a scalar reward signal to the agent after each action or at the end of an episode. This reward function is the primary learning signal, guiding the agent to discover behaviors that maximize expected cumulative reward. For many applications, this approach works well when the reward function can be precisely specified. In game-playing scenarios, for example, the reward might be the game score, which is clearly defined and measurable. In control tasks, the reward might be based on measurable quantities like distance traveled, energy consumed, or task completion metrics.

However, many important applications involve objectives that are difficult or impossible to encode as precise reward functions. Consider a robotic agent learning to assist with household tasks. An engineer might try to define a reward based on metrics like task completion time or the number of objects moved, but these metrics might miss important aspects of what makes assistance helpful: being gentle with fragile items, cleaning up afterward, or understanding the user's unstated preferences. Similarly, for language tasks, a model trained to maximize likelihood might generate text that is grammatically correct but unhelpful, verbose, or unsafe. What makes text "good" often involves subtle qualities that resist simple mathematical formulation.

Another challenge arises when a reward function does not fully capture the desired behavior. Optimization can exploit gaps in such a function, producing high reward without the intended result. Examples include game-playing agents that pause indefinitely to avoid negative rewards and cleaning robots that hide messes instead of removing them. A reward that appears reasonable during design can therefore encourage unintended behavior.

Human preferences also involve trade-offs that are difficult to capture in one scalar function. An evaluator might prefer a response that balances information with a friendly, concise presentation over one optimized only for informativeness. Encoding such judgments requires engineers to choose weights and assumptions about qualities whose relative importance changes across users or contexts.

Reward-function engineering also becomes more expensive as task variety grows. A new scenario can require a revised reward structure followed by testing and adjustment based on observed behavior. Systems that must handle varied situations or user preferences can therefore spend substantial effort on reward design.

Evaluation posed a related problem. Engineers still needed human judgments to determine whether a hand-written reward function produced acceptable behavior. When evaluation exposed a mismatch, they revised the function and retrained the agent. Incorporating comparisons into the learning process moved those judgments into the reward model instead of using them only after training.