InstructGPT & RLHF (2022) › The Solution

InstructGPT solved the alignment problem through a three-stage training process that systematically incorporated human feedback into model optimization. This approach built on earlier work in reinforcement learning from human preferences, adapting those techniques specifically for large language models. The three stages worked together to create models that were more aligned with human preferences than models trained solely through next-token prediction.

The first stage involved collecting a dataset of high-quality prompt-response pairs written by human labelers. These labelers were given prompts submitted by users to GPT-3 via the OpenAI API and wrote ideal responses that demonstrated the desired behavior. This created a dataset showing how the model should respond to various types of requests.

The model was then fine-tuned on this supervised dataset using standard language modeling loss. This fine-tuning stage taught the model to recognize and generate the types of responses that humans found useful, establishing a baseline of instruction-following capability. The model learned patterns like answering questions directly, following explicit formatting instructions, and adapting tone and detail level appropriately.

Supervised fine-tuning provided the model with explicit examples of desired behavior. Rather than having to infer what users wanted from ambiguous prompts, the model saw clear demonstrations of how to respond helpfully. The diversity of prompts in this dataset helped the model learn general patterns of helpful behavior that could transfer to new prompts, not just memorize specific responses.

The second stage involved training a separate reward model that could predict how much humans would prefer a given model output. To create this reward model, human labelers were shown multiple outputs generated by the model in response to the same prompt. They ranked these outputs from best to worst based on criteria like helpfulness, accuracy, and safety. This created pairwise comparison data showing human preferences among different possible outputs.

The reward model was trained to predict these human preferences, learning to score outputs based on how much humans would prefer them. This model learned to encode aspects of output quality that supervised fine-tuning could not easily capture. It could recognize that a concise but incomplete answer was less preferred than a thorough explanation, or that a technically correct but inappropriate response should receive a lower score.

Training the reward model on pairwise comparisons rather than absolute scores proved important. Humans are generally better at comparing outputs than assigning absolute quality scores, as comparisons are more reliable and less subjective. The reward model learned patterns that could generalize to new prompts and outputs, enabling it to score responses the model might generate during reinforcement learning.

The final stage used reinforcement learning to optimize the fine-tuned model's outputs based on the reward model. The model generated responses to prompts, and the reward model scored these responses. The model's parameters were then updated to increase the probability of generating outputs that received higher reward scores. This process iteratively improved the model's alignment with human preferences as encoded in the reward model.

The reinforcement learning process used the Proximal Policy Optimization (PPO) algorithm, which was adapted for language model training. PPO helped ensure that the model's policy didn't change too drastically in any single update, maintaining stability during training. The algorithm balanced exploration of new response patterns with exploitation of known high-reward behaviors, gradually shifting the model toward generating outputs that humans preferred.

A key challenge in this stage was preventing the model from "reward hacking," where it might exploit quirks in the reward model to achieve high scores without improving helpfulness. Techniques like KL divergence penalties helped maintain similarity between the fine-tuned model and the original pretrained model, preventing the model from drifting too far from its original capabilities while still optimizing for human preferences.

The three-stage process worked synergistically. Supervised fine-tuning established a baseline of helpful behavior. The reward model learned to recognize and quantify human preferences. Reinforcement learning then optimized the model to generate outputs that maximized these preferences. Together, these stages created models that were systematically more aligned with what humans wanted.