InstructGPT & RLHF (2022) › The Problem

Large language models trained through unsupervised learning on vast text corpora learned to predict the next token in sequences, but they weren't trained to be helpful, honest, or harmless according to human preferences. When these models were deployed for practical applications, they exhibited several critical limitations that prevented effective real-world use. These limitations explain why InstructGPT added explicit human preference optimization.

Models trained only on next-token prediction would generate outputs that were fluent and coherent but didn't necessarily match human intent or preferences. A model might produce factually accurate but overly verbose explanations when a user wanted a concise answer. It might generate harmful content when asked to write from a particular perspective. It might refuse to help with legitimate tasks while gladly completing requests that violated safety guidelines. These misalignments arose because the training objective didn't distinguish between different types of outputs based on how helpful, accurate, or safe they were.

The misalignment problem manifested in several specific ways. Models often generated outputs that were technically plausible but not useful. For example, when asked to write code, a model might produce syntactically correct programs that didn't solve the actual problem. When asked to summarize documents, models might include irrelevant details or miss key points. The model's understanding of what constituted a "good" output came from statistical patterns in training data, not from explicit human preferences about helpfulness or accuracy.

Safety concerns represented another critical dimension of misalignment. Models trained on web text would reproduce harmful content from their training data, including biases and stereotypes. Without explicit optimization for safety, models couldn't reliably distinguish between requests that should be fulfilled and those that should be refused. This made deployment risky, as models might generate content that was inappropriate, biased, or potentially harmful.

Even when models could perform tasks correctly, they often didn't match human preferences for how tasks should be performed. A model might answer a question correctly but use overly formal language when a casual response was preferred. It might provide too much detail when a brief answer was requested, or too little detail when a thorough answer was needed. These preferences weren't captured in the next-token prediction objective, so models couldn't learn to adapt their behavior based on what humans found useful.

Traditional fine-tuning approaches addressed some of these issues but had fundamental limitations. Supervised fine-tuning on high-quality datasets could improve performance on specific tasks, but it required manually creating datasets of desired outputs. This approach didn't scale well because covering every possible use case would be prohibitively expensive. Supervised fine-tuning also struggled to capture context-dependent human preferences that were difficult to express as explicit examples.

The few-shot learning capabilities of models like GPT-3 also had limitations for alignment. While few-shot examples could guide model behavior in specific instances, they didn't systematically improve the model's fundamental alignment with human preferences. Each interaction required crafting effective prompts, and the model's behavior varied depending on prompt formulation rather than being reliably aligned with user intent.

Researchers needed a method that could systematically optimize models for human preferences across diverse contexts, rather than relying on prompt engineering or task-specific fine-tuning. The challenge was creating a training process that could learn context-dependent judgments about output quality and safety from human feedback at scale. The three-stage RLHF process that InstructGPT pioneered addressed this challenge.