InstructGPT & RLHF (2022) › Limitations

Despite its successes, the InstructGPT approach faced several important limitations. The process required extensive human labor to create training datasets and collect preference comparisons. This made RLHF expensive to scale, as each iteration required new human annotations. The cost and time required to collect feedback created bottlenecks in model development and limited how frequently models could be updated based on new feedback.

The quality and consistency of human feedback also varied, creating challenges for training stable reward models. Different labelers might have different preferences or standards for what constituted a "good" output, leading to noisy training signals. While techniques like multiple labeler aggregation helped, variation in human judgment remained a fundamental challenge. This variation could also lead to inconsistencies in how models handled edge cases or ambiguous requests.

The reward model learned from a finite set of human comparisons, so it could only approximate human preferences within the distribution of examples it saw during training. For prompts or outputs very different from the training data, the reward model's predictions might not accurately reflect human preferences. This limitation meant that aligned models could still produce misaligned outputs in novel contexts.

The RLHF process also didn't fully solve the problem of objective specification. Humans might have conflicting preferences or preferences that changed over time. Different groups of users might value different aspects of output quality, making it challenging to create models that satisfied everyone. The process optimized for an aggregate of human preferences, which might not match individual user preferences in specific contexts.

The computational cost of RLHF training was substantial. Running reinforcement learning on large language models required significant computational resources, making the process expensive and time-consuming. This limited how frequently models could be updated and made experimentation with different reward model architectures or training procedures costly.

Additionally, the three-stage process created dependencies between stages that could lead to failure modes. If the supervised fine-tuning dataset was biased or limited, those biases would propagate through subsequent stages. If the reward model had systematic errors, those errors would be amplified during reinforcement learning. Each stage needed to be carefully executed, and problems in earlier stages could be difficult to fix in later stages.

The approach also left alignment concerns unresolved. Models trained with RLHF could still produce biased outputs, hallucinate information, or fail in ways that weren't captured in the training data. While RLHF improved alignment substantially, it didn't completely solve the alignment problem, leaving room for continued research and improvement.