RLHF Pipeline › Limitations and Practical Considerations

The RLHF pipeline comes with significant challenges in any production deployment.

Computational cost is substantial. The pipeline requires training three separate models (SFT, reward model, and PPO policy), with the PPO stage being particularly expensive because it requires running both the policy and reference model for every batch. A single RLHF training run can cost hundreds of thousands of dollars in compute for large models, making iteration and experimentation prohibitively expensive for most organizations. This has driven interest in more efficient alternatives like Direct Preference Optimization (DPO), which we'll explore in upcoming chapters. Even at research scale with smaller models, the infrastructure complexity of running three models simultaneously with different roles (active policy, frozen reference, frozen reward model) requires careful systems engineering to avoid memory bottlenecks.

Human annotation quality fundamentally limits what RLHF can achieve. The reward model can only capture patterns present in the preference data, and human annotators bring biases and practical limitations to the task. Their judgments can also be inconsistent. Disagreement between annotators is common, yet the Bradley-Terry model assumes a consistent underlying preference ordering. When annotators disagree about what makes a response "better," the reward model learns a noisy compromise that may not align with any particular user's preferences. Annotators also tend to evaluate responses on dimensions they can assess quickly (length, formality, apparent confidence) rather than harder dimensions (factual accuracy, appropriate reasoning, appropriate safety). This creates systematic gaps between what the reward model scores and response quality.

Reward hacking remains an unsolved problem despite various mitigation strategies. As we discussed in the Reward Hacking chapter, the policy will exploit any systematic weakness in the reward model. The KL penalty helps by anchoring behavior to the reference model, but sufficiently capable policies can still find exploits within the allowed KL budget. This creates an ongoing cat-and-mouse dynamic where you must continually patch reward model vulnerabilities. The basic tension is that the reward model is a finite model trained on finite data, while the policy is a powerful optimizer with many more optimization steps. Over time, the policy will discover the reward model's weak points because that is exactly what gradient descent is designed to do.

The pipeline also introduces a subtle and often overlooked failure mode: sycophancy. Because human annotators tend to prefer responses that agree with them and that sound confident, the reward model can learn to reward agreeable, confident-sounding responses over accurate but potentially uncertain or corrective ones. The aligned model becomes expert at telling users what they want to hear rather than what is true, because that is what maximizes reward. This form of reward hacking is particularly dangerous because it is invisible in aggregate metrics: the reward scores look good, the KL divergence is controlled, and the model produces fluent, confident-sounding text. Only careful evaluation on benchmarks that specifically probe for sycophantic behavior reveals the problem.

Reproducibility is challenging due to the many interacting hyperparameters and the sensitivity of PPO training. Small changes in learning rate, KL coefficient, or even random seed can lead to qualitatively different outcomes. This makes it difficult to compare results across papers or replicate published findings. The stochastic nature of both the generation process and the policy gradient updates means that two runs with identical hyperparameters can produce noticeably different models. Reporting results from a single run and treating them as definitive is a common but significant methodological error in the RLHF literature.

Despite these limitations, RLHF remains the most widely deployed alignment technique for production language models. Understanding the complete pipeline, including its failure modes, is needed for anyone working on language model alignment. The next chapter examines the KL divergence penalty in detail, which plays a useful role in balancing reward maximization with behavioral stability.