DPO Concept › The RLHF Complexity Problem

As we discussed in the RLHF Pipeline chapter, aligning a language model with human preferences involves a multi-stage process. First, we train a reward model on preference data, teaching a neural network to predict which responses humans prefer. Then we use that reward model to guide policy optimization through PPO, while constraining the policy to stay close to the original model via a KL divergence penalty. This pipeline, while effective, introduces several sources of complexity that create both engineering challenges and potential failure modes.

Think of the RLHF training loop as a four-person assembly line where each person's output feeds the next, and a mistake by any one of them cascades forward. The policy generates a response, the reward model grades it, the value function estimates how good this step was, and finally PPO decides how to update the policy. Each handoff introduces noise. Each model has its own imperfections. And unlike a simple supervised learning pipeline, this one runs in a loop: the policy's outputs change the distribution of inputs the reward model sees, which changes the reward signal, which changes the policy's outputs. Keeping this feedback loop stable requires careful engineering.

The first challenge is managing multiple interacting models. During RLHF training, we must maintain four separate neural networks, each with its own role in the training process:

• The policy model $\pi_\theta$ being trained, which generates responses and receives gradient updates
• The reward model $r_\phi$ scoring generations, which provides the signal for what constitutes good behavior
• The reference model $\pi_{\text{ref}}$ giving the KL anchor, which prevents the policy from drifting too far from its starting point
• The value function $V_\psi$ for variance reduction in PPO, which estimates expected returns to make gradient estimates more stable

Each model requires its own memory allocation, and for large language models, this memory burden is enormous. If your policy model takes 80 GB of GPU memory, you need roughly 320 GB just to hold all four models, before accounting for optimizer states, activations, and gradients. The interactions between these components can cause instability. The reward model's imperfections can lead to reward hacking, as we saw previously, where the policy finds outputs that score well but don't satisfy human preferences. The value function introduces its own approximation errors, potentially leading to high-variance gradient estimates. And coordinating updates across all these components requires careful hyperparameter tuning, with different learning rates, update frequencies, and regularization strategies for each model.

The second challenge is the computational overhead of online generation. PPO requires the policy to generate completions during training, creating full responses through autoregressive sampling. It then scores those completions with the reward model, computes advantages, and updates the policy. This generation step is expensive, particularly for large language models, where creating a single response might require hundreds of forward passes through the model. Beyond the raw computational cost, online generation introduces the additional complexity of managing sampling strategies during training. Should you use temperature sampling? Nucleus sampling? How do you balance exploration (trying diverse outputs) with exploitation (refining outputs the model already produces well)? Each choice affects training dynamics, and the wrong choice can cause training to collapse or oscillate.

The third challenge is the instability inherent in policy gradient methods. Despite PPO's improvements over vanilla policy gradients, including clipping mechanisms and advantage normalization, RLHF training remains sensitive to hyperparameters like learning rate, clipping thresholds, and the KL penalty coefficient $\beta$. Training runs can diverge, with loss values exploding and model outputs becoming incoherent. They can collapse to repetitive outputs, where the model finds a single response template that reliably earns moderate reward and refuses to explore alternatives. Or they can find reward-hacking solutions that score well according to the reward model but produce poor actual outputs when evaluated by humans. The key point here is that each failure mode requires a different diagnostic approach and a different fix, and sometimes the fixes interfere with each other.

Researchers questioned whether the reinforcement learning framework was necessary, or whether a more direct path existed from preferences to an aligned model. Could we somehow skip the intermediate reward model and the complex RL optimization, directly using preference data to update the language model? The answer to this question led to DPO, and the path to that answer runs through some elegant mathematics.