DPO Concept › Comparing DPO to RLHF

Let's contrast the training procedures concretely, walking through each step to see exactly where the complexity reduction occurs.

The RLHF training loop discussed previously proceeds as follows. First, we sample prompts from the dataset. Second, we generate full responses using the current policy through expensive autoregressive sampling. Third, we score each response with the reward model, running another forward pass through a large neural network. Fourth, we compute advantages using the value function, which requires a separate forward pass through yet another network and involves complex bootstrap estimation. Fifth, we update the policy using PPO, including the clipping mechanism and importance sampling corrections. Sixth, we update the value function. Seventh, we monitor KL divergence and adjust the penalty coefficient if the policy is drifting too far from the reference. This entire loop repeats thousands of times, with each iteration requiring multiple forward and backward passes through multiple large models.

The DPO training loop is dramatically simpler. First, we sample preference pairs from the dataset. Second, we compute log-probabilities under both the policy and the reference model, running two forward passes through two models for each preference pair. Third, we compute the DPO loss using the formula above. Fourth, we update the policy with standard gradient descent using AdamW or a similar optimizer. That is the entire loop. No sampling, no reward model, no value function, no PPO clipping, no advantage estimation.

Complexity is significantly reduced in every dimension. DPO eliminates online generation, removing the need to sample from the policy during training. It eliminates the reward model, removing an entire neural network and its associated training. It eliminates the value function, removing another neural network. It eliminates PPO's clipping mechanism, replacing reinforcement learning with simple gradient descent. And it eliminates the adaptive KL penalty, since the constraint is built into the loss function itself. What remains is essentially supervised fine-tuning with a specially designed loss function.

The following visualization illustrates this architectural difference. In the RLHF pipeline, information flows through multiple models in a complex loop: prompts go to the policy, which generates responses, which go to the reward model, which produces scores, which combine with value estimates to produce advantages, which finally update the policy. In DPO, the flow is much simpler: preference pairs are fed directly to the policy and reference model, their probabilities are combined into a loss, and the policy is updated.