DPO Derivation › Summary

This chapter derived the DPO loss function from first principles, showing how it emerges naturally from the RLHF objective.

The key steps in the derivation were:

1. RLHF objective: Maximize expected reward with a KL penalty to stay close to the reference policy
2. Optimal policy: Using Lagrange multipliers, we found that the optimal policy is a Boltzmann distribution that reweights the reference by exponentiated rewards
3. Reward reparameterization: Inverting this relationship, we expressed the implicit reward as a log-ratio between the optimal and reference policies, plus a partition function
4. Bradley-Terry substitution: Plugging the reparameterized rewards into the preference model, the partition functions cancel, leaving a loss in terms of log-ratios only
5. Classification view: The resulting DPO loss is binary cross-entropy, learning to classify which response is preferred based on relative log-probabilities

The gradient of the DPO loss has an automatic weighting scheme: examples where the model is confidently wrong receive larger gradients, while examples where it's already correct receive smaller gradients. This makes DPO training stable and efficient.

The derivation also reveals DPO's key assumptions: that human preferences follow the Bradley-Terry model, that the KL-regularized RLHF objective is the right starting point, and that the policy class is expressive enough to represent the optimal Boltzmann distribution. When those assumptions hold, DPO provides a principled, efficient alternative to RLHF. When they break down, the variants we explore in the following chapter offer modified objectives that relax different aspects of those assumptions.

In the next chapter, we'll implement DPO end-to-end, covering practical details like computing sequence log-probabilities, handling padding, and integrating with Hugging Face's training infrastructure.