DPO Concept › Summary

Direct Preference Optimization offers a fundamentally different approach to alignment by recognizing that we do not need to learn an explicit reward function. The optimal policy for any reward function can be computed in closed form, and this relationship allows us to express preference learning as a direct supervised learning problem on preference pairs.

The core insights are:

• Reward-policy duality: Any reward function corresponds to an optimal policy, and vice versa. The implicit reward of a policy is the log-probability ratio between that policy and a reference model, scaled by $\beta$.

• Partition function cancellation: When we compute preference probabilities using the policy-based reward, the intractable partition function $Z(x)$ cancels out identically, making the expression tractable.

• Eliminating the reward model: By substituting the policy-based reward into the Bradley-Terry preference model, we can predict preferences using only language model log-probabilities, without a separate reward model.

• Supervised learning objective: DPO minimizes a binary cross-entropy loss on preference pairs, pushing the model to assign higher probability to preferred responses and lower probability to rejected ones, relative to the reference model.

• Built-in regularization: The reference model appears directly in the loss through the log-ratio terms. This keeps the trained policy stays close to the original model without requiring explicit KL computation during training.

• Adaptive gradient weighting: The DPO gradient is naturally weighted by the model's current uncertainty, focusing updates on preference pairs where the model is most wrong and giving smaller gradients for pairs the model already handles correctly.

The practical benefits include dramatically simpler training, reduced memory requirements, elimination of online generation, and more stable convergence. The limitations include offline distribution shift, loss of reward interpretability, and challenges with iterative refinement and complex multi-objective alignment scenarios. These advantages and disadvantages have made DPO a popular alternative to RLHF for many practical alignment tasks, while keeping RLHF relevant for scenarios that require online learning or explicit reward interpretability.