DPO Concept › Looking Ahead

This chapter developed intuition for why DPO works and what makes it attractive as an alignment technique. We saw that the key insight is recognizing the duality between reward functions and optimal policies, a mathematical relationship that allows us to bypass the reward model entirely. The resulting algorithm is simpler, more stable, and more memory-efficient than RLHF, with a training loop that mirrors standard supervised fine-tuning. We also examined the limitations: the distribution shift problem, the loss of reward interpretability, and the challenges of iterative refinement and multi-objective alignment.

In the next chapter, we will derive the DPO objective rigorously, showing exactly how it emerges from the RLHF optimization problem through a careful sequence of algebraic manipulations. We will then implement DPO from scratch, walking through the computation of log-probabilities, the loss function, and the training loop. Finally, we will explore variants of DPO that address some of its limitations, including approaches for handling noisy preferences (Identity Policy Optimization), length-controlled alignment (SimPO), and improving out-of-distribution generalization through iterative preference collection.