DPO Derivation › DPO Derivation

In the previous chapter, we introduced Direct Preference Optimization as a simpler alternative to RLHF that eliminates the need for an explicit reward model. We saw that DPO directly optimizes a language model using preference data, but we didn't examine why this works or how the DPO loss function is derived. This chapter fills that gap by tracing every algebraic step, from the original reinforcement learning objective all the way to the final training loss.

The DPO derivation is a key result in alignment research. It shows that the optimal policy for the RLHF objective has a closed-form solution, and this solution can be rearranged to express the implicit reward in terms of the policy itself. When we substitute this reparameterized reward into the Bradley-Terry preference model, we obtain the DPO loss function directly, with no reinforcement learning required. This chain of reasoning is surprisingly elegant, and understanding it will fundamentally change how you think about what a language model is doing when it trains on human preference data.

Understanding this derivation reveals why DPO works, its underlying assumptions, and how to interpret the model's learning process. By the end of this chapter, you'll see that DPO is essentially solving a classification problem where the model learns to assign higher probability to preferred responses. Each gradient step simultaneously rewards the model for raising the probability of good outputs and penalizes it for raising the probability of bad ones, all without ever explicitly parameterizing a reward function.

Think of the whole derivation as a three-act story. Act one asks: if we had the ideal reward function, what would the optimal policy look like? Act two reverses that question and asks: if we have the optimal policy, what reward function does it imply? Act three substitutes that reversed reward into the preference model and discovers that the messy partition function term vanishes, leaving a clean, tractable loss. Each act is mathematically precise, and each act builds directly on the previous one. When you finish reading, the chain from RLHF to DPO will feel inevitable rather than mysterious.

One reason this derivation matters beyond academic interest is that it reveals the hidden assumptions baked into DPO. The loss function is not a heuristic: it follows from specific, clearly stated premises about the form of the RLHF objective, the Boltzmann structure of optimal policies, and the Bradley-Terry preference model. Knowing those premises tells you exactly when DPO is appropriate, what might break it, and how to modify it for different settings. We will examine those assumptions carefully in the section on limitations.