DPO Derivation › The DPO Loss Function

Now we can derive the DPO loss by substituting our reward reparameterization into the Bradley-Terry preference model. Recall from the chapter on the Bradley-Terry Model that the probability of preferring response $y_w$ over response $y_l$ is:

$$p(y_w \succ y_l | x) = \sigma\left( r(x, y_w) - r(x, y_l) \right)$$

where:

• $p(y_w \succ y_l | x)$: probability that response $y_w$ is preferred over $y_l$
• $\sigma(z) = \frac{1}{1 + e^{-z}}$: the sigmoid function mapping values to $(0, 1)$
• $r(x, y_w)$: reward for the winning response
• $r(x, y_l)$: reward for the losing response

The Bradley-Terry model is elegant in its simplicity: the probability of preferring one response over another depends only on the difference in their rewards, passed through a sigmoid function. This means responses with much higher reward are strongly preferred, while responses with similar rewards have preference probabilities close to 0.5.

Substituting our reparameterized reward:

$$p(y_w \succ y_l | x) = \sigma\left( \left[ \beta \log \frac{\pi^*(y_w|x)}{\pi_{\text{ref}}(y_w|x)} + \beta \log Z(x) \right] - \left[ \beta \log \frac{\pi^*(y_l|x)}{\pi_{\text{ref}}(y_l|x)} + \beta \log Z(x) \right] \right)$$

where:

• $\sigma$: the sigmoid function
• $\beta$: the temperature parameter
• $\pi^*$: the optimal policy
• $\pi_{\text{ref}}$: the reference policy
• $Z(x)$: the partition function terms which appear with opposite signs

Notice that the $\beta \log Z(x)$ terms cancel! This cancellation is not a coincidence; it's a direct consequence of using reward differences in the Bradley-Terry model. The partition function contributes equally to both rewards, so when we subtract them, it disappears. This cancellation is needed for the practical success of DPO because computing $Z(x)$ would require summing over all possible responses, which is computationally intractable for language models with exponentially large output spaces.

The key insight is that this cancellation is guaranteed by the structure of the Bradley-Terry model. Because preferences are defined as comparisons between pairs of responses (rather than absolute scores), any term that depends only on the prompt $x$ and not on the response $y$ will cancel in the subtraction. The partition function is exactly such a term, depending on $x$ through the sum over all possible responses but not on any particular response $y_w$ or $y_l$. This is why the DPO derivation only works cleanly with pairwise comparison preference models and would need modification for other preference structures.

This leaves:

$$p(y_w \succ y_l | x) = \sigma\left( \beta \log \frac{\pi^*(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi^*(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right)$$

where:

• $\sigma$: the sigmoid function
• $\beta$: temperature parameter
• $\pi^*$: the optimal policy
• $\pi_{\text{ref}}$: the reference policy

We can simplify this using logarithm properties:

$$p(y_w \succ y_l | x) = \sigma\left( \beta \left[ \log \frac{\pi^*(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi^*(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right] \right)$$

where:

• $\sigma$: the sigmoid function
• $\beta$: the KL penalty coefficient
• $\log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)}$: the log-likelihood ratio of the optimal policy vs reference

This expression is remarkably clean. The probability of the correct preference depends entirely on the difference between two log ratios: how much the optimal policy prefers the winning response (relative to the reference) versus how much it prefers the losing response. When this difference is large and positive, the sigmoid outputs a value close to 1, showing strong confidence in the correct preference.

This is the probability that the optimal policy assigns to the correct preference. To train a policy $\pi_\theta$ to match this optimal policy, we maximize the log-likelihood of the observed preferences:

$$\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\left( \beta \left[ \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right] \right) \right]$$

where:

• $\mathcal{L}_{\text{DPO}}$: the DPO loss function
• $\pi_\theta$: the policy network being trained (parameterized by $\theta$)
• $\pi_{\text{ref}}$: the reference policy
• $\mathcal{D}$: the dataset of preference pairs $(x, y_w, y_l)$
• $\mathbb{E}$: expectation over the dataset
• $\beta$: the KL penalty coefficient
• $\sigma$: the sigmoid function

The negative sign appears because we're minimizing a loss rather than maximizing likelihood.

To make the notation more compact, define the log-ratio for a response $y$:

$$\hat{r}_\theta(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$$

where:

• $\hat{r}_\theta(x, y)$: the implicit reward assigned by the current model $\pi_\theta$
• $\beta$: the KL penalty coefficient scaling the reward
• $\pi_\theta(y|x)$: the policy probability
• $\pi_{\text{ref}}(y|x)$: the reference policy probability
• $\log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$: the log-ratio of the policy probability to the reference probability

This is sometimes called the "implicit reward" because it's what the reward would be if $\pi_\theta$ were optimal. The terminology is fitting: we never explicitly compute or learn a reward function, yet the policy implicitly defines one through its log-probability ratios with the reference model.

The DPO loss becomes:

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\left( \hat{r}_\theta(x, y_w) - \hat{r}_\theta(x, y_l) \right) \right]$$

where:

• $\mathcal{L}_{\text{DPO}}$: the DPO loss function
• $\hat{r}_\theta(x, y)$: the implicit reward function
• $\hat{r}_\theta(x, y_w) - \hat{r}_\theta(x, y_l)$: the margin between implicit rewards for winning and losing responses
• $\sigma$: the sigmoid function
• $\mathbb{E}$: expectation over the dataset

This is remarkably simple. The loss encourages the model to increase its log probability (relative to the reference) on preferred responses and decrease it on dispreferred responses.