DPO Derivation › DPO as Classification

The DPO loss has a revealing interpretation: it's essentially binary classification with a specific parameterization. Consider a standard binary cross-entropy loss for classifying which of two responses is preferred:

$$\mathcal{L}_{\text{BCE}} = -\mathbb{E} \left[ \log \sigma(f(x, y_w, y_l)) \right]$$

where:

• $\mathcal{L}_{\text{BCE}}$: Binary Cross-Entropy loss
• $f(x, y_w, y_l)$: a scoring function showing preference strength
• $\sigma$: the sigmoid function
• $\mathbb{E}$: expectation over the dataset

In this context, $f$ is some function that should output a positive value when $y_w$ is truly preferred. In DPO, this function is:

$$f(x, y_w, y_l) = \beta \left[ \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right]$$

where:

• $f(x, y_w, y_l)$: the logit fed into the sigmoid classifier
• $\beta$: the KL penalty coefficient
• $\pi_\theta$: the policy network
• $\pi_{\text{ref}}$: the reference policy

The model is learning to classify preference pairs by adjusting its output probabilities. When the loss is low, the model assigns higher relative probability to the preferred response.

This classification view explains why DPO is so stable compared to RLHF. Instead of learning a separate reward model and then using policy gradients with their high variance, DPO directly optimizes the likelihood of preference labels. The gradients flow directly through the model's log probabilities, just like in standard language model training.

The gradient of the DPO loss with respect to $\theta$ has a particularly intuitive form. For a single example:

$$\nabla_\theta \mathcal{L}_{\text{DPO}} = -\beta \sigma(-\hat{r}_\theta(x, y_w) + \hat{r}_\theta(x, y_l)) \left[ \nabla_\theta \log \pi_\theta(y_w|x) - \nabla_\theta \log \pi_\theta(y_l|x) \right]$$

where:

• $\nabla_\theta$: gradient with respect to model parameters $\theta$
• $\mathcal{L}_{\text{DPO}}$: the DPO loss
• $\sigma$: the sigmoid function
• $\hat{r}_\theta$: the implicit reward function
• $\sigma(\dots)$: the weighting term derived from the sigmoid derivative (probability of the wrong preference)
• $\nabla_\theta \log \dots$: the geometric gradient direction
• $\beta$: the KL penalty coefficient

This gradient expression reveals the geometry of DPO optimization. The term $\nabla_\theta \log \pi_\theta(y_w|x) - \nabla_\theta \log \pi_\theta(y_l|x)$ points in the direction that increases the probability of the winning response while decreasing the probability of the losing response. This is the natural direction to push given our objective.

The term $\sigma(-\hat{r}_\theta(x, y_w) + \hat{r}_\theta(x, y_l))$ is the probability that the model currently assigns to the wrong preference. This acts as an implicit weighting:

• When the model is confident and correct (low wrong-preference probability), the gradient is small
• When the model is confident but wrong, the gradient is large
• When the model is uncertain, the gradient is moderate

This automatic weighting helps the model focus on examples it's getting wrong, similar to how hard negative mining works in contrastive learning.