DPO Concept › The DPO Training Objective

With this insight, designing a training objective is straightforward. We want to find a policy $\pi_\theta$ that assigns high probability to the preferences in our dataset. In other words, when our dataset says humans prefer $y_w$ over $y_l$, we want our expression $P(y_w \succ y_l | x)$ to be close to 1. When the dataset indicates the opposite preference, we want it to be close to 0.

Think of the DPO objective as teaching a student to reason about quality without giving them an answer key. In RLHF, the reward model is the answer key: it assigns explicit scores that tell the policy exactly how good each response is. In DPO, the student learns by comparing pairs of responses directly. The lesson is not "this response scores 7.3 points" but rather "this response is better than that one." It turns out that pairwise comparisons, if we have enough of them, provide all the information we need to learn the complete preference ordering.

For a dataset of preference pairs $\{(x^{(i)}, y_w^{(i)}, y_l^{(i)})\}_{i=1}^N$, where each example contains a prompt, a preferred (chosen) response, and a dispreferred (rejected) response, we maximize the log-likelihood of the observed preferences:

$$\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right) \right]$$

where:

• $\mathcal{L}_{\text{DPO}}$: the DPO objective function (to be maximized), representing how well the policy matches observed preferences
• $\mathbb{E}$: expectation over the dataset $\mathcal{D}$, meaning we average over all preference pairs
• $x, y_w, y_l$: the prompt, preferred completion, and rejected completion from a preference pair
• $\pi_\theta$: the policy model, whose parameters $\theta$ we are optimizing
• $\pi_{\text{ref}}$: the reference model, kept frozen throughout training
• $\sigma$: the logistic sigmoid function, converting the margin into a probability
• $\beta$: the parameter controlling deviation from the reference model, inherited from the RLHF objective

In practice, we minimize the negative log-likelihood, which is equivalent to maximizing the expression above:

$$\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\left(\beta \left( \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right)\right) \right]$$

The components of this loss function work together in the following way:

• $-\mathbb{E}[\dots]$: minimization of the negative expectation, converting our maximization into a minimization problem for standard optimizers
• $\beta$: the KL penalty coefficient, scaling how much we weight the log-ratio differences
• $\log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$: the log-likelihood ratio of the policy to the reference model, measuring how much the policy has changed for a particular response
• $\sigma$: the sigmoid function applied to the scaled log-ratio difference, converting the margin to a probability between 0 and 1

This is a standard cross-entropy loss, the same loss function used throughout deep learning for binary classification tasks. For each preference pair, we compute four log-probabilities: the chosen response under the policy, the rejected response under the policy, the chosen response under the reference model, and the rejected response under the reference model. We combine these into a single logit representing the preference margin, and apply binary cross-entropy. The training signal is clear and interpretable: increase the probability of preferred responses relative to rejected ones, while staying grounded to the reference model through the log-ratio structure.

We will derive this objective rigorously in the next chapter, showing exactly how it emerges from the RLHF objective through a careful series of mathematical steps. For now, the key intuition is that DPO directly uses preference pairs as training data, with the preference structure built into the loss function itself. We never generate outputs during training, never score outputs with a reward model, and never perform reinforcement learning. We simply run supervised learning on preference pairs with a specially designed loss.

Let's work through a concrete numerical example to solidify the mechanics. Suppose we have trained a reference model and a policy model, and we want to compute the DPO loss for a single preference pair.

Setup: We have a prompt $x$ = "What is the capital of France?" with two responses:

• $y_w$ = "The capital of France is Paris." (chosen)
• $y_l$ = "I think it might be Lyon, but I'm not sure." (rejected)

We use $\beta = 0.5$.

Step 1: Compute log-probabilities under the reference model.

After running both responses through the frozen reference model, we obtain:

• $\log \pi_{\text{ref}}(y_w | x) = -3.2$ (a factually correct response that the reference model assigns moderate probability)
• $\log \pi_{\text{ref}}(y_l | x) = -5.8$ (an uncertain response the reference model is less likely to generate)

Step 2: Compute log-probabilities under the current policy.

After running both responses through the policy model (at a hypothetical early point in training):

• $\log \pi_\theta(y_w | x) = -3.5$ (policy has slightly reduced probability compared to reference; early in training this can happen)
• $\log \pi_\theta(y_l | x) = -5.4$ (policy has slightly increased probability of the rejected response)

Step 3: Compute the log-ratios.

For the chosen response:

$$\log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} = -3.5 - (-3.2) = -0.3$$

The policy has reduced the chosen response's probability relative to the reference. This is a warning sign: the policy should be increasing it, not decreasing it.

For the rejected response:

$$\log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} = -5.4 - (-5.8) = 0.4$$

The policy has increased the rejected response's probability relative to the reference. This is the wrong direction.

Step 4: Compute the preference margin.

$$\text{margin} = \beta \left( \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) = 0.5 \times (-0.3 - 0.4) = 0.5 \times (-0.7) = -0.35$$

The margin is negative, meaning the policy currently "prefers" the rejected response over the chosen one (in a relative sense).

Step 5: Compute the DPO loss.

$$\mathcal{L} = -\log \sigma(-0.35) = -\log\left(\frac{1}{1 + e^{0.35}}\right) = -\log(0.413) \approx 0.884$$

Step 6: Interpret the result.

A loss of 0.884 is relatively high (perfect alignment would give a loss near 0). The gradient of this loss will push the policy to:

• Increase $\pi_\theta(y_w|x)$ relative to $\pi_{\text{ref}}(y_w|x)$ (make the chosen response more likely)
• Decrease $\pi_\theta(y_l|x)$ relative to $\pi_{\text{ref}}(y_l|x)$ (make the rejected response less likely)

After several gradient steps, the log-ratios should reverse: the chosen response's log-ratio should become positive, the rejected response's log-ratio should become negative, and the margin should become positive, driving the loss toward zero.

This example illustrates an important property: the reference model acts as a calibration anchor. We are not simply trying to make $y_w$ the highest-probability response in absolute terms. We are trying to make it higher-probability relative to where it started, while keeping $y_l$ lower than where it started. This relative framing is what prevents the aligned model from collapsing to a narrow distribution that always produces a single canned response.