DPO Variants › Worked Example: Computing Losses Step by Step

To make the differences between these methods concrete, let's walk through a specific numerical example using a single preference pair. This will illustrate how each loss function responds to the same underlying data and why their behaviors diverge.

Suppose we are training a language model to answer questions more helpfully. For the prompt "What is the boiling point of water?" we have two candidate responses:

• $y_w$ (chosen): "Water boils at 100 degrees Celsius (212 degrees Fahrenheit) at standard atmospheric pressure."
• $y_l$ (rejected): "Water boils at a really high temperature."

Our reference model assigns log-probabilities:

• $\log \pi_{\text{ref}}(y_w|x) = -3.2$ (moderately likely under the base model)
• $\log \pi_{\text{ref}}(y_l|x) = -2.8$ (also reasonably likely, since it is short and plausible)

After some training steps, our current policy assigns:

• $\log \pi_\theta(y_w|x) = -2.4$ (the policy has learned to prefer the specific response)
• $\log \pi_\theta(y_l|x) = -4.1$ (the policy has learned to penalize the vague response)

Let's set $\beta = 0.1$.

Step 1: Compute the implicit rewards.

The implicit reward for each response is the log-ratio relative to the reference:

$$\begin{aligned} r_\theta(x, y_w) &= \log \pi_\theta(y_w|x) - \log \pi_{\text{ref}}(y_w|x) \\ &= -2.4 - (-3.2) = 0.8 \end{aligned}$$ $$\begin{aligned} r_\theta(x, y_l) &= \log \pi_\theta(y_l|x) - \log \pi_{\text{ref}}(y_l|x) \\ &= -4.1 - (-2.8) = -1.3 \end{aligned}$$

The chosen response has a positive implicit reward (the policy has moved toward it relative to the reference), while the rejected response has a negative implicit reward (the policy has moved away from it).

Step 2: Compute $h_\theta$.

The log-ratio difference is:

$$h_\theta = r_\theta(x, y_w) - r_\theta(x, y_l) = 0.8 - (-1.3) = 2.1$$

Step 3: Compute the DPO loss.

$$\mathcal{L}_{\text{DPO}} = -\log \sigma(\beta \cdot h_\theta) = -\log \sigma(0.1 \times 2.1) = -\log \sigma(0.21)$$

$\sigma(0.21) = \frac{1}{1 + e^{-0.21}} \approx \frac{1}{1 + 0.811} \approx 0.552$

$\mathcal{L}_{\text{DPO}} = -\log(0.552) \approx 0.594$

This loss is still substantial. DPO would continue pushing $h_\theta$ higher, even though the model already clearly prefers the chosen response.

Step 4: Compute the IPO loss with $\beta = 0.1$, giving target $\frac{1}{2\beta} = 5.0$.

$$\mathcal{L}_{\text{IPO}} = (h_\theta - 5.0)^2 = (2.1 - 5.0)^2 = (-2.9)^2 = 8.41$$

IPO reports a large loss because the current $h_\theta = 2.1$ is far below the target of 5.0. The gradient is:

$$\frac{\partial \mathcal{L}_{\text{IPO}}}{\partial h_\theta} = 2(2.1 - 5.0) = -5.8$$

This gradient is negative, meaning IPO is pushing $h_\theta$ upward toward the target, just as DPO does in this case. If instead $h_\theta = 8.0$, the IPO gradient would be $2(8.0 - 5.0) = +6.0$, pushing $h_\theta$ back down. DPO would never push downward.

Step 5: Compute the cDPO loss with $\epsilon = 0.2$.

$$\begin{aligned} \mathcal{L}_{\text{cDPO}} &= -(1 - 0.2) \log \sigma(0.21) - 0.2 \log \sigma(-0.21) \\ &= -0.8 \log(0.552) - 0.2 \log(0.448) \\ &= -0.8 \times (-0.594) - 0.2 \times (-0.802) \\ &\approx 0.475 + 0.160 = 0.635 \end{aligned}$$

The cDPO loss is slightly higher than DPO here (0.635 vs. 0.594). More importantly, the equilibrium for cDPO with $\epsilon = 0.2$ and $\beta = 0.1$ is:

$$h_\theta^* = \frac{\log((1 - 0.2)/0.2)}{0.1} = \frac{\log(4)}{0.1} \approx \frac{1.386}{0.1} = 13.86$$

This is a finite target, far smaller than DPO's implicit infinite target.

Step 6: Compute the KTO value for the chosen response, assuming a reference point $z_0 = 0.05$.

$$v(x, y_w) = \sigma(\beta \cdot r_\theta(x, y_w) - z_0) = \sigma(0.1 \times 0.8 - 0.05) = \sigma(0.03) \approx 0.507$$

The KTO loss contribution for this desirable example (with $\lambda_d = 1.0$) is:

$$\lambda_d \cdot (1 - v) = 1.0 \times (1 - 0.507) = 0.493$$

Since the implicit reward just barely exceeds the reference point, the model has done only slightly better than average for this example. The loss is close to 0.5, the maximum possible, suggesting the model should continue improving this response.

This worked example illustrates a key pattern: DPO, IPO, and cDPO all respond to the same $h_\theta$ value but with different equilibria, while KTO operates on individual implicit rewards rather than their difference.