DPO Derivation › Worked Example

Let's trace through the derivation with concrete numbers to build intuition. Suppose we have a prompt $x$ = "Write a haiku about autumn" and two responses:

• $y_w$ (preferred): "Crimson leaves descend / Dancing on October wind / Earth prepares to sleep"
• $y_l$ (dispreferred): "Leaves fall down in fall / The weather gets cold outside / I like pumpkin spice"

The preferred response demonstrates poetic craft through concrete imagery ("crimson," "dancing"), sensory detail, and thematic resonance with the natural world. The dispreferred response is grammatically correct but prosaic, repeating "fall" clumsily and ending on an unrelated tangent about pumpkin spice. A human evaluator would clearly prefer the first response, and a well-trained reward model should assign it a higher score.

Assume our reference model $\pi_{\text{ref}}$ assigns:

• $\log \pi_{\text{ref}}(y_w|x) = -45.2$ (log probability of generating the preferred response)
• $\log \pi_{\text{ref}}(y_l|x) = -38.7$ (log probability of generating the dispreferred response)

The reference model assigns higher probability to the dispreferred response because it's simpler and uses more common words. The word "fall" is far more common in training data than "crimson" or "descend," and the dispreferred response is shorter and uses everyday vocabulary throughout. This is exactly the distribution shift DPO is designed to fix: the reference model, trained to predict natural text, has no built-in preference for poetic quality over simplicity.

Now suppose our current policy $\pi_\theta$ assigns:

• $\log \pi_\theta(y_w|x) = -42.1$
• $\log \pi_\theta(y_l|x) = -39.5$

With $\beta = 0.1$, the implicit rewards are:

$$\begin{aligned} \hat{r}_\theta(x, y_w) &= 0.1 \times (-42.1 - (-45.2)) \\ &= 0.1 \times 3.1 \\ &= 0.31 \end{aligned}$$ $$\begin{aligned} \hat{r}_\theta(x, y_l) &= 0.1 \times (-39.5 - (-38.7)) \\ &= 0.1 \times (-0.8) \\ &= -0.08 \end{aligned}$$

The difference is:

$$\begin{aligned} \hat{r}_\theta(x, y_w) - \hat{r}_\theta(x, y_l) &= 0.31 - (-0.08) \\ &= 0.39 \end{aligned}$$

The loss contribution from this example:

$$\begin{aligned} -\log \sigma(0.39) &= -\log(0.596) \\ &= 0.517 \end{aligned}$$

The model is doing okay on this example (probability of correct preference is about 60%), but there's room to improve. The gradient will push the model to further increase $\pi_\theta(y_w|x)$ relative to the reference and decrease $\pi_\theta(y_l|x)$ relative to the reference.

Let's also trace through what a "good" and a "bad" model state look like for this same example. Suppose after more training the policy reaches:

• $\log \pi_\theta(y_w|x) = -38.0$ (probability has increased from reference)
• $\log \pi_\theta(y_l|x) = -41.5$ (probability has decreased from reference)

Now the implicit rewards are:

$$\begin{aligned} \hat{r}_\theta(x, y_w) &= 0.1 \times (-38.0 - (-45.2)) = 0.1 \times 7.2 = 0.72 \\ \hat{r}_\theta(x, y_l) &= 0.1 \times (-41.5 - (-38.7)) = 0.1 \times (-2.8) = -0.28 \end{aligned}$$

The reward margin is now $0.72 - (-0.28) = 1.00$, and the preference probability is $\sigma(1.00) \approx 0.73$. The loss drops to $-\log(0.73) \approx 0.31$. The model has learned to "see" the quality difference between these two responses, even though it never directly observed a reward signal during this training phase.