DPO Implementation › Worked Example: Manual DPO Loss Computation

Before looking at the full training loop, let us trace through a single DPO loss computation by hand to make every step concrete. Understanding the numerical path from raw model outputs to a loss value is invaluable when debugging.

Suppose we have a batch of one preference pair with the following hypothetical log probabilities after tokenization and masking:

• Policy model log probability of chosen response: $\log \pi_\theta(y_w | x) = -12.4$
• Policy model log probability of rejected response: $\log \pi_\theta(y_l | x) = -15.7$
• Reference model log probability of chosen response: $\log \pi_\text{ref}(y_w | x) = -13.1$
• Reference model log probability of rejected response: $\log \pi_\text{ref}(y_l | x) = -15.3$

These numbers represent plausible values for short responses from a GPT-2 scale model. The policy has made the chosen response somewhat more likely than the reference did, while the reference considered the rejected response slightly more likely than the policy does.

Step 1: Compute the log ratios (implicit rewards before scaling)

$$\begin{aligned} r_w &= \log \pi_\theta(y_w | x) - \log \pi_\text{ref}(y_w | x) \\ &= -12.4 - (-13.1) \\ &= +0.7 \end{aligned}$$ $$\begin{aligned} r_l &= \log \pi_\theta(y_l | x) - \log \pi_\text{ref}(y_l | x) \\ &= -15.7 - (-15.3) \\ &= -0.4 \end{aligned}$$

The policy has increased its probability for the chosen response (positive ratio) and decreased its probability for the rejected response (negative ratio) relative to the reference. This is the desired behavior: the model has started learning the preference.

Step 2: Scale by $\beta$ to get implicit rewards

With $\beta = 0.1$:

$$\begin{aligned} \hat{r}_w &= \beta \cdot r_w = 0.1 \times 0.7 = +0.07 \\ \hat{r}_l &= \beta \cdot r_l = 0.1 \times (-0.4) = -0.04 \end{aligned}$$

Step 3: Compute the reward margin

$$\Delta = \hat{r}_w - \hat{r}_l = 0.07 - (-0.04) = +0.11$$

A positive reward margin means the model correctly prefers the chosen response. The magnitude of 0.11 is modest, showing the model has made some progress but has not strongly internalized the preference yet.

Step 4: Apply the sigmoid and compute the loss

$$\begin{aligned} \sigma(\Delta) &= \sigma(0.11) = \frac{1}{1 + e^{-0.11}} \approx \frac{1}{1 + 0.896} \approx 0.527 \\ \mathcal{L} &= -\log(0.527) \approx 0.640 \end{aligned}$$

A loss of 0.640 makes sense: we are closer to the maximum loss at $\Delta = 0$ (which gives loss $= \log 2 \approx 0.693$) than to near-zero loss, because the reward margin is still small. As training progresses and the margin grows, say to 2.0, the loss would drop to approximately $-\log(\sigma(2.0)) \approx 0.127$.

Step 5: Interpret the gradient direction

The gradient of the loss with respect to the reward margin is $-\sigma(-\Delta) = -(1 - \sigma(\Delta)) \approx -(1 - 0.527) = -0.473$. This negative gradient means the optimizer will increase the reward margin, which happens by either increasing the chosen log ratio, decreasing the rejected log ratio, or both. In terms of model weights, the update nudges the model toward assigning higher probability to the chosen response and lower probability to the rejected response relative to the reference.

This worked example reveals that DPO training is gradual and incremental. Each update makes a small change to the model's implicit preferences, and over many batches these changes accumulate into a policy that reflects the preference data. The key insight is that no single preference pair causes a large jump in model behavior: the regularization effect of $\beta$ keeps each update modest, and alignment emerges from the accumulation of thousands of such modest updates.