Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

DPO Concept › From Reward Models to Preference Models

Michael BrenndoerferDecember 31, 20255 min read

Language AI HandbookAll chapters

Part 37 of 67Alignment and RLHF

Chapter 282DPO Concept › From Reward Models to Preference Models

Reading section 4 of 14 · From Reward Models to Preference Models · 5 minSection 4 of 14 · 5 min

Section 4 of 14 · 5 min read

DPO Concept › From Reward Models to Preference ModelsOne section · Sections in this chapter4 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

With this reparameterization in hand, we can reconsider how preference learning works and see whether we can eliminate the reward model entirely. The key question is: can we express preference predictions using only policy probabilities, without needing an explicit reward function?

Recall from our discussion of the Bradley-Terry model that we model the probability of preferring response $y_w$ over $y_l$ as:

P(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l))

where:

$P(y_w \succ y_l | x)$ : the probability that response $y_w$ is preferred over $y_l$ given prompt $x$
$\sigma$ : the sigmoid function, $\sigma(z) = \frac{1}{1+e^{-z}}$ , which maps any real number to a probability between 0 and 1
$r(x, y)$ : the scalar reward value for a response, representing its quality

The reward model's job in traditional RLHF is to learn $r(x, y)$ such that this probability matches human preferences. When humans prefer $y_w$ over $y_l$ , the reward model should assign higher reward to $y_w$ , making the sigmoid output close to 1. When humans prefer $y_l$ , the reward model should assign higher reward to $y_l$ , making the sigmoid output close to 0.

Out[4]:

Visualization

Sigmoid curve mapping reward differences to preference probabilities, with green shading where the model correctly prefers the chosen response and red shading where it incorrectly prefers the rejected response. — The Bradley-Terry model converts reward differences into preference probabilities via the sigmoid function. Positive reward differences (chosen response beats rejected) map to probabilities above 0.5 (green region), while negative differences map to probabilities below 0.5 (red region). The sigmoid's smooth S-shape means that even modest reward margins produce meaningful preference signals.

Substituting the Policy-Based RewardLink Copied

Now let's substitute our reparameterization of the reward in terms of the policy. We replace $r(x, y)$ with $\beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)$ in both the chosen and rejected reward terms. For the chosen response:

r(x, y_w) = \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} + \beta \log Z(x)

For the rejected response:

r(x, y_l) = \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} + \beta \log Z(x)

Taking the difference, the $\beta \log Z(x)$ terms cancel identically:

\begin{aligned} r(x, y_w) - r(x, y_l) &= \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} + \beta \log Z(x) \\ &\quad - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} - \beta \log Z(x) \\ &= \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \end{aligned}

This cancellation is important because $Z(x)$ is intractable to compute for language models. Computing it would require summing over all possible sequences the model could generate, an astronomical number for any realistic vocabulary and sequence length. The fact that $Z(x)$ cancels means we never need to compute it.

Substituting back into the Bradley-Terry formula, we obtain:

P(y_w \succ y_l | x) = \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)

where:

$P(y_w \succ y_l | x)$ : the probability that the policy implicitly assigns to preferring $y_w$ over $y_l$
$\sigma$ : the sigmoid function that maps the scaled log-ratio difference to a probability
$\beta$ : the KL coefficient inherited from the RLHF objective
$\pi_\theta(y_w|x)$ : the policy's probability of generating the chosen response
$\pi_{\text{ref}}(y_w|x)$ : the reference model's probability of generating the chosen response
$\pi_\theta(y_l|x)$ : the policy's probability of generating the rejected response
$\pi_{\text{ref}}(y_l|x)$ : the reference model's probability of generating the rejected response

The reward model has vanished entirely from this expression. We can now directly compute the probability of a preference using only language model probabilities. Both the policy and reference model are language models that can compute $\log \pi(y|x)$ for any sequence $y$ and prompt $x$ . This is a standard operation: we run the sequence through the model and sum up the log probabilities of each token given its predecessors. No separate reward model is needed.

This observation is the core theoretical insight behind DPO. Preference probabilities can be expressed purely in terms of policy and reference model probabilities, without any explicit reward function. If we want to train a model to match human preferences, we can directly optimize the policy to make this expression match the observed preferences in our dataset.