DPO Derivation › Reparameterizing the Reward

The next step uses the relationship between the optimal policy and the reward function in reverse. Given an optimal policy, we can solve for the reward function it implies.

This reverse direction might seem like a mathematical curiosity, but it turns out to be the key insight that makes DPO possible. If we can express the reward purely in terms of policies (without needing to train a separate reward model), then we can substitute this expression into the preference model and optimize directly. The reward becomes implicit in the policy rather than explicit in a separate neural network.

Think of it this way: the optimal policy and the reward function are two sides of the same coin. Knowing one completely determines the other, up to an additive constant per prompt. The optimal policy is just a "softmax" over rewards (scaled by $1/\beta$) relative to the reference. Inverting that relationship is like going from probabilities back to log-probabilities: algebraically straightforward once you know the connection exists.

Starting from:

$$\pi^*(y|x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y|x) \exp\left( \frac{r(x, y)}{\beta} \right)$$

where:

• $\pi^*(y|x)$: the optimal policy distribution
• $Z(x)$: the partition function
• $\pi_{\text{ref}}(y|x)$: the reference policy distribution
• $r(x, y)$: the reward function
• $\beta$: the temperature parameter

We can rearrange to isolate the reward. First, take the log of both sides:

$$\log \pi^*(y|x) = \log \pi_{\text{ref}}(y|x) + \frac{r(x, y)}{\beta} - \log Z(x)$$

where:

• $\log \pi^*$: log-probability under the optimal policy
• $\log \pi_{\text{ref}}(y|x)$: log-probability under the reference policy
• $r(x, y)$: the reward function
• $\beta$: the KL penalty coefficient
• $\log Z(x)$: log of the partition function (depends only on $x$)

Taking the logarithm turns our multiplicative relationship into an additive one. This makes algebraic manipulation simpler. The logarithm of a product becomes a sum of logarithms, and the logarithm of the exponential simply returns its argument.

Now, we rearrange terms to solve for $r(x, y)$:

$$\begin{aligned} \frac{r(x, y)}{\beta} &= \log \pi^*(y|x) - \log \pi_{\text{ref}}(y|x) + \log Z(x) && \text{(isolate reward term)} \\ \frac{r(x, y)}{\beta} &= \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \log Z(x) && \text{(combine logs)} \\ r(x, y) &= \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x) && \text{(multiply by } \beta \text{)} \end{aligned}$$

where:

• $r(x, y)$: the implicit reward
• $\beta$: the KL penalty coefficient
• $\pi^*(y|x)$: the optimal policy
• $\pi_{\text{ref}}(y|x)$: the reference policy
• $Z(x)$: the partition function
• $\beta \log (\cdot)$: the scaled log-ratio term

This is the reward reparameterization. It tells us that for any optimal policy $\pi^*$, we can express the implicit reward purely in terms of log probability ratios, plus a prompt-dependent constant $\beta \log Z(x)$.

The log ratio $\log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)}$ has a natural interpretation: it measures how much the optimal policy has increased or decreased the probability of response $y$ compared to the reference. Responses that the optimal policy strongly prefers will have large positive log ratios, while responses it disfavors will have large negative log ratios. This log ratio, scaled by $\beta$, gives us the implicit reward up to an additive constant.

The partition function $Z(x)$ doesn't depend on $y$; it only depends on the prompt $x$. This will turn out to be important because when we compute preference probabilities, this term cancels out.

The reward reparameterization eliminates the need for an explicit reward model. In standard RLHF, you train a separate neural network $r_\phi(x, y)$ on human preference data, and then you use that trained reward model to provide training signal to the language model policy. That two-step pipeline doubles the computational cost, introduces an additional source of approximation error, and creates the reward hacking problem when the policy finds outputs that score well on $r_\phi$ but do not match human preferences.

The reparameterization shows that you never needed a separate reward model. The reward is already encoded in the ratio between two language models: the policy being trained and the reference model. Every time you update the policy's weights, the implicit reward function changes accordingly. The reward model and the policy are the same object, just viewed from different angles. This is the deep reason why DPO is simpler than RLHF: it reduces the engineering work and the number of conceptual entities you have to reason about.