DPO Concept › The Key Insight: Reparameterizing the Reward

The key insight behind DPO comes from recognizing a mathematical relationship within the RLHF objective. To understand this insight, we need to look carefully at the RLHF objective and ask whether we can express the same goal in a different, more tractable way.

The insight, stated plainly before we get into the math: any reward function uniquely determines the optimal aligned policy, and conversely, any policy uniquely determines the reward function that would make it optimal. Reward functions and optimal policies are two descriptions of the same thing. Once you realize this, the reward model becomes unnecessary. You can work directly in policy space, using the policy itself as an implicit reward model.

Consider the RLHF objective we have been working with throughout this book:

$$\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(y|x)} \left[ r_\phi(x, y) - \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)} \right]$$

Let's carefully examine each component of this expression to build understanding:

• $\pi_\theta$: the policy model being trained, parameterized by weights $\theta$
• $\mathcal{D}$: the dataset of prompts $x$ that we want the model to respond to well
• $r_\phi(x, y)$: the reward model score for completion $y$ given prompt $x$, representing how much humans would prefer this response
• $\beta$: the KL penalty coefficient that controls how much the policy can deviate from the reference, balancing preference optimization against stability
• $\pi_{\text{ref}}$: the reference model (usually the initial SFT model) used as an anchor to prevent the policy from changing too drastically

This objective captures a basic tradeoff. We want a policy that generates high-reward outputs, meaning outputs that humans would prefer according to our reward model. At the same time, we want to stay close to the reference model. This keeps we don't lose the useful capabilities the model learned during pretraining and supervised fine-tuning. The term $\log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ measures how much the policy has diverged from the reference, and $\beta$ controls how heavily we penalize this divergence. Small $\beta$ allows large deviations; large $\beta$ keeps the policy anchored near its starting point.

Now here is where the key insight emerges. This constrained optimization problem, despite its apparent complexity, has a closed-form solution. Given any reward function $r(x, y)$, we can write down exactly what the optimal policy looks like without doing any iterative optimization:

$$\pi^*(y|x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y|x) \exp\left(\frac{1}{\beta} r(x, y)\right)$$

where:

• $\pi^*(y|x)$: the optimal policy for the given reward function, meaning the policy that maximizes our objective
• $y$: a completion sequence, any possible output the model could generate
• $x$: a prompt sequence, the input the model is responding to
• $Z(x)$: the partition function (normalizing constant) that ensures probabilities sum to 1 over all possible completions for prompt $x$
• $\pi_{\text{ref}}(y|x)$: the reference model distribution, our starting point before reward-based shaping
• $r(x, y)$: the reward function defining what outputs we prefer
• $\beta$: the temperature parameter controlling the sharpness of the distribution

The formula has a beautiful interpretation. The optimal policy takes the reference model and reweights each possible output according to its reward. Outputs with high reward get amplified by the exponential term, while outputs with low reward get suppressed. The partition function $Z(x)$ then normalizes everything so probabilities sum to one. The parameter $\beta$ controls how aggressive this reweighting is: small $\beta$ creates sharp distributions concentrated on high-reward outputs, while large $\beta$ keeps the distribution closer to the original reference. Think of $\beta$ as controlling how strongly you "listen" to the reward signal versus how strongly you "remember" where you started.