DPO Derivation: From RLHF Objective to Direct Optimization

In the previous chapter, we introduced Direct Preference Optimization as a simpler alternative to RLHF that eliminates the need for an explicit reward model. We saw that DPO directly optimizes a language model using preference data, but we didn't examine why this works or how the DPO loss function is derived. This chapter fills that gap by tracing every algebraic step, from the original reinforcement learning objective all the way to the final training loss.

The DPO derivation is a key result in alignment research. It shows that the optimal policy for the RLHF objective has a closed-form solution, and this solution can be rearranged to express the implicit reward in terms of the policy itself. When we substitute this reparameterized reward into the Bradley-Terry preference model, we obtain the DPO loss function directly, with no reinforcement learning required. This chain of reasoning is surprisingly elegant, and understanding it will fundamentally change how you think about what a language model is doing when it trains on human preference data.

Understanding this derivation reveals why DPO works, its underlying assumptions, and how to interpret the model's learning process. By the end of this chapter, you'll see that DPO is essentially solving a classification problem where the model learns to assign higher probability to preferred responses. Each gradient step simultaneously rewards the model for raising the probability of good outputs and penalizes it for raising the probability of bad ones, all without ever explicitly parameterizing a reward function.

Think of the whole derivation as a three-act story. Act one asks: if we had the ideal reward function, what would the optimal policy look like? Act two reverses that question and asks: if we have the optimal policy, what reward function does it imply? Act three substitutes that reversed reward into the preference model and discovers that the messy partition function term vanishes, leaving a clean, tractable loss. Each act is mathematically precise, and each act builds directly on the previous one. When you finish reading, the chain from RLHF to DPO will feel inevitable rather than mysterious.

One reason this derivation matters beyond academic interest is that it reveals the hidden assumptions baked into DPO. The loss function is not a heuristic: it follows from specific, clearly stated premises about the form of the RLHF objective, the Boltzmann structure of optimal policies, and the Bradley-Terry preference model. Knowing those premises tells you exactly when DPO is appropriate, what might break it, and how to modify it for different settings. We will examine those assumptions carefully in the section on limitations.

Let's begin with the objective that RLHF aims to optimize. As we discussed in the chapters on PPO for Language Models and KL Divergence Penalty, the goal is to find a policy $\pi$ that maximizes expected reward while staying close to a reference policy $\pi_{\text{ref}}$ (typically the supervised fine-tuned model).

Before diving into the mathematics, it helps to understand the intuition behind this objective. We want our language model to generate responses that humans prefer, which is captured by the reward function. However, if we optimize the reward too aggressively, the model might find unexpected shortcuts or produce degenerate outputs that technically achieve high reward but don't represent helpful behavior. The reference policy is an anchor, representing the model's pre-trained knowledge and natural language capabilities. By penalizing deviations from this anchor, we encourage the model to improve its outputs while maintaining coherent, fluent generation.

Think of the reference policy as the starting point on a map, and the reward function as a compass pointing toward a better destination. The KL penalty acts like a tether that keeps you from wandering too far afield in a single step. You want to move in the direction of higher reward, but you don't want to teleport so far that you lose the path entirely. In language modeling, that "losing the path" corresponds to mode collapse or reward hacking: outputs that score well on a proxy reward metric but are nonsensical or harmful. The KL constraint is what prevents those degenerate solutions.

The constrained optimization problem is:

where:

• $\pi$: the policy being optimized
• $\mathbb{E}$: the expectation operator
• $x$: a prompt sampled from the data distribution $\mathcal{D}$
• $y$: a response sampled from the policy
• $r(x, y)$: the learned reward function
• $\beta$: a hyperparameter controlling the strength of the KL constraint
• $D_{\text{KL}}$: the Kullback-Leibler divergence
• $\pi_{\text{ref}}$: the reference policy

To work with this objective more directly, we need to expand the KL divergence term. Recall that the KL divergence measures how different one probability distribution is from another, expressed as the expected log ratio of the two distributions. Writing out the KL divergence explicitly, this becomes:

where:

• $\mathbb{E}$: the expectation operator
• $\mathcal{D}$: the data distribution
• $r(x, y)$: the reward function
• $\beta$: the KL penalty coefficient
• $\log$: the natural logarithm
• $\pi(y|x)$: probability of response $y$ given prompt $x$ under the policy
• $\pi_{\text{ref}}(y|x)$: probability of response $y$ given prompt $x$ under the reference model

This reformulation reveals the objective's structure more clearly. The log ratio $\log \frac{\pi(y|x)}{\pi_{\text{ref}}(y|x)}$ measures how much the policy has shifted away from the reference for a particular response. When this ratio is positive (the policy assigns more probability than the reference), the KL term subtracts from the objective, penalizing the deviation. When the ratio is negative (the policy assigns less probability), the term adds to the objective, which might seem like a reward, but since we're sampling from $\pi$, we're unlikely to generate responses where $\pi$ assigns low probability.

The key insight is that this scalar objective, combining a reward signal and a distributional constraint, can be solved analytically. That analytical solution is the foundation of everything that follows. The reward-KL tradeoff creates a specific mathematical structure that admits an exact solution in closed form, rather than acting as an arbitrary heuristic regularizer.

This objective captures a basic tension in alignment: we want the model to produce high-reward outputs (as judged by human preferences), but we don't want it to deviate too far from its pre-trained behavior. Without the KL penalty, the model might find degenerate solutions that exploit flaws in the reward model, a phenomenon we explored in the chapter on Reward Hacking.

Before proceeding to the derivation, it is worth spending a moment on why the KL divergence is the natural choice of regularizer here. Many other distance measures between distributions exist, such as total variation distance, Renyi divergences, or the Wasserstein distance. Each would produce a different optimization problem and a different optimal policy. The KL divergence $D_{\text{KL}}(\pi \| \pi_{\text{ref}})$ is particularly convenient because it has exactly the right algebraic structure to yield a clean closed-form solution. Its definition as $\mathbb{E}_\pi[\log(\pi / \pi_{\text{ref}})]$ means that when we write the full objective as an expectation under $\pi$, everything becomes a single expectation of a per-token scalar quantity, which is tractable.

The KL divergence also has information-theoretic grounding. It measures the number of extra "bits" required to encode samples from $\pi$ using a code optimized for $\pi_{\text{ref}}$. Keeping this quantity small means the policy's outputs remain within the statistical neighborhood of the reference, which in practice means they retain the grammatical fluency, factual grounding, and stylistic coherence of the original supervised fine-tuned model.

The key insight behind DPO is that this optimization problem has a closed-form solution. Unlike many optimization problems in machine learning that require iterative gradient descent, this particular formulation admits an analytical answer. For a fixed prompt $x$, we're optimizing over the distribution $\pi(y|x)$ for all possible responses $y$. This is a constrained optimization problem over probability distributions: we must ensure our solution is a valid probability distribution that sums to one and assigns non-negative probability to every possible response.

The special structure of the RLHF objective allows for this closed-form solution. The expectation under $\pi$ combined with the KL divergence creates what's known as a variational problem over distributions. Such problems often have elegant solutions when the constraints are simple probability simplex constraints.

Let's work through the derivation step by step. For a single prompt $x$, the objective is:

where:

• $\sum_{y}$: sum over all possible responses in the vocabulary
• $\pi(\cdot|x)$: the entire probability distribution over responses for prompt $x$
• $\pi(y|x)$: probability of response $y$ given prompt $x$
• $r(x, y)$: the reward function
• $\beta$: the KL penalty coefficient
• $\log$: the natural logarithm
• $\pi_{\text{ref}}(y|x)$: the reference policy probability

We need to maximize this subject to the constraint that $\pi(\cdot|x)$ is a valid probability distribution: $\sum_y \pi(y|x) = 1$ and $\pi(y|x) \geq 0$ for all $y$.

To solve this constrained optimization problem, we use the method of Lagrange multipliers, a classical technique from calculus. The idea is to incorporate the constraint directly into the objective by introducing a new variable (the Lagrange multiplier) that penalizes violations of the constraint. At the optimal solution, the gradient of the objective with respect to the decision variables must be proportional to the gradient of the constraint.

The Lagrangian is:

where:

• $\mathcal{L}$: the Lagrangian function
• $\sum_{y}$: sum over all responses
• $\pi(y|x)$: probability of response $y$
• $r(x, y)$: the reward function
• $\beta$: the KL penalty coefficient
• $\log$: the natural logarithm
• $\pi_{\text{ref}}(y|x)$: the reference policy probability
• $\lambda$: the Lagrange multiplier for the constraint that probabilities sum to 1

Notice that we've expanded the log ratio term from the original objective, separating it into $-\beta \log \pi(y|x)$ and $+\beta \log \pi_{\text{ref}}(y|x)$. This separation makes taking derivatives more straightforward. The term $\lambda(1 - \sum_y \pi(y|x))$ enforces our normalization constraint: if the probabilities don't sum to one, this term will be nonzero, and the Lagrange multiplier $\lambda$ will adjust to push us toward a valid distribution.

Taking the derivative with respect to $\pi(y|x)$ for a specific $y$:

where:

• $\frac{\partial \mathcal{L}}{\partial \pi(y|x)}$: the partial derivative of the Lagrangian with respect to the probability of response $y$
• $r(x, y)$: the reward function
• $\beta$: the KL penalty coefficient
• $\log$: the natural logarithm
• $\pi(y|x)$: the policy probability
• $\pi_{\text{ref}}(y|x)$: the reference policy probability
• $\lambda$: the Lagrange multiplier

The $-\beta$ term comes from differentiating $\pi \log \pi$, which gives $\log \pi + 1$. This is a standard result from calculus: when we differentiate $\pi \log \pi$ with respect to $\pi$, we apply the product rule, obtaining $\log \pi + \pi \cdot (1/\pi) = \log \pi + 1$. Setting this derivative equal to zero gives us the first-order optimality conditions.

Solving for $\log \pi(y|x)$:

where:

• $\log \pi(y|x)$: the log-probability of response $y$
• $r(x, y)$: the reward function
• $\beta$: the KL penalty coefficient
• $\pi_{\text{ref}}(y|x)$: the reference policy probability
• $\lambda/\beta$: constant terms related to the normalization constraint

This expression for the log-probability has an illuminating structure. The log-probability under our optimal policy equals the log-probability under the reference, adjusted by the scaled reward $r(x,y)/\beta$, plus some constants that don't depend on $y$. Higher reward responses get higher log-probabilities, with $\beta$ controlling how much the reward matters relative to staying close to the reference.

Exponentiating both sides:

where:

• $\pi(y|x)$: the policy probability
• $\pi_{\text{ref}}(y|x)$: the reference policy probability
• $\exp(\cdot)$: the exponential function
• $\exp(r(x, y)/\beta)$: the reward-scaling term
• $r(x, y)$: the reward function
• $\beta$: the KL penalty coefficient
• $\exp(-1 - \lambda/\beta)$: the normalization term constant across all $y$

The term $\exp(-1 - \lambda/\beta)$ is just a normalizing constant that ensures the distribution sums to 1. This term doesn't depend on $y$; it only depends on $x$ through the constraint that all probabilities must sum to one. We can absorb it into a partition function $Z(x)$:

where:

• $\pi^*(y|x)$: the optimal policy distribution
• $Z(x) = \sum_{y'} \pi_{\text{ref}}(y'|x) \exp\left( \frac{r(x, y')}{\beta} \right)$: the partition function (normalizing constant) for prompt $x$
• $\pi_{\text{ref}}(y|x)$: the reference policy distribution
• $r(x, y)$: the reward function
• $\beta$: the temperature parameter

This is the optimal policy for the RLHF objective. It has a clear interpretation: the optimal policy takes the reference distribution and reweights each response by the exponentiated reward, then normalizes. The exponential function ensures all probabilities are positive, naturally satisfying the $\pi(y|x) \ge 0$ constraint. Responses with higher reward get exponentially more probability mass, with $\beta$ controlling how aggressively we reweight.

Consider what happens at extreme values of beta to see how this reweighting works. When $\beta$ is very large, the reward term $r(x,y)/\beta$ becomes small, and $\exp(r(x,y)/\beta) \approx 1$ for all responses. In this limit, the optimal policy stays very close to the reference distribution, barely adjusting for reward at all. Conversely, when $\beta$ is very small, the exponential amplifies small reward differences into massive probability shifts, putting all probability mass on the highest-reward response. The choice of $\beta$ therefore controls the exploration-exploitation trade-off: larger values favor staying close to the reference (exploration), while smaller values favor chasing high reward (exploitation).

Before moving on, we should acknowledge a serious computational obstacle lurking in the optimal policy formula. The partition function $Z(x)$ is defined as:

where the sum ranges over all possible responses $y'$. For a language model, "all possible responses" means all sequences of tokens up to some maximum length. The number of such sequences is astronomical: a modest vocabulary of 32,000 tokens with a maximum length of 200 tokens gives $32000^{200}$ possible responses, a number larger than the estimated number of atoms in the observable universe. Computing $Z(x)$ directly is completely intractable.

This intractability is precisely why naive approaches to optimizing the RLHF objective are so expensive. Algorithms like PPO work around it by using sample-based estimates and importance weighting, but those approaches have their own difficulties, including high variance and sensitivity to hyperparameters. The DPO derivation sidesteps this problem entirely, and the key step is coming up next in the reparameterization.

The next step uses the relationship between the optimal policy and the reward function in reverse. Given an optimal policy, we can solve for the reward function it implies.

This reverse direction might seem like a mathematical curiosity, but it turns out to be the key insight that makes DPO possible. If we can express the reward purely in terms of policies (without needing to train a separate reward model), then we can substitute this expression into the preference model and optimize directly. The reward becomes implicit in the policy rather than explicit in a separate neural network.

Think of it this way: the optimal policy and the reward function are two sides of the same coin. Knowing one completely determines the other, up to an additive constant per prompt. The optimal policy is just a "softmax" over rewards (scaled by $1/\beta$) relative to the reference. Inverting that relationship is like going from probabilities back to log-probabilities: algebraically straightforward once you know the connection exists.

Starting from:

where:

• $\pi^*(y|x)$: the optimal policy distribution
• $Z(x)$: the partition function
• $\pi_{\text{ref}}(y|x)$: the reference policy distribution
• $r(x, y)$: the reward function
• $\beta$: the temperature parameter

We can rearrange to isolate the reward. First, take the log of both sides:

where:

• $\log \pi^*$: log-probability under the optimal policy
• $\log \pi_{\text{ref}}(y|x)$: log-probability under the reference policy
• $r(x, y)$: the reward function
• $\beta$: the KL penalty coefficient
• $\log Z(x)$: log of the partition function (depends only on $x$)

Taking the logarithm turns our multiplicative relationship into an additive one. This makes algebraic manipulation simpler. The logarithm of a product becomes a sum of logarithms, and the logarithm of the exponential simply returns its argument.

Now, we rearrange terms to solve for $r(x, y)$:

where:

• $r(x, y)$: the implicit reward
• $\beta$: the KL penalty coefficient
• $\pi^*(y|x)$: the optimal policy
• $\pi_{\text{ref}}(y|x)$: the reference policy
• $Z(x)$: the partition function
• $\beta \log (\cdot)$: the scaled log-ratio term

This is the reward reparameterization. It tells us that for any optimal policy $\pi^*$, we can express the implicit reward purely in terms of log probability ratios, plus a prompt-dependent constant $\beta \log Z(x)$.

The log ratio $\log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)}$ has a natural interpretation: it measures how much the optimal policy has increased or decreased the probability of response $y$ compared to the reference. Responses that the optimal policy strongly prefers will have large positive log ratios, while responses it disfavors will have large negative log ratios. This log ratio, scaled by $\beta$, gives us the implicit reward up to an additive constant.

The partition function $Z(x)$ doesn't depend on $y$; it only depends on the prompt $x$. This will turn out to be important because when we compute preference probabilities, this term cancels out.

The reward reparameterization eliminates the need for an explicit reward model. In standard RLHF, you train a separate neural network $r_\phi(x, y)$ on human preference data, and then you use that trained reward model to provide training signal to the language model policy. That two-step pipeline doubles the computational cost, introduces an additional source of approximation error, and creates the reward hacking problem when the policy finds outputs that score well on $r_\phi$ but do not match human preferences.

The reparameterization shows that you never needed a separate reward model. The reward is already encoded in the ratio between two language models: the policy being trained and the reference model. Every time you update the policy's weights, the implicit reward function changes accordingly. The reward model and the policy are the same object, just viewed from different angles. This is the deep reason why DPO is simpler than RLHF: it reduces the engineering work and the number of conceptual entities you have to reason about.

Now we can derive the DPO loss by substituting our reward reparameterization into the Bradley-Terry preference model. Recall from the chapter on the Bradley-Terry Model that the probability of preferring response $y_w$ over response $y_l$ is:

where:

• $p(y_w \succ y_l | x)$: probability that response $y_w$ is preferred over $y_l$
• $\sigma(z) = \frac{1}{1 + e^{-z}}$: the sigmoid function mapping values to $(0, 1)$
• $r(x, y_w)$: reward for the winning response
• $r(x, y_l)$: reward for the losing response

The Bradley-Terry model is elegant in its simplicity: the probability of preferring one response over another depends only on the difference in their rewards, passed through a sigmoid function. This means responses with much higher reward are strongly preferred, while responses with similar rewards have preference probabilities close to 0.5.

Substituting our reparameterized reward:

where:

• $\sigma$: the sigmoid function
• $\beta$: the temperature parameter
• $\pi^*$: the optimal policy
• $\pi_{\text{ref}}$: the reference policy
• $Z(x)$: the partition function terms which appear with opposite signs

Notice that the $\beta \log Z(x)$ terms cancel! This cancellation is not a coincidence; it's a direct consequence of using reward differences in the Bradley-Terry model. The partition function contributes equally to both rewards, so when we subtract them, it disappears. This cancellation is needed for the practical success of DPO because computing $Z(x)$ would require summing over all possible responses, which is computationally intractable for language models with exponentially large output spaces.

The key insight is that this cancellation is guaranteed by the structure of the Bradley-Terry model. Because preferences are defined as comparisons between pairs of responses (rather than absolute scores), any term that depends only on the prompt $x$ and not on the response $y$ will cancel in the subtraction. The partition function is exactly such a term, depending on $x$ through the sum over all possible responses but not on any particular response $y_w$ or $y_l$. This is why the DPO derivation only works cleanly with pairwise comparison preference models and would need modification for other preference structures.

This leaves:

where:

• $\sigma$: the sigmoid function
• $\beta$: temperature parameter
• $\pi^*$: the optimal policy
• $\pi_{\text{ref}}$: the reference policy

We can simplify this using logarithm properties:

where:

• $\sigma$: the sigmoid function
• $\beta$: the KL penalty coefficient
• $\log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)}$: the log-likelihood ratio of the optimal policy vs reference

This expression is remarkably clean. The probability of the correct preference depends entirely on the difference between two log ratios: how much the optimal policy prefers the winning response (relative to the reference) versus how much it prefers the losing response. When this difference is large and positive, the sigmoid outputs a value close to 1, showing strong confidence in the correct preference.

This is the probability that the optimal policy assigns to the correct preference. To train a policy $\pi_\theta$ to match this optimal policy, we maximize the log-likelihood of the observed preferences:

where:

• $\mathcal{L}_{\text{DPO}}$: the DPO loss function
• $\pi_\theta$: the policy network being trained (parameterized by $\theta$)
• $\pi_{\text{ref}}$: the reference policy
• $\mathcal{D}$: the dataset of preference pairs $(x, y_w, y_l)$
• $\mathbb{E}$: expectation over the dataset
• $\beta$: the KL penalty coefficient
• $\sigma$: the sigmoid function

The negative sign appears because we're minimizing a loss rather than maximizing likelihood.

To make the notation more compact, define the log-ratio for a response $y$:

where:

• $\hat{r}_\theta(x, y)$: the implicit reward assigned by the current model $\pi_\theta$
• $\beta$: the KL penalty coefficient scaling the reward
• $\pi_\theta(y|x)$: the policy probability
• $\pi_{\text{ref}}(y|x)$: the reference policy probability
• $\log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$: the log-ratio of the policy probability to the reference probability

This is sometimes called the "implicit reward" because it's what the reward would be if $\pi_\theta$ were optimal. The terminology is fitting: we never explicitly compute or learn a reward function, yet the policy implicitly defines one through its log-probability ratios with the reference model.

The DPO loss becomes:

where:

• $\mathcal{L}_{\text{DPO}}$: the DPO loss function
• $\hat{r}_\theta(x, y)$: the implicit reward function
• $\hat{r}_\theta(x, y_w) - \hat{r}_\theta(x, y_l)$: the margin between implicit rewards for winning and losing responses
• $\sigma$: the sigmoid function
• $\mathbb{E}$: expectation over the dataset

This is remarkably simple. The loss encourages the model to increase its log probability (relative to the reference) on preferred responses and decrease it on dispreferred responses.

The DPO loss has a revealing interpretation: it's essentially binary classification with a specific parameterization. Consider a standard binary cross-entropy loss for classifying which of two responses is preferred:

where:

• $\mathcal{L}_{\text{BCE}}$: Binary Cross-Entropy loss
• $f(x, y_w, y_l)$: a scoring function showing preference strength
• $\sigma$: the sigmoid function
• $\mathbb{E}$: expectation over the dataset

In this context, $f$ is some function that should output a positive value when $y_w$ is truly preferred. In DPO, this function is:

where:

• $f(x, y_w, y_l)$: the logit fed into the sigmoid classifier
• $\beta$: the KL penalty coefficient
• $\pi_\theta$: the policy network
• $\pi_{\text{ref}}$: the reference policy

The model is learning to classify preference pairs by adjusting its output probabilities. When the loss is low, the model assigns higher relative probability to the preferred response.

This classification view explains why DPO is so stable compared to RLHF. Instead of learning a separate reward model and then using policy gradients with their high variance, DPO directly optimizes the likelihood of preference labels. The gradients flow directly through the model's log probabilities, just like in standard language model training.

The gradient of the DPO loss with respect to $\theta$ has a particularly intuitive form. For a single example:

where:

• $\nabla_\theta$: gradient with respect to model parameters $\theta$
• $\mathcal{L}_{\text{DPO}}$: the DPO loss
• $\sigma$: the sigmoid function
• $\hat{r}_\theta$: the implicit reward function
• $\sigma(\dots)$: the weighting term derived from the sigmoid derivative (probability of the wrong preference)
• $\nabla_\theta \log \dots$: the geometric gradient direction
• $\beta$: the KL penalty coefficient

This gradient expression reveals the geometry of DPO optimization. The term $\nabla_\theta \log \pi_\theta(y_w|x) - \nabla_\theta \log \pi_\theta(y_l|x)$ points in the direction that increases the probability of the winning response while decreasing the probability of the losing response. This is the natural direction to push given our objective.

The term $\sigma(-\hat{r}_\theta(x, y_w) + \hat{r}_\theta(x, y_l))$ is the probability that the model currently assigns to the wrong preference. This acts as an implicit weighting:

• When the model is confident and correct (low wrong-preference probability), the gradient is small
• When the model is confident but wrong, the gradient is large
• When the model is uncertain, the gradient is moderate

This automatic weighting helps the model focus on examples it's getting wrong, similar to how hard negative mining works in contrastive learning.

It is instructive to compare the DPO gradient to what happens during ordinary language model pre-training. In standard cross-entropy training on tokens, the gradient for a single token at position $t$ increases the log-probability of the correct next token $w_t$ and the magnitude is governed by the current probability of that token: if the model already assigns high probability, the gradient is small. DPO follows the same pattern but at the sequence level: it increases log-probability for the entire chosen sequence $y_w$ and decreases it for the rejected sequence $y_l$, weighted by how wrong the model currently is.

This parallel makes DPO feel like a natural extension of language modeling rather than a fundamentally different training paradigm. You can implement DPO in virtually any training framework that supports computing sequence log-probabilities, because the loss is just a scalar function of those log-probabilities. The practical implementation details, including how to handle variable-length sequences, padding masks, and batched reference model inference, are covered in the next chapter on DPO Implementation.

Let's trace through the derivation with concrete numbers to build intuition. Suppose we have a prompt $x$ = "Write a haiku about autumn" and two responses:

• $y_w$ (preferred): "Crimson leaves descend / Dancing on October wind / Earth prepares to sleep"
• $y_l$ (dispreferred): "Leaves fall down in fall / The weather gets cold outside / I like pumpkin spice"

The preferred response demonstrates poetic craft through concrete imagery ("crimson," "dancing"), sensory detail, and thematic resonance with the natural world. The dispreferred response is grammatically correct but prosaic, repeating "fall" clumsily and ending on an unrelated tangent about pumpkin spice. A human evaluator would clearly prefer the first response, and a well-trained reward model should assign it a higher score.

Assume our reference model $\pi_{\text{ref}}$ assigns:

• $\log \pi_{\text{ref}}(y_w|x) = -45.2$ (log probability of generating the preferred response)
• $\log \pi_{\text{ref}}(y_l|x) = -38.7$ (log probability of generating the dispreferred response)

The reference model assigns higher probability to the dispreferred response because it's simpler and uses more common words. The word "fall" is far more common in training data than "crimson" or "descend," and the dispreferred response is shorter and uses everyday vocabulary throughout. This is exactly the distribution shift DPO is designed to fix: the reference model, trained to predict natural text, has no built-in preference for poetic quality over simplicity.

Now suppose our current policy $\pi_\theta$ assigns:

• $\log \pi_\theta(y_w|x) = -42.1$
• $\log \pi_\theta(y_l|x) = -39.5$

With $\beta = 0.1$, the implicit rewards are:

The difference is:

The loss contribution from this example:

The model is doing okay on this example (probability of correct preference is about 60%), but there's room to improve. The gradient will push the model to further increase $\pi_\theta(y_w|x)$ relative to the reference and decrease $\pi_\theta(y_l|x)$ relative to the reference.

Let's also trace through what a "good" and a "bad" model state look like for this same example. Suppose after more training the policy reaches:

• $\log \pi_\theta(y_w|x) = -38.0$ (probability has increased from reference)
• $\log \pi_\theta(y_l|x) = -41.5$ (probability has decreased from reference)

Now the implicit rewards are:

The reward margin is now $0.72 - (-0.28) = 1.00$, and the preference probability is $\sigma(1.00) \approx 0.73$. The loss drops to $-\log(0.73) \approx 0.31$. The model has learned to "see" the quality difference between these two responses, even though it never directly observed a reward signal during this training phase.

Let's implement the DPO loss function and verify our derivation with code.

The key parameter for DPO is:

• beta: The temperature parameter (often denoted as $\beta$) that scales the log-ratio of the policy and reference probabilities. It controls the strength of the KL divergence penalty, with larger values keeping the policy closer to the reference model.

Choosing $\beta$ in practice requires balancing two failure modes. If $\beta$ is too small, the DPO loss will push the policy aggressively in the direction of human preferences without adequate constraint from the reference model. This can cause the policy to drift far from the reference distribution and start generating fluent-but-wrong outputs, such as responses that contain the right topics but with incorrect factual content, or responses that are overly verbose because longer responses were slightly preferred in the training data. If $\beta$ is too large, the policy can barely move away from the reference, which means it learns slowly and may not improve much even after thousands of training steps. Empirically, values in the range 0.01 to 0.5 are common, with 0.1 being a popular default starting point.

The DPO derivation makes several assumptions worth examining carefully, because each one represents a potential point of failure in real-world applications.

The first assumption is that the Bradley-Terry model correctly captures human preferences. Specifically, DPO assumes that the probability of preferring response $y_w$ over $y_l$ can be expressed as a sigmoid of a scalar reward difference. Real human preferences are far messier. They are noisy (the same annotator might rate the same pair differently on different days), inconsistent across annotators (different people have different values), and often non-transitive (a human might prefer $y_1$ over $y_2$, prefer $y_2$ over $y_3$, but also prefer $y_3$ over $y_1$). The Bradley-Terry model cannot represent this complexity. DPO inherits this limitation from the underlying preference model: if your preference data is very noisy or inconsistent, the loss will receive contradictory training signals and may not converge to a meaningful policy.

The second assumption is that the optimal policy has the Boltzmann form derived earlier. This is guaranteed only for the specific objective we started with, namely expected reward minus a KL divergence from the reference. Different alignment objectives, such as objectives using different divergence measures or adding additional safety constraints, would yield different optimal policies and potentially different direct alignment algorithms. The DPO loss is not a general-purpose alignment loss; it is the specific loss that corresponds to the KL-regularized RLHF objective with the Bradley-Terry preference model. When researchers have proposed modifications, such as using an $f$-divergence instead of KL, they obtain modified versions of DPO with different properties. The next chapter discusses several of these variants.

The third assumption is that model capacity is sufficient. DPO optimizes toward the optimal policy but doesn't guarantee reaching it. The policy $\pi_\theta$ is constrained by the model's architecture and capacity. A small model may not be able to represent the true optimal policy, in which case DPO finds the best approximation within the model class. This is not unique to DPO, since all finite-parameter models face this limitation, but the theoretical guarantees assume that the policy class is expressive enough to represent the optimal Boltzmann distribution.

A fourth, more subtle assumption concerns the relationship between the data distribution and the behavior of the optimal policy. The derivation assumes we optimize over the true expectation under the training distribution $\mathcal{D}$. In practice, we minimize an empirical loss over a finite dataset. If the preference dataset is small or unrepresentative of the prompts the model will encounter at deployment, DPO may overfit to the dataset distribution. This manifests as mode collapse on training prompts or failure to generalize to held-out prompt types. This issue is partly mitigated by the KL constraint (via the reference policy), but a badly distributed training set will still cause problems.

Finally, DPO requires that we can compute $\pi_{\text{ref}}(y|x)$ exactly for the same responses we're evaluating under $\pi_\theta$. This is straightforward when both are the same model architecture, but becomes complex if the reference model is unavailable or uses different tokenization. The next chapter on DPO Implementation will address these practical considerations in detail.

This chapter derived the DPO loss function from first principles, showing how it emerges naturally from the RLHF objective.

The key steps in the derivation were:

1. RLHF objective: Maximize expected reward with a KL penalty to stay close to the reference policy
2. Optimal policy: Using Lagrange multipliers, we found that the optimal policy is a Boltzmann distribution that reweights the reference by exponentiated rewards
3. Reward reparameterization: Inverting this relationship, we expressed the implicit reward as a log-ratio between the optimal and reference policies, plus a partition function
4. Bradley-Terry substitution: Plugging the reparameterized rewards into the preference model, the partition functions cancel, leaving a loss in terms of log-ratios only
5. Classification view: The resulting DPO loss is binary cross-entropy, learning to classify which response is preferred based on relative log-probabilities

The gradient of the DPO loss has an automatic weighting scheme: examples where the model is confidently wrong receive larger gradients, while examples where it's already correct receive smaller gradients. This makes DPO training stable and efficient.

The derivation also reveals DPO's key assumptions: that human preferences follow the Bradley-Terry model, that the KL-regularized RLHF objective is the right starting point, and that the policy class is expressive enough to represent the optimal Boltzmann distribution. When those assumptions hold, DPO provides a principled, efficient alternative to RLHF. When they break down, the variants we explore in the following chapter offer modified objectives that relax different aspects of those assumptions.

In the next chapter, we'll implement DPO end-to-end, covering practical details like computing sequence log-probabilities, handling padding, and integrating with Hugging Face's training infrastructure.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about the DPO derivation.