DPO Derivation › Assumptions and Validity

The DPO derivation makes several assumptions worth examining carefully, because each one represents a potential point of failure in real-world applications.

The first assumption is that the Bradley-Terry model correctly captures human preferences. Specifically, DPO assumes that the probability of preferring response $y_w$ over $y_l$ can be expressed as a sigmoid of a scalar reward difference. Real human preferences are far messier. They are noisy (the same annotator might rate the same pair differently on different days), inconsistent across annotators (different people have different values), and often non-transitive (a human might prefer $y_1$ over $y_2$, prefer $y_2$ over $y_3$, but also prefer $y_3$ over $y_1$). The Bradley-Terry model cannot represent this complexity. DPO inherits this limitation from the underlying preference model: if your preference data is very noisy or inconsistent, the loss will receive contradictory training signals and may not converge to a meaningful policy.

The second assumption is that the optimal policy has the Boltzmann form derived earlier. This is guaranteed only for the specific objective we started with, namely expected reward minus a KL divergence from the reference. Different alignment objectives, such as objectives using different divergence measures or adding additional safety constraints, would yield different optimal policies and potentially different direct alignment algorithms. The DPO loss is not a general-purpose alignment loss; it is the specific loss that corresponds to the KL-regularized RLHF objective with the Bradley-Terry preference model. When researchers have proposed modifications, such as using an $f$-divergence instead of KL, they obtain modified versions of DPO with different properties. The next chapter discusses several of these variants.

The third assumption is that model capacity is sufficient. DPO optimizes toward the optimal policy but doesn't guarantee reaching it. The policy $\pi_\theta$ is constrained by the model's architecture and capacity. A small model may not be able to represent the true optimal policy, in which case DPO finds the best approximation within the model class. This is not unique to DPO, since all finite-parameter models face this limitation, but the theoretical guarantees assume that the policy class is expressive enough to represent the optimal Boltzmann distribution.

A fourth, more subtle assumption concerns the relationship between the data distribution and the behavior of the optimal policy. The derivation assumes we optimize over the true expectation under the training distribution $\mathcal{D}$. In practice, we minimize an empirical loss over a finite dataset. If the preference dataset is small or unrepresentative of the prompts the model will encounter at deployment, DPO may overfit to the dataset distribution. This manifests as mode collapse on training prompts or failure to generalize to held-out prompt types. This issue is partly mitigated by the KL constraint (via the reference policy), but a badly distributed training set will still cause problems.

Finally, DPO requires that we can compute $\pi_{\text{ref}}(y|x)$ exactly for the same responses we're evaluating under $\pi_\theta$. This is straightforward when both are the same model architecture, but becomes complex if the reference model is unavailable or uses different tokenization. The next chapter on DPO Implementation will address these practical considerations in detail.