DPO Variants › Summary

This chapter examined four influential DPO variants, each addressing specific limitations of the original algorithm while preserving the core insight that preference learning can bypass explicit reward modeling.

IPO reformulates preference learning as regression with a target margin $\frac{1}{2\beta}$, preventing the unbounded preference growth that can occur with standard DPO. The squared loss creates self-correcting gradients that converge to a stable equilibrium, and the gradient changes sign when the model overshoots the target. This gives natural protection against overconfidence.

KTO lets learning from unpaired binary feedback by incorporating insights from prospect theory. The asymmetric value function and adaptive reference point allow training on thumbs-up and thumbs-down data without requiring direct comparisons between responses. The loss aversion weighting shows the empirical finding that humans are more sensitive to bad outcomes than equivalent good ones.

ORPO eliminates the reference model entirely by combining SFT with odds-ratio preference optimization. This reduces memory requirements by half and collapses two-stage training into a single pass. The coupling between the SFT loss and the odds ratio loss gives implicit regularization that substitutes for the KL constraint provided by the reference model in other methods.

cDPO handles annotation noise through label smoothing, modeling preferences probabilistically rather than deterministically. The smoothing parameter $\epsilon$ should reflect actual annotation uncertainty and directly determines the finite optimal gap at which training stabilizes.

Method selection depends on data format, computational constraints, annotation quality, and training duration. No single method dominates across all scenarios. IPO and cDPO both produce bounded optimization with stable equilibria; they differ in how that equilibrium is derived. KTO handles unpaired data; ORPO saves memory. Understanding these trade-offs at the level of loss functions and gradients, rather than just benchmark numbers, equips you to make informed choices and to diagnose problems when experiments do not go as expected.