DPO Variants › DPO Variants: IPO, KTO, ORPO, and cDPO

In the previous chapters, we derived Direct Preference Optimization from first principles and implemented it as a simpler alternative to RLHF. DPO was a breakthrough: it showed that you could align language models with human preferences using a simple classification-style loss, without ever explicitly training a reward model or running a reinforcement learning loop. The key derivation revealed that the optimal policy under the KL-constrained reward maximization objective could be expressed directly in terms of the policy's own log-probabilities, collapsing what had been a two-stage pipeline into a single supervised objective. That elegance made DPO enormously attractive.

Yet elegance often conceals hidden fragility. As practitioners began applying DPO at scale, they encountered a cluster of practical problems that the theoretical derivation had glossed over. The training loss continued to decrease even when models were creating clearly overconfident predictions. Systems trained on crowd-sourced preference data often behaved erratically, apparently because some annotators had labeled responses incorrectly. Teams working with limited GPU memory struggled to keep two copies of a large model in VRAM simultaneously. And researchers collecting feedback from deployed products found that most of their data came in the form of thumbs-up and thumbs-down ratings on individual responses, not as head-to-head comparisons that DPO's paired format requires. Each of these problems was real, each caused real engineering pain, and each motivated a distinct line of research.

This chapter explores four of the most influential responses to those problems. Identity Preference Optimization (IPO) addresses the unbounded reward growth problem by replacing DPO's monotone loss with a regression target, giving optimization a well-defined stopping point. Kahneman-Tversky Optimization (KTO) discards the paired data requirement entirely, drawing on behavioral economics to define a learning objective that works with simple binary labels. Odds Ratio Preference Optimization (ORPO) eliminates the reference model, combining supervised fine-tuning and preference learning into a single forward pass that requires only one copy of the model. Conservative DPO (cDPO) tackles annotation noise through label smoothing, acknowledging that human judgments are probabilistic rather than deterministic.

Think of these four variants as a family of tools, each sharpened for a different job. Understanding when to reach for each one requires understanding the problem each one solves and the mechanism it uses to solve it. We will work through the mathematics of all four variants carefully, compare their gradient behavior, implement them in code, and walk through a concrete numerical example that makes the differences tangible. By the end of this chapter, you will have the conceptual toolkit to select the right alignment method for your data characteristics and computational constraints, rather than defaulting to the original DPO when a better-matched alternative exists.