DPO Variants › Identity Preference Optimization (IPO)

IPO emerged from a careful analysis of DPO's theoretical properties. Azar et al. (2024) identified a subtle but important issue: as DPO training progresses, the implicit reward gap between chosen and rejected responses can grow unboundedly, causing the policy to assign extreme probabilities that do not reflect the actual strength of human preferences. This section explains exactly why that happens, what consequences it produces, and how IPO's squared-error reformulation resolves the problem.

Think of DPO's loss as a one-sided ramp: no matter how strongly the model already prefers the chosen response, the ramp keeps tilting in the same direction, always pushing the model to make the chosen response even more probable. IPO replaces the ramp with a bowl: there is a specific level of preference the model should reach, and any deviation from that level, whether too weak or too strong, incurs a penalty. The bowl has a well-defined minimum, and once the model reaches it, training stops pushing.

To understand why IPO was developed, we must first examine a basic tension within DPO's optimization dynamics. Recall from our DPO derivation that the loss function is:

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \left( \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \right) \right]$$

where:

• $\mathcal{L}_{\text{DPO}}$: the DPO loss function
• $\mathbb{E}_{(x, y_w, y_l)}$: the expectation over the dataset of preference pairs
• $\pi_\theta(y|x)$: the probability of response $y$ given prompt $x$ under the policy being trained
• $\pi_{\text{ref}}(y|x)$: the probability under the frozen reference model
• $y_w, y_l$: the chosen (winner) and rejected (loser) responses
• $\beta$: the temperature parameter controlling the strength of the KL divergence penalty
• $\sigma$: the sigmoid function

To appreciate the overfitting problem, consider what this loss function is asking the model to do. The expression inside the sigmoid is the difference between two log-ratios: how much more likely the policy makes the chosen response compared to the reference, minus the same quantity for the rejected response. The negative log-sigmoid of this quantity becomes the loss. When training minimizes this loss, it maximizes what is inside the log-sigmoid.

The sigmoid function $\sigma$ approaches 1 as its argument grows large. To minimize this loss, the model is incentivized to make the log-ratio difference as large as possible. In the limit, this pushes the model toward assigning probability approaching 1 to chosen responses and probability approaching 0 to rejected ones. There is no natural stopping point in this formulation: the gradient always points toward making the gap larger, even when the gap is already enormous.

This behavior is problematic for several reasons. First, human preferences are not absolute. A response labeled "chosen" is not infinitely better than the "rejected" alternative; it is merely preferred in that comparison. Two responses might differ only slightly in quality, yet the labeling treats one as definitively better. When the model learns to assign extreme probabilities based on these labels, it develops a false sense of certainty that does not match the underlying reality of human judgment. Second, training instability compounds as log probabilities approach $-\infty$ for rejected responses, causing gradients to become numerically unreliable. Third, and most subtly, the model may generalize poorly: it in effect memorizes that specific responses are "infinitely good" or "infinitely bad" rather than learning the generalizable patterns that distinguish better responses from worse ones.

IPO addresses the overfitting problem by reformulating preference learning as a regression problem with a specific target. The reason is that instead of letting the reward gap to grow without bound, we should specify how large we want that gap to be and then train the model to reach exactly that target. This turns an unbounded optimization problem into a bounded one with a clear convergence criterion.

The main point is: bounded optimization requires a target. DPO has no target for the log-ratio difference; it simply wants that difference to be as large as possible. IPO sets the target at $\frac{1}{2\beta}$ and penalizes any deviation from it, whether the model has not learned a strong enough preference or has learned an excessively strong one. This single design choice fundamentally changes the optimization landscape.

Instead of pushing the reward gap to infinity, IPO targets a fixed margin:

$$\mathcal{L}_{\text{IPO}} = \mathbb{E}_{(x, y_w, y_l)} \left[ \left( \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} - \frac{1}{2\beta} \right)^2 \right]$$

where:

• $\mathcal{L}_{\text{IPO}}$: the IPO loss function
• $\mathbb{E}_{(x, y_w, y_l)}$: the expectation over the dataset of preference pairs
• $\frac{1}{2\beta}$: the target margin derived from the regularization strength $\beta$
• $\pi_\theta, \pi_{\text{ref}}$: the policy and reference models

The key innovation is the squared loss with a specific target. Rather than rewarding the model for making the gap as large as possible, IPO rewards the model for making the gap equal to a specific value. Any deviation from this target, whether too small or too large, incurs a penalty. This is the essence of regression: we have a target value and we minimize the squared distance to that target.

To analyze these dynamics more precisely, let's define the log-ratio difference as:

$$h_\theta(x, y_w, y_l) = \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}$$

where:

• $h_\theta(x, y_w, y_l)$: the difference in log-probability ratios between chosen and rejected responses
• $\log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$: the "implicit reward" for a specific response $y$

This quantity $h_\theta$ captures the essence of what preference optimization reaches. It measures how much the policy has learned to prefer the chosen response over the rejected one, relative to what the reference model would predict. A positive value means the policy has shifted probability mass toward the chosen response; a larger positive value means a stronger learned preference.

With this notation, the IPO loss becomes elegantly simple:

$$\mathcal{L}_{\text{IPO}} = \mathbb{E} \left[ \left( h_\theta - \frac{1}{2\beta} \right)^2 \right]$$

where:

• $\mathcal{L}_{\text{IPO}}$: the IPO loss function
• $\mathbb{E}$: the expectation over the dataset
• $h_\theta$: the log-ratio difference computed by the model
• $\frac{1}{2\beta}$: the specific target value that the difference should converge to

This formulation makes the regression nature of IPO crystal clear. We are asking the model to make $h_\theta$ equal to $\frac{1}{2\beta}$ for every preference pair in the dataset. The squared loss penalizes deviations in either direction: if the model has not learned a strong enough preference ($h_\theta < \frac{1}{2\beta}$), the loss is positive and gradients push toward a larger gap; if the model has learned too strong a preference ($h_\theta > \frac{1}{2\beta}$), the loss is also positive and gradients push toward a smaller gap.

The gradient with respect to $h_\theta$ reveals this self-correcting behavior:

$$\frac{\partial \mathcal{L}_{\text{IPO}}}{\partial h_\theta} = 2 \left( h_\theta - \frac{1}{2\beta} \right)$$

where:

• $\frac{\partial \mathcal{L}_{\text{IPO}}}{\partial h_\theta}$: the gradient of the loss with respect to the log-ratio difference
• $h_\theta - \frac{1}{2\beta}$: the error term (distance from target) that drives the update

This gradient is zero precisely when $h_\theta = \frac{1}{2\beta}$, creating a stable equilibrium. Once the log-ratio difference reaches the target margin, there is no further pressure to increase it. The equilibrium is stable and attractive: regardless of where training starts, the gradients always point toward the target, and the strength of the gradient is proportional to the distance from the target.

The target $\frac{1}{2\beta}$ has a principled interpretation that connects IPO back to the theoretical foundations of preference learning. The $\beta$ parameter controls the strength of the KL constraint. A larger $\beta$ means weaker regularization, letting larger deviations from the reference policy. With IPO's target:

• When $\beta$ is large (weak KL penalty), the target margin $\frac{1}{2\beta}$ is small.
• When $\beta$ is small (strong KL penalty), the target margin is large.

This might seem counterintuitive until you consider a budget analogy. If you have a strict overall spending limit, you can afford to splurge on individual items because you are being careful everywhere else. Conversely, if your overall limit is loose, you must be more conservative on each purchase to avoid overshooting. Stronger KL constraints (small $\beta$) permit larger per-example margins because the overall deviation is more tightly controlled.

The target margin also has an interpretation in terms of the Bradley-Terry model that underlies preference learning. In that model, the probability that response $y_w$ is preferred to $y_l$ depends on the difference in their rewards. The target $\frac{1}{2\beta}$ corresponds to a specific preference probability representing a moderate preference rather than absolute certainty. This aligns with the reality that human annotations express preferences, not certainties. An annotator who chooses response A over response B is not asserting that A is infinitely better; they are expressing a judgment that may be uncertain, context-dependent, and occasionally reversed.

Different loss functions produce different gradient behaviors, and those differences explain why methods perform differently in practice. The gradient determines how the model updates its parameters at each step, so differences in gradient behavior translate directly into differences in training dynamics. Understanding these gradients is the key to predicting how a method will behave when you deploy it on your data.

Think of the gradient as the engine that drives training. DPO's engine runs at full throttle when the model is uncertain and idles as the model becomes confident, but it never turns off. IPO's engine runs proportionally to how far the model is from the target and cuts off completely once the target is reached.

The DPO gradient magnitude is:

$$\left| \frac{\partial \mathcal{L}_{\text{DPO}}}{\partial h_\theta} \right| = \beta \sigma(-\beta h_\theta)$$

where:

• $\beta$: the temperature parameter
• $\sigma(-\beta h_\theta)$: the sigmoid of the negative scaled log-ratio, acting as a weighting factor

This sigmoid term means DPO's gradient is largest when $h_\theta$ is near zero (uncertain predictions) and vanishes as $h_\theta \to \infty$ (confident predictions). The intuition is that DPO pushes hardest when the model is uncertain about which response is preferred and pushes more gently when the model is already confident. While vanishing gradients prevent infinite growth in principle, they also mean learning slows sharply once the model becomes confident. This creates a problematic dynamic: the model can still drift toward extreme probabilities because the gradients, though small, remain consistently positive, never reversing direction.

The IPO gradient magnitude is:

$$\left| \frac{\partial \mathcal{L}_{\text{IPO}}}{\partial h_\theta} \right| = 2 \left| h_\theta - \frac{1}{2\beta} \right|$$

where:

• $| h_\theta - \frac{1}{2\beta} |$: the absolute distance from the target margin

IPO's gradient magnitude is proportional to distance from the target. If the model overshoots the target ($h_\theta > \frac{1}{2\beta}$), the gradient reverses direction and pushes back. This self-correcting behavior is entirely absent in DPO. The gradient grows stronger as the model strays further from the target, which means that even large deviations are corrected rather than allowed to persist indefinitely.