DPO Variants › Conservative DPO (cDPO)

The variants discussed so far address algorithmic limitations. cDPO, introduced by Mitchell et al. (2023), addresses a data quality issue: label noise in preference annotations. While IPO and ORPO assume that the training labels are reliable (even if the optimization dynamics are problematic), cDPO starts from the recognition that human annotations are inherently uncertain and builds that uncertainty directly into the loss function.

Think of cDPO as the difference between a GPS that reports its location with false precision and one that reports a confidence interval. Standard DPO says "response A is better than response B, and that fact is certain." cDPO says "response A is probably better than response B, with probability $1 - \epsilon$." The probabilistic framing leads to a fundamentally different loss landscape with a bounded optimal preference gap rather than an infinite one.

Human preference annotations are inherently noisy, and this noise can arise from several sources. Annotators may disagree with each other on the same comparison, make mistakes due to fatigue or inattention, apply inconsistent criteria across examples, or be influenced by surface features such as formatting and length rather than actual quality differences.

Studies of inter-annotator agreement on preference tasks typically show agreement rates of 70 to 80 percent, meaning 20 to 30 percent of labels may be "wrong" in the sense that a different annotator would have labeled them oppositely. Even gold-standard human annotations on tasks like toxicity detection or factual accuracy show substantial disagreement when the same examples are labeled by different annotators. The problem is not that annotators are careless; preferences are subjective, and the ranking of two similar responses is often not clear-cut.

Standard DPO treats all preference labels as ground truth. If the training data says $y_w \succ y_l$, the model learns to prefer $y_w$ with full confidence. When this label is wrong, the model learns an incorrect preference, and because DPO's unbounded optimization pushes that preference to an extreme, even a small percentage of mislabeled examples can cause significant downstream problems. The model in effect memorizes the noise in the data and tries to push it to infinity.

cDPO applies label smoothing to preference labels. The idea behind label smoothing is familiar from classification: instead of training on hard targets (0 or 1), we train on soft targets that acknowledge uncertainty. For preferences, instead of treating each comparison as a certainty, we model the possibility that the annotation might be wrong.

Instead of treating preferences as deterministic ($P(y_w \succ y_l) = 1$), cDPO models uncertainty:

$$\begin{aligned} P(y_w \succ y_l) &= 1 - \epsilon \\ P(y_l \succ y_w) &= \epsilon \end{aligned}$$

where:

• $\epsilon$: the label smoothing parameter, $\epsilon \in [0, 0.5)$, representing uncertainty about annotation correctness

The parameter $\epsilon$ is our belief about how often the annotations are wrong. If $\epsilon = 0$, we trust all annotations completely, recovering standard DPO. If $\epsilon = 0.3$, we believe that 30 percent of the time the annotators got it backwards and the rejected response was better.

The cDPO loss modifies the Bradley-Terry model probability accordingly:

$$\mathcal{L}_{\text{cDPO}} = -\mathbb{E}_{(x, y_w, y_l)} \Big[ (1-\epsilon) \log \sigma(\beta \cdot h_\theta) + \epsilon \log \sigma(-\beta \cdot h_\theta) \Big]$$

where:

• $\mathcal{L}_{\text{cDPO}}$: the cDPO loss function
• $\mathbb{E}_{(x, y_w, y_l)}$: the expectation over the dataset
• $h_\theta$: the log-ratio difference between chosen and rejected responses
• $\epsilon$: the label smoothing parameter
• $\sigma$: the sigmoid function
• $\beta$: the temperature parameter

This loss function has two terms. The first term, weighted by $(1-\epsilon)$, is the standard DPO loss: it rewards the model for preferring the labeled chosen response. The second term, weighted by $\epsilon$, is the opposite: it rewards the model for preferring the labeled rejected response. The combination is the expected loss under our belief about annotation accuracy. We train the model to do well on average across both the case where the annotation is correct and the case where it is wrong.

Using the identity $\sigma(-z) = 1 - \sigma(z)$, we can rewrite this loss to explicitly show the probability mass assigned to the rejected response:

$$\mathcal{L}_{\text{cDPO}} = -\mathbb{E} \Big[ (1-\epsilon) \log \sigma(\beta \cdot h_\theta) + \epsilon \log (1 - \sigma(\beta \cdot h_\theta)) \Big]$$

where:

• $1 - \sigma(\beta \cdot h_\theta)$: the probability assigned to the rejected response (equivalent to $\sigma(-\beta \cdot h_\theta)$)
• $\epsilon$: the weighting factor for the "incorrect" preference direction

This form makes the label smoothing interpretation particularly clear: we are mixing the standard DPO loss with its inverse, with the mixing coefficient $\epsilon$ representing the probability of annotation error.

Label smoothing fundamentally changes the loss landscape. The gradient of cDPO with respect to $h_\theta$ is:

$$\frac{\partial \mathcal{L}_{\text{cDPO}}}{\partial h_\theta} = -\beta \left[ (1-\epsilon) \cdot \sigma(-\beta \cdot h_\theta) - \epsilon \cdot \sigma(\beta \cdot h_\theta) \right]$$

where:

• $\frac{\partial \mathcal{L}_{\text{cDPO}}}{\partial h_\theta}$: the gradient of the loss with respect to the log-ratio difference

This gradient has two competing terms. The first term pushes toward larger $h_\theta$ (stronger preference for chosen). The second term pushes toward smaller $h_\theta$ (acknowledging that the rejected response might be better). These competing forces create a balance point where the gradient is zero.

Setting the gradient to zero allows us to solve for the optimal log-ratio difference $h_\theta^*$:

$$\begin{aligned} \sigma(\beta \cdot h_\theta^*) &= \frac{1 - \epsilon}{(1 - \epsilon) + \epsilon} && \text{(rearrange terms)} \\ &= 1 - \epsilon && \text{(simplify denominator)} \\ \beta \cdot h_\theta^* &= \log \left( \frac{1-\epsilon}{\epsilon} \right) && \text{(invert sigmoid)} \end{aligned}$$

where:

• $h_\theta^*$: the optimal value of the log-ratio difference
• $\epsilon$: the label smoothing parameter

This means cDPO has a finite optimal gap (similar to IPO), which depends on $\epsilon$. With standard DPO ($\epsilon = 0$), the optimal gap is infinite. With cDPO ($\epsilon > 0$), the model converges to a bounded preference strength. The relationship between $\epsilon$ and the optimal gap is intuitive: higher annotation uncertainty (larger $\epsilon$) leads to a smaller optimal gap, because the model should not be confident when the labels might be wrong.

The smoothing parameter $\epsilon$ should reflect actual annotation noise. When inter-annotator agreement data is available, $\epsilon$ can be estimated directly as the disagreement rate. When that data is unavailable, practical guidelines based on dataset characteristics give reasonable starting points:

• $\epsilon = 0.1$: Assumes 90% annotation accuracy, appropriate for high-quality curated datasets with expert annotators.
• $\epsilon = 0.2$: Assumes 80% accuracy, appropriate for crowd-sourced annotations where annotators receive limited instructions.
• $\epsilon = 0.3$: Assumes 70% accuracy, appropriate for noisy or automated annotations generated by another model.

The effective training signal strength scales with $(1 - 2\epsilon)$. At $\epsilon = 0.1$, the signal is 80% as strong as standard DPO. At $\epsilon = 0.3$, it is only 40% as strong. This means that high smoothing values require more training data or more epochs to reach the same degree of preference learning, creating a practical trade-off between robustness to noise and data efficiency.