Iterative Alignment › Iterative DPO

As we discussed in the DPO chapters, Direct Preference Optimization implicitly defines a reward function through the log-probability ratio between the policy and reference model. This mathematical relationship makes the reference model central to what DPO learns. The reference model is an anchor point that defines the baseline against which improvements are measured. When we iterate DPO, we have a choice: keep the original reference model fixed, or update it at each iteration. This choice significantly affects training dynamics and the final model's behavior.

The simplest form of iterative DPO keeps the original reference model $\pi_{\text{ref}}$ fixed across all iterations. At each round $t$, we generate responses from the current policy $\pi_t$, collect preferences, and train to get $\pi_{t+1}$. This approach maintains a consistent anchor point throughout the entire iterative process, always measuring progress relative to where the model started.

The loss at iteration $t$ becomes:

$$\mathcal{L}^{(t)} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}^{(t)}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \right]$$

Unpacking the formula components clarifies how they guide iterative training:

• $\mathcal{L}^{(t)}$: the loss function minimized at iteration $t$
• $\mathbb{E}$: the expectation operator averaging over all preference pairs in the dataset
• $\mathcal{D}^{(t)}$: the dataset of preferences collected from responses generated by the current policy $\pi_t$
• $\pi_\theta$: the policy model currently being trained (parameterized by $\theta$)
• $\pi_{\text{ref}}$: the fixed reference model (usually the initial policy)
• $x$: the input prompt
• $y_w, y_l$: the winning and losing responses, respectively, for prompt $x$
• $\sigma$: the logistic sigmoid function
• $\beta$: the temperature parameter controlling the strength of the KL penalty
• $\beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$: the implicit reward term (higher values indicate the policy is more likely to generate $y$ than the reference)

The key change from standard DPO is that $\mathcal{D}^{(t)}$ contains preferences over responses generated by $\pi_t$, not the original reference model. This subtle but important difference ensures the model learns from comparisons that reflect its current capabilities rather than from stale comparisons between outputs it no longer produces. The model receives feedback on the kinds of responses it generates now. This makes the training signal directly relevant to improving current behavior.

However, keeping a fixed reference creates a growing gap between the reference and policy distributions. As the policy improves over iterations, it diverges further from $\pi_{\text{ref}}$. The implicit reward function $r(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ now compares very different distributions. When the policy assigns high probability to a response that the reference considers extremely unlikely, the log ratio becomes very large. Conversely, when the policy considers a response unlikely but the reference assigns moderate probability, the log ratio becomes very negative. These extreme values can destabilize training by creating large gradients that push the model erratically.

A more common approach updates the reference model at each iteration. After training round $t$, we set $\pi_{\text{ref}}^{(t+1)} = \pi_t$ before starting round $t+1$. This keeps the reference close to the current policy, maintaining stable gradients and so that the log probability ratios remain in a reasonable range throughout training.

The rolling reference focuses on incremental improvements instead of measuring progress against the initial model. By using the previous iteration's policy as the reference, we are essentially asking: "Given where we are now, which responses represent improvements?" This framing naturally prevents the extreme probability ratios that arise when comparing against a very different reference distribution.

The iterative procedure becomes:

1. Initialize $\pi_0$ with a supervised fine-tuned model
2. Set $\pi_{\text{ref}}^{(1)} = \pi_0$
3. For each iteration $t = 1, 2, \ldots, T$:
   • Generate responses from $\pi_{t-1}$ on a set of prompts
   • Collect preferences over response pairs
   • Train $\pi_t$ using DPO with reference $\pi_{\text{ref}}^{(t)}$
   • Update $\pi_{\text{ref}}^{(t+1)} = \pi_t$

This rolling reference approach is sometimes called online DPO because the training data comes from the current policy rather than a static offline dataset. The term "online" here refers to the fact that data generation is interleaved with training, creating a dynamic feedback loop between model behavior and the training signal.

Rolling reference training starts from a neutral position. The policy and reference are identical at the beginning of each round, so the log probability ratios start at zero for all responses. This means the training signal comes entirely from the preference comparisons in the current iteration's dataset, without any accumulated bias from previous iterations. The model learns to improve relative to its immediate predecessor, and the cumulative effect of many such improvements can be substantial.

Recall from our discussion of DPO variants that Identity Preference Optimization (IPO) avoids the length exploitation issues that can plague DPO over multiple iterations. The problem with standard DPO becomes particularly acute in iterative settings: if the model learns that slightly longer responses tend to win preference comparisons, it will generate longer responses, which will then be compared against each other, and the longest will again tend to win. This feedback loop can drive response length to extreme values over many iterations.

IPO addresses this concern through a fundamentally different objective formulation. Rather than using the sigmoid function to bound the preference probability, IPO directly targets a specific margin between the preferred and dispreferred log probability ratios. The IPO objective is:

$$\mathcal{L}_{\text{IPO}} = \mathbb{E}_{(x, y_w, y_l)} \left[ \left( \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} - \frac{1}{2\beta} \right)^2 \right]$$

The components of the IPO formula prevent the runaway behavior seen in DPO:

• $\mathcal{L}_{\text{IPO}}$: the Identity Preference Optimization loss
• $\mathbb{E}$: the expectation operator averaging over the dataset
• $\pi_\theta$: the policy model being trained
• $\pi_{\text{ref}}$: the reference model
• $x$: the input prompt
• $y_w, y_l$: the preferred and dispreferred responses for prompt $x$
• $\beta$: the hyperparameter derived from the KL regularization strength

This objective minimizes the squared error between the log-likelihood ratio gap and a finite target margin of $1/(2\beta)$. The key insight is that IPO does not simply try to make the preferred response more likely than the dispreferred one; it tries to make the gap between them equal to a specific target value. Once the model achieves this target margin, further increasing the gap raises the loss. This built-in saturation prevents the model from pushing probability ratios to extreme values.