Iterative Alignment › The Case for Iteration

Consider what happens in standard DPO training. You start with a reference model $\pi_{\text{ref}}$, collect human preferences comparing pairs of responses, and train the model to prefer the winning responses. After training, you have a new model $\pi_{\theta}$ that behaves differently from $\pi_{\text{ref}}$. But your training data was generated from $\pi_{\text{ref}}$, so it reflects the kinds of responses that model produced and the relative quality judgments between them.

To understand why this matters, imagine teaching someone to write by showing them examples of mediocre essays and asking them to identify which is slightly better. They might learn to avoid the worst mistakes, but they never see what excellent writing looks like. The training signal is fundamentally limited by the quality of the examples being compared. When your model improves beyond the capability level present in the training data, the preference comparisons become less informative because they no longer represent the kinds of decisions the improved model needs to make.

This creates several problems:

• Distribution mismatch: The preference comparisons were between responses from $\pi_{\text{ref}}$, but the model now generates different responses. The training signal becomes less relevant.
• Ceiling effects: If both responses in a preference pair are poor, learning to prefer the "less bad" option doesn't teach the model what a truly good response looks like.
• Static feedback: Human values and expectations evolve. A dataset collected once becomes stale.

Distribution mismatch is the primary reason why iteration matters. When you train a model to prefer response A over response B, you are implicitly teaching it something about the boundary between good and bad responses in that region of output space. But after training, the model no longer generates responses in that same region. It has moved to a different part of the output distribution, where the old preference comparisons may not provide useful guidance. The model might now be generating responses that are uniformly better than anything in the original training set, or it might be making new types of errors that the original comparisons never addressed.

Iterative alignment addresses these issues by making alignment a continuous process. After each training round, we generate new responses from the updated model, collect fresh preferences, and train again. This keeps the training distribution matched to the current policy and allows the model to learn from increasingly sophisticated comparisons. Each iteration provides feedback on the model's actual current behavior, creating a feedback loop that can continuously refine alignment quality.