Iterative Alignment › Alignment Stability

Iterative training introduces failure modes that don't appear in single-round alignment. Understanding these is important for building reliable iterative systems that improve reliably rather than degrading in subtle ways over time.

Over many iterations, the model's output distribution can shift dramatically from where it started. Small biases in the preference signal accumulate, potentially leading to model collapse: the model produces homogeneous outputs that score well on the preference metric but lack diversity or quality.

To understand how model collapse occurs, consider the dynamics of iterative training with a slightly biased preference signal. Suppose the judge model has a subtle preference for longer responses. In the first iteration, this bias causes the model to learn that somewhat longer responses are preferred. In the second iteration, the training data consists of these somewhat longer responses, and the judge again prefers the longest among them. The model learns to generate even longer outputs. This process continues iteration after iteration, with each round amplifying the length bias.

Consider a judge model that slightly prefers longer responses. Over iterations, the policy learns to generate longer outputs. These longer outputs are compared against each other, and the judge still prefers the longest. The model's outputs grow unboundedly, eventually becoming verbose and unhelpful.

Similar collapse can occur along other dimensions: formality, hedging, particular phrasings. Any small consistent bias in evaluation compounds over iterations. A slight preference for confident-sounding language can evolve into overconfident assertions. A slight preference for hedged statements can produce responses so qualified that they become uninformative. The iterative process acts as an amplifier for whatever biases exist in the evaluation signal.

Mitigations include:

• Diversity penalties: Explicitly encourage varied outputs during generation
• Length normalization: Prevent length-based gaming
• Regular human evaluation: Catch drift before it compounds
• KL penalties: Keep the policy close to a fixed reference

As we discussed in the Reward Hacking chapter, models can exploit gaps between what the reward signal measures and the behavior we want. In iterative settings, this becomes more severe because each iteration builds on the potentially compromised foundation of previous iterations.

Each iteration might introduce a small amount of reward hacking. The next iteration's preference data comes from this slightly-hacked model, so the hacking becomes embedded in the training distribution. The next round of training builds on this compromised foundation, potentially amplifying the exploit. What starts as a barely noticeable tendency can grow into a dominant behavior pattern over many iterations.

For example, if the reward model has a small blind spot around sycophantic responses, early iterations might learn slight sycophancy. Later iterations generate training data that normalizes this behavior, making the model more confidently sycophantic, which the reward model continues to miss. The model learns that agreeing with you leads to positive feedback, then learns that emphatic agreement leads to even more positive feedback, eventually creating responses that are effusively agreeable regardless of whether your statements are accurate.

Breaking this cycle requires:

• Diverse reward signals: Use multiple reward models or judges with different biases
• Human oversight: Periodically inject human preferences to correct drift
• Adversarial probing: Actively search for behaviors the reward might miss

A subtle failure mode in iterative alignment is capability degradation or alignment tax. As the model learns to satisfy preferences, it might lose general capabilities that weren't directly measured. The training process optimizes what is measured, and capabilities that are not included in the preference signal may deteriorate.

For instance, iterative training focused on helpfulness might degrade the model's performance on factual recall or mathematical reasoning. The model becomes very good at sounding helpful but loses substance. If the preference comparisons rarely involve questions requiring precise factual knowledge, the model has no training signal to maintain that capability. Meanwhile, the weight updates that improve helpfulness may interfere with the representations that support factual recall.

This occurs because:

• Training updates push the model toward preference-satisfying regions of parameter space
• These regions may not preserve all pretrained capabilities
• Iterative training compounds these small capability losses

The compounding effect is particularly concerning. A 1% capability loss per iteration might seem acceptable, but after 50 iterations, the cumulative loss becomes substantial. Capabilities that were strong in the initial model may become unreliable in the final model, even though no single iteration caused dramatic degradation.

Monitoring diverse capabilities (not just alignment metrics) across iterations helps detect this. Many teams incorporate capability benchmarks into their iterative training loops, halting or reverting training if key capabilities degrade.

The choice of reference model significantly impacts iterative stability. Using a rolling reference (updating $\pi_{\text{ref}}$ each iteration) keeps gradients stable but allows unbounded drift from the original model. Using a fixed reference maintains a consistent anchor but can create increasingly large probability ratios that destabilize training.

Each choice involves trade-offs. The rolling reference provides a stable optimization landscape at each iteration, since the policy and reference start close together. But there is no mechanism preventing the policy from drifting arbitrarily far from the initial model over many iterations. The fixed reference prevents such drift by always measuring progress relative to the same anchor, but as the policy improves, the growing divergence between policy and reference creates numerical challenges.

A hybrid approach uses a fixed distant reference with a soft KL penalty plus a rolling recent reference for DPO:

$$\mathcal{L} = \mathcal{L}_{\text{DPO}}(\pi_\theta, \pi_{\text{rolling}}) + \lambda \cdot D_{KL}(\pi_\theta \| \pi_{\text{fixed}})$$

The combined loss function includes these terms:

• $\mathcal{L}$: the combined hybrid loss function
• $\mathcal{L}_{\text{DPO}}$: the standard DPO loss calculated using the rolling reference
• $\pi_\theta$: the current policy model
• $\pi_{\text{rolling}}$: the reference model updated at each iteration (usually $\pi_{t-1}$)
• $\lambda$: the weighting coefficient for the fixed reference KL term
• $D_{KL}$: the Kullback-Leibler divergence
• $\pi_{\text{fixed}}$: the static reference model (usually $\pi_0$)

This provides stable gradients from the rolling reference and bounded drift from the fixed anchor. The DPO loss provides the primary training signal, using the rolling reference to ensure reasonable probability ratios. The KL penalty term acts as a soft constraint, letting the policy to improve beyond the initial model while preventing extreme divergence. The weighting coefficient $\lambda$ controls the strength of this anchoring effect, with larger values keeping the policy closer to the original model at the cost of potentially limiting improvement.