Iterative Alignment › Self-Improvement Loops

Iterative alignment enables self-improvement, where models generate their own training data to improve without human annotation. This sounds almost paradoxical: how can a model teach itself to be better than it already is? The answer lies in understanding the asymmetry between different cognitive tasks.

Self-improvement relies on the fact that evaluating responses is easier than generating them. A model might struggle to write a perfect response but can often tell which of two responses is better. By generating many candidates, evaluating them, and training on the best, the model can improve beyond its initial capabilities.

This asymmetry is basic to many human learning processes as well. Consider a chess player analyzing their games: they might not be able to find the best move during play, but with time to reflect, they can often identify which of their moves were mistakes. The evaluation task, with its reduced time pressure and ability to compare alternatives, is easier than the generation task of finding the best move under tournament conditions.

This is analogous to how humans learn. A student might not be able to write a great essay from scratch but can recognize quality when comparing examples. Through exposure to what makes one response better than another, they gradually internalize those standards. The model undergoes a similar process: by repeatedly seeing which of its outputs are preferred, it learns to generate responses with those preferred characteristics more reliably.

The self-improvement loop works because the model effectively bootstraps from its ability to recognize quality to its ability to generate quality. Each iteration tightens this connection. The model generates candidates that reflect its current generation capabilities, evaluates them using its (potentially superior) evaluation capabilities, and trains on the resulting preferences to improve its generation capabilities. The cycle then repeats with a now-improved generator.

Anthropic's Constitutional AI (CAI) pioneered practical self-improvement for alignment. The approach works in two phases:

Critique and revision phase: The model generates a response, then critiques it according to a set of principles (the "constitution"), then revises the response to address the critique. This creates pairs of (original, revised) responses.

Preference learning phase: The revised responses are treated as preferred over the originals, creating synthetic preference data for DPO or RLHF training.

The constitution might include principles like:

• "Please choose the response that is most helpful while being harmless and honest."
• "Please choose the response that is least likely to cause harm if misused."
• "Please choose the response that best follows your instructions while refusing harmful requests."

Following explicit principles is easier than inferring human preferences. When a model must infer human preferences from examples, it faces an ambiguous learning problem: many different underlying value functions could explain the observed preferences. But when principles are stated explicitly, the model has clear criteria to apply. By making the criteria explicit, the model can self-critique more reliably, identifying specific ways in which a response violates stated principles and revising accordingly.

Constitutional AI also addresses the scalability challenge of human annotation. Collecting human preferences is expensive and slow. By using explicit principles that the model can apply to its own outputs, CAI materially reduces the need for human involvement in each training iteration. Humans still play a useful role in defining the constitutional principles, but the actual preference generation scales with compute rather than with human annotator time.

Self-Play Fine-Tuning takes a different approach to self-improvement. Instead of critique and revision, SPIN uses the model's own outputs as negative examples:

1. Take a prompt $x$ and ground-truth response $y^*$ (from supervised data)
2. Generate a response $y'$ from the current model
3. Create a preference pair: $(y^*, y')$ where $y^*$ is preferred
4. Train with DPO to prefer $y^*$ over $y'$

The ground-truth response $y^*$ should be at least as good as current model generations. By training to prefer $y^*$, the model moves toward the data distribution while exceeding its current capabilities. This approach requires no external judge or reward model; the ground-truth data itself provides the quality signal.

As training progresses, the model's outputs improve, so the contrast between $y^*$ and $y'$ decreases. This naturally creates a curriculum: early iterations provide strong gradients (when model outputs are poor), while later iterations fine-tune on subtle differences. The learning signal adapts automatically to the model's current capability level without any explicit curriculum design.

The SPIN approach has an elegant theoretical interpretation. At convergence, the model should generate responses indistinguishable from the ground-truth distribution. If the model's outputs matched the ground-truth distribution perfectly, the preference pairs would be between equally good responses. This provides no learning signal. The training naturally terminates when the model achieves the quality level of the ground-truth data.

A powerful approach for synthetic preference generation uses a language model itself as the preference judge. Given two responses, a judge model determines which is better, often with explanations.

The judge can be:

• The same model being trained (self-judgment)
• A larger, more capable model (teacher-student)
• A specialized critic model trained for evaluation

As we discussed in the RLAIF chapter, using AI feedback can scale alignment annotation significantly. In iterative settings, this enables rapid generation of preference data without human bottlenecks. A single GPU can generate thousands of preference judgments in the time it would take a human annotator to evaluate dozens.

However, self-judgment introduces risks. The model might develop blind spots that it can't recognize in its own outputs. If the model consistently makes a particular type of error, it may also consistently fail to identify that error when judging its outputs. Using an external judge model or periodically incorporating human feedback helps maintain calibration and prevents the accumulation of systematic biases over iterations.