Iterative Alignment › Iterative Alignment

The alignment methods we've explored so far, including RLHF and DPO variants, typically treat alignment as a one-shot process. You collect preferences, train once, and deploy the model. But alignment is an ongoing challenge, not a single event. Models that seem well-aligned can drift as they encounter new situations, and static preference datasets quickly become outdated as the model's behavior evolves. This chapter explores iterative approaches to alignment, where models undergo multiple rounds of training with fresh preference data, sometimes generating that data themselves.

Iterative alignment is necessary because training a model on preference data changes its distribution. Its outputs no longer match the distribution that generated the original training examples. This creates a distribution mismatch that can degrade alignment quality over time. By iteratively collecting new preferences on the model's current outputs and retraining, we can maintain tighter alignment and even enable models to improve beyond what any single training run could achieve.