Iterative Alignment › Summary

Iterative alignment extends single-shot methods like DPO and RLHF into ongoing processes that maintain tight alignment as models and expectations evolve. Key concepts from this chapter:

• Distribution mismatch after training motivates iterative approaches: the model's outputs no longer match the distribution that generated preference data
• Iterative DPO runs multiple training rounds, with either a fixed reference (stable anchor, growing ratios) or rolling reference (stable ratios, cumulative drift)
• Online preference learning generates fresh data from the current policy, keeping training distribution matched but requiring continuous annotation
• Self-improvement through SPIN, Constitutional AI, and LLM-as-judge enables scaling beyond human annotation bottlenecks
• Stability concerns include model collapse, reward hacking accumulation, capability degradation, and subtle value drift
• Monitoring diversity, capability retention, and KL divergence from the original model helps detect problems early
• Practical deployment requires balancing iteration frequency, mixing historical data, incorporating human checkpoints, and maintaining rollback capabilities

Iterative alignment isn't a solved problem; it's an active research area with open questions about training stability and safety. The cost of scaling the process remains another constraint. But the core insight is clear: alignment should be a continuous process, not a one-time event.