Iterative Alignment › Limitations and Impact

Iterative alignment represents a significant advance over single-shot training but introduces its own challenges. The computational cost multiplies with each iteration: you need to generate new data, potentially annotate it, and train again. For large models, this becomes expensive quickly.

The self-improvement approaches, while promising, have limits. A model cannot improve beyond what its evaluation criteria capture. If the judge model (whether human or AI) has blind spots, iterative training amplifies rather than corrects them. Constitutional AI partially addresses this with explicit principles, but those principles themselves may be incomplete or contradictory.

Perhaps most concerning is the possibility of subtle value drift. Small consistent biases in the preference signal, whether from the reward model, the AI judge, or sampling strategies, compound over iterations. After many rounds of training, the model might satisfy measured preferences while diverging from underlying human values in ways that are difficult to detect until something goes wrong.

These challenges don't diminish the importance of iterative alignment; they highlight the need for careful monitoring, diverse evaluation, and appropriate human oversight. As models become more capable, the alignment process must become more sophisticated. Single-shot training may suffice for simple tasks, but truly aligned AI systems likely require continuous, iterative refinement guided by ongoing human feedback and evaluation.

The upcoming chapters on inference optimization cover the practical machinery needed to deploy these iteratively-aligned models efficiently. KV caching and quantization reduce inference costs, while efficient attention helps the model handle longer contexts. Together, these techniques make it feasible to run aligned models at scale while maintaining the quality achieved through careful iterative training.