Alignment Problem › Summary

The alignment problem addresses the gap between AI capabilities and beneficial behavior. An aligned language model is helpful (provides value), harmless (avoids causing damage), and honest (represents information accurately). These properties often conflict with each other, requiring careful balancing rather than simple optimization. There is no single metric to maximize; good alignment requires judgment about tradeoffs.

Alignment is technically challenging because we cannot fully specify what we want, struggle to measure whether we have achieved it, and worry about generalization to new situations and more capable models. The specification problem, measurement problem, generalization problem, scalability problem, and multi-stakeholder problem each contribute to the difficulty. Current approaches include RLHF for learning from human preferences, constitutional AI for self-improvement based on principles, and red-teaming for discovering failure modes. These approaches complement each other, and effective alignment likely requires combining multiple techniques.

Current limitations, including the proxy optimization problem, the lack of interpretability, the training-deployment gap, and the sociotechnical dimensions of alignment, mean that the field has substantial work ahead. Progress will require advances in both technical methods and in the governance structures that determine how AI systems are deployed and held accountable.

The following chapters will dive deep into the technical machinery of alignment. We will start by exploring how to collect and structure human preference data, then build up through reward modeling, the mathematics of preference learning, and the full RLHF pipeline. Understanding these techniques is needed for working with modern language models, whether training new systems or understanding the behavior of existing ones.