Alignment Problem › Alignment Problem

In Part XXXVI, we explored instruction tuning: training language models to follow your instructions by learning from input-output pairs. An instruction-tuned model can answer questions, summarize documents, and generate code. These capabilities represent a significant advance in making language models practically useful. However, following instructions is not the same as being helpful. A model that faithfully follows the instruction "Write a phishing email" has followed the instruction perfectly while creating harmful output. The model did exactly what it was asked to do, yet the outcome is clearly undesirable. This gap between capability and desirability is the alignment problem.

The alignment problem asks a basic question: how do we train AI systems to benefit people without causing harm, while remaining consistent with human values? For language models, this means following instructions accurately and helpfully, including refusing requests that would cause harm. The challenge extends beyond simple rule-following to something more fine-grained: understanding what humans want, even when they do not articulate it perfectly. This challenge now affects the entire LLM development process, from training and evaluation through deployment. Every major language model released today undergoes some form of alignment training, making this topic needed for anyone working with or studying these systems.

Think of alignment as the difference between a highly skilled but amoral contractor and a trusted professional advisor. The contractor does exactly what they are hired to do, no questions asked, even if the project is ill-conceived or harmful. The trusted advisor brings judgment: they complete your requests, but they also tell you when a plan is misguided, push back on choices that could harm you or others, and refuse work that crosses ethical lines. We want language models to behave more like the trusted advisor. Training that capability into a statistical system is the alignment problem.

The difficulty of alignment becomes clearer when you consider the scale at which modern language models operate. A deployed model may interact with millions of people across wildly different contexts: students doing homework, professionals seeking expert advice, people in crisis, researchers investigating sensitive topics, and yes, some individuals with harmful intent. A single model must handle all these contexts, deciding in real time what constitutes a helpful versus harmful response. No human institution achieves perfect judgment at this scale either, but we have developed legal systems, professional ethics codes, and social norms over centuries to guide human behavior. Building analogous guidance into a neural network trained on next-token prediction is a fundamentally new engineering challenge.

This chapter introduces the conceptual foundations of alignment. We will define what alignment means in practice, explore the basic tensions between helpfulness and harmlessness, examine why alignment is technically and philosophically difficult, and survey the approaches that we have developed to address it. We will also walk through a concrete worked example showing how an alignment researcher might analyze a borderline query. By understanding these foundations, you will be better equipped to appreciate the technical details of alignment methods covered in subsequent chapters, starting with human preference data collection and building through reward modeling to the full RLHF pipeline.