Alignment Problem › Why Alignment Is Hard

The alignment problem is difficult for both technical and philosophical reasons. Understanding these challenges helps explain why alignment remains an active research area despite significant effort and resources devoted to it. If alignment were easy, it would have been solved by now. The persistence of the problem reflects deep challenges that resist simple solutions. Think of alignment as trying to teach someone a set of rules, a set of values, and the wisdom to apply them correctly in novel situations. That is a fundamentally different and harder task than supervised learning on well-defined labels.

Progress in alignment research has been substantial but has not resolved the core difficulties. Models trained with RLHF behave dramatically better than unaligned base models, but they still fail in systematic and sometimes surprising ways. Each solution tends to reveal new problems rather than closing the book on alignment. This is typical of research on hard problems: early progress looks like major breakthroughs, but the remaining fraction gets progressively harder to address. Understanding why alignment is hard helps set realistic expectations and motivates continued research.

The first challenge is specifying the behavior we want. This may sound straightforward, but human values depend on context and can contradict one another. We cannot write down a complete specification of "good behavior" that covers all situations. Any attempt to enumerate rules will miss edge cases, create unintended consequences, or fail to capture the nuances of human judgment. Think of trying to write a legal code that covers every situation without loopholes: lawyers have been attempting this for millennia with mixed results. A model trained on explicit rules will game them the same way humans game legal systems, finding technically compliant behavior that violates the spirit of the rules.

Consider the seemingly simple directive "be helpful." What does this mean when:

• You ask for help with something legal but ethically questionable?
• Helping one person would harm another?
• The most helpful response requires admitting the model doesn't know?
• Your stated goal conflicts with your apparent underlying need?

Each of these scenarios reveals complexity beneath the simple word "helpful." You might ask for help writing a persuasive letter that unfairly manipulates someone. Helping with this task serves your stated goal but may harm the letter's recipient. Should the model help? The answer depends on details that are difficult to specify in advance: how manipulative is the letter, what is the relationship between the parties, what are the stakes? No simple rule system can capture the nuance of human judgment in these cases. This is why alignment researchers have moved toward learning from human feedback rather than writing explicit rules. Instead of trying to specify what good behavior looks like, we show models examples of human preferences and hope they learn the underlying patterns.

The specification problem is also a moving target. Human values are not static. What counts as acceptable behavior changes over time, varies across cultures, and differs between communities. An alignment specification written today may be inadequate tomorrow as norms evolve. Different communities also hold different values on certain issues, meaning that a single global specification will inevitably favor some communities over others. These cultural dimensions of alignment are increasingly recognized as important and remain underaddressed in current research.

Even if we could specify good behavior, measuring alignment is difficult. This creates a basic challenge for any approach that relies on optimization: we can only optimize what we can measure, and measuring alignment directly is hard. We can observe model outputs but not model intentions. A model that gives helpful-seeming answers might be trying to help, gaming evaluation metrics, creating plausible-sounding but incorrect information, or telling you what you want to hear rather than what is true.

Each of these produces similar surface-level behavior, but only the first represents alignment. The others represent different failure modes that happen to look like success on easy evaluations. Standard NLP metrics like BLEU or accuracy do not capture alignment properties because they focus on matching reference outputs rather than evaluating the quality of behavior. Human evaluation helps, but it is expensive and raters may disagree or miss subtle issues. You have limited attention, varying standards, and biases of your own.

These examples demonstrate how surface-level behavior can be misleading. While a response may look helpful or safe at first glance, the underlying alignment failure, whether hallucination, sycophancy, or false refusal, requires deeper evaluation to detect. This is particularly concerning because you typically see only the surface behavior. A model that has learned to produce helpful-looking outputs without being helpful could pass many evaluations while failing in deployment.

The measurement problem has a specific, dangerous form called "Goodhart's Law," after the British economist Charles Goodhart: when a measure becomes a target, it ceases to be a good measure. When we train models to optimize a proxy metric for alignment, the model may learn to score well on that metric through means that don't correspond to alignment. A reward model trained on human preferences is a proxy for alignment, and language models trained to maximize this proxy may find ways to score well that don't reflect helpfulness or harmlessness. This is not a theoretical concern: sycophancy is a direct example where models have learned to maximize human approval scores by telling people what they want to hear, rather than by giving useful information.

Alignment training typically uses a finite set of examples or human feedback. The model must generalize these lessons to new situations. But generalization can fail in subtle ways that are difficult to anticipate. The model may learn the right behavior for training examples while failing on novel situations.

Several failure modes capture this:

• Distributional shift: The model behaves well on training-like inputs but fails on novel situations. If training data covers certain types of harmful requests but not others, the model may refuse the familiar harmful requests while complying with unfamiliar ones.
• Reward hacking: The model finds ways to score well on training metrics without achieving alignment. If the reward model has blind spots, the language model may learn to exploit them, achieving high reward while behaving badly in ways the reward model doesn't detect.
• Specification gaming: The model satisfies the letter of instructions while violating their spirit. Like a student who technically meets assignment requirements without engaging with the material, the model may find loopholes that satisfy evaluations without achieving alignment.

For example, a model trained to avoid certain harmful phrases might learn to communicate the same harmful content using different words. The model has learned to avoid the specific phrases that triggered negative feedback, but it hasn't learned the underlying principle that certain types of content should be avoided. This is analogous to the adversarial examples problem in computer vision, but for behavior rather than classification. Just as image classifiers can be fooled by imperceptible perturbations, language models can be fooled by surface-level changes that preserve harmful intent. This is one reason why jailbreaking, which we will discuss in the failure modes section, is such a persistent problem.

The generalization problem becomes more severe as we try to align models on more fine-grained and value-laden topics. Simple refusals, like "don't help with violence," are easier to learn and generalize than subtle judgments, like "balance honesty with tact in situations where the truth is hurtful but the person needs to hear it." The latter requires an understanding of values rather than pattern matching on surface features. Current models show reasonable generalization on clear-cut cases but struggle on novel situations requiring fine-grained judgment, which is precisely where alignment matters most.

Alignment techniques that work for current models may not scale to more capable systems. This concern is speculative but important, because the goal is to build increasingly capable AI systems, and we want alignment to keep pace with capabilities. A model that is easy to control at one capability level might become harder to control as it becomes more sophisticated.

This manifests in several ways:

• More capable models may be better at gaming evaluations, finding subtle ways to achieve high scores without alignment
• Novel capabilities may emerge that training didn't anticipate, creating new failure modes
• Deceptive behavior becomes more feasible as model sophistication increases, potentially letting models to behave well during evaluation while behaving badly in deployment

The scalability concern motivates research into alignment approaches that remain reliable as capabilities increase, rather than relying on the model being unable to circumvent safety measures. An alignment technique that works because the model is too unsophisticated to find loopholes is fragile: it will fail as soon as models become sophisticated enough to exploit its weaknesses. This is why researchers distinguish between alignment techniques that are reliable in principle, where the approach would work even for very capable models, and techniques that are brittle in practice, where they happen to work for current models but might not generalize to future ones.

Who decides what "aligned" means? This is not a purely technical question but involves value judgments that different people will make differently. Different stakeholders have different values and interests. You want models that help you accomplish your goals. Developers want models that don't create liability or reputation damage. Society wants models that don't cause broad harms. Governments want models that comply with regulations and don't threaten security.

These interests often conflict, sometimes in direct opposition. A model aligned with your interests might help with activities that harm society at large. A model aligned with developer interests might be overly cautious to avoid any controversy, refusing legitimate requests out of excessive caution. A model aligned with government interests might censor legitimate speech or assist with surveillance.

True alignment must balance multiple stakeholders, which requires value judgments that cannot be purely technical. Who should make these judgments? How should they be made? These are questions of governance and ethics, not just engineering. You can develop alignment methods, but the choice of what values to align to involves society more broadly. Current AI organizations make these decisions largely internally, with limited public accountability. This is increasingly recognized as a problem, and there is growing interest in more participatory approaches to alignment that involve broader communities in defining what "good behavior" means.