Alignment Problem › Limitations of Current Alignment Approaches

Despite substantial progress, current alignment approaches face important limitations that prevent them from fully solving the alignment problem. Understanding these limitations is needed for anyone working in this space, both to set realistic expectations and to identify the most important directions for future research.

The most basic limitation is that we are optimizing for proxies of alignment rather than alignment itself. Reward models trained on human preference data capture something real about what humans want, but they are imperfect proxies. Human raters have biases, make inconsistent judgments, and may prefer responses that seem good on the surface without deeply evaluating their accuracy or long-term consequences. A language model trained to maximize reward model scores is optimizing for what looks good to human raters, which is related to but not identical to satisfying the HHH criteria. As models become more capable at finding ways to score well on reward models, the gap between proxy optimization and alignment may grow.

A second important limitation is the lack of interpretability. We train models with RLHF and observe that their behavior improves, but we do not understand mechanistically why. We cannot look inside the model and verify that it has learned the right values rather than learned to mimic the surface features of aligned behavior. This "black box" nature of current models makes it difficult to provide strong guarantees about alignment, to predict failure modes in advance, or to diagnose and fix specific alignment failures in a targeted way. The field of mechanistic interpretability is working to address this limitation, but current tools are far from sufficient for the task. True confidence in alignment would require being able to inspect a model's internal representations and verify that they encode the intended values, which remains a distant goal.

A third limitation is the training-deployment gap. Alignment training occurs in controlled conditions with carefully curated examples, but deployment involves the full diversity of human requests across contexts that training may not have anticipated. Models may behave well on the distribution of inputs seen during training while failing on distribution shifts that occur in deployment. This is a general problem in machine learning, but it is particularly acute for alignment because the stakes of failure are high and because adversarial users actively probe for distribution shifts that break safety training. Red-teaming helps address this limitation, but it cannot fully characterize the distribution of deployment inputs in advance.

Finally, alignment is a sociotechnical challenge as well as a technical one. The best alignment training cannot substitute for accountable governance, appropriate use cases, and good deployment practices. A model trained to satisfy the HHH criteria can still be misused in deployment contexts that are fundamentally incompatible with these values. Alignment research therefore needs work on policy and law alongside organizational changes that address the broader sociotechnical system in which language models operate. This broader perspective is sometimes missing from technically focused alignment research, and its absence leads to overconfidence in purely technical solutions.