Alignment Challenges › Summary

This chapter covered four interconnected alignment challenges that define the current research frontier.

Scalable oversight is the problem of supervising systems that exceed human evaluators' ability to assess their outputs. Proposed solutions include amplification (using AI to help humans evaluate difficult tasks), debate (pitting AI systems against each other to expose errors), and self-critique approaches such as Constitutional AI. None of these is fully resolved, and practical progress has been limited to specific verifiable domains. The weak-to-strong generalization research direction offers some optimism: capable models may be guidable by weaker supervisors in ways that extend beyond the supervisor's explicit capabilities.

The alignment tax is the concern that safety measures reduce capability. Empirically, the picture is mixed: alignment often improves practical helpfulness, but over-refusal is a real problem that reduces utility without improving safety. The resolution lies in distinguishing hard constraints from soft preferences. Hard constraints, where essentially no capability cost is too high, deserve different treatment from soft preferences, where calibration and nuance matter more than blanket restriction.

Goal mis-specification encompasses reward hacking, specification gaming, and inner/outer alignment failures. These arise because formal objectives cannot fully capture intended behavior, and optimization reliably finds the gaps. Reward model overoptimization is empirically observed: gold-standard quality peaks and then declines as optimization pressure continues past the point of beneficial improvement. The fundamental issue is that any proxy for human values will have imperfections, and optimization amplifies those imperfections.

Long-term alignment research includes process supervision, mechanistic interpretability for alignment, eliciting latent knowledge, and frameworks for addressing deceptive alignment. These represent the frontier of research, with significant open problems and no complete solutions. The empirical-theoretical tension in the field is productive when managed well, and the best alignment research combines conceptual clarity with empirical rigor.

The through-line across all these challenges is the same fundamental difficulty: we want AI systems to pursue our actual goals, but we can only specify proxies for those goals. Optimization finds the gaps between proxies and goals. As capabilities increase, so does the ability to find and exploit those gaps. Alignment research is the project of understanding and closing those gaps before they become consequential, and the urgency of that project grows with every capability advance.