Alignment Problem › Approaches to Alignment

Reinforcement Learning from Human Feedback (RLHF) is currently the dominant alignment technique for large language models. The approach turns human preferences into training signal, letting models to learn complex notions of quality that are difficult to specify explicitly. Rather than writing rules about what constitutes a good response, RLHF lets human judgment implicitly define quality through the patterns of which responses humans prefer.

The approach involves three main stages:

1. Collecting human preference data: Humans compare model outputs and indicate which they prefer
2. Training a reward model: A neural network learns to predict human preferences
3. Optimizing the policy: The language model is fine-tuned using reinforcement learning to maximize predicted human preference

RLHF allows learning complex preferences that are hard to specify explicitly. Rather than writing rules about what makes a response good, we show the model examples of what humans prefer. The reward model learns to predict these preferences, and then the language model learns to produce outputs that the reward model rates highly. This approach has proven effective at making models more helpful and less harmful.

The key insight behind RLHF is that human judgment can evaluate quality along many dimensions simultaneously without decomposing that judgment into explicit rules. When a human rater says they prefer response A over response B, that preference can jointly encode the HHH criteria alongside fluency and other factors. The reward model learns to capture this joint judgment, and the language model learns to satisfy it. We will explore RLHF in depth in the upcoming chapters, starting with human preference data collection and building through reward modeling to the full training pipeline.

Direct Preference Optimization (DPO) is a more recent approach that achieves similar goals to RLHF without explicit reward modeling. The key motivation is simplifying the training pipeline: RLHF requires training a separate reward model and then using reinforcement learning, which introduces complexity and instability. DPO directly optimizes the language model using preference data, bypassing these intermediate steps.

The key insight behind DPO is that the optimal policy under certain reward model assumptions can be expressed in closed form. This mathematical insight allows direct optimization via supervised learning, bypassing the need for a separate reward model and reinforcement learning loop. DPO simplifies the training pipeline while often achieving comparable results to full RLHF. The reduction in training complexity makes DPO more accessible to researchers and organizations without the resources to run full RLHF pipelines.

DPO and its variants have become important tools in the alignment practitioner's toolkit. The mathematical relationship between DPO and RLHF also provides theoretical insight: by understanding why DPO works, we learn something about the RLHF objective. This kind of theoretical clarity is valuable for predicting when these methods will and will not work, and for designing improvements. We will cover the mathematics of DPO in detail in a later chapter.

Constitutional AI (CAI) provides the model with explicit principles and trains it to follow them. The approach addresses a limitation of RLHF: the need for human feedback on every type of situation the model might encounter. Instead, CAI teaches models to evaluate their own outputs against stated principles. The approach involves:

1. Defining a "constitution" that states how the model should help users, avoid harm, and communicate truthfully
2. Generating model critiques of its own outputs based on these principles
3. Training on improved outputs that better follow the constitution

CAI reduces dependence on human feedback for every type of harmful content, instead teaching the model to self-evaluate and improve. This is more scalable than pure RLHF and can help with novel situations that weren't covered in training data. The constitutional approach also has the advantage of making alignment principles explicit and auditable: you can read the constitution and understand what values the model is trained to embody. This transparency is valuable for accountability and trust.

The idea of a "constitution" for AI behavior draws on an analogy to constitutional law: just as a constitution provides principles that constrain how more specific laws are interpreted, an AI constitution provides principles that constrain how the model interprets and responds to specific requests. The analogy is imperfect, but it captures the aspiration for principled, consistent behavior grounded in explicit values rather than ad-hoc rules.

Red-teaming involves deliberately trying to make models fail. Dedicated teams attempt to elicit harmful outputs through adversarial prompting, discovering failure modes before deployment. This is a important part of alignment evaluation, because the goal is to identify failures before users encounter them. The name comes from military exercises where a "red team" plays the role of adversaries to test defenses.

Red-teaming serves multiple purposes:

• Discovering vulnerabilities that training missed
• Generating training data for addressing specific failure modes
• Evaluating alignment before and after interventions
• Building understanding of how models can be misused

Effective red-teaming requires creativity and expertise. Red team members must think like adversaries, anticipating the techniques that bad actors might use. The findings from red-teaming inform both immediate patches and longer-term research directions. Red-teaming has become an industry standard practice before major model releases, with organizations dedicating significant resources to it. Some companies also run external red-team programs, inviting security researchers to probe for vulnerabilities in exchange for recognition or compensation.

Automated red-teaming, where language models are used to generate adversarial prompts against other language models, is an increasingly important complement to human red-teaming. Human red-teamers are expensive and slow, and they may not think of every possible attack vector. Automated approaches can generate large numbers of diverse attacks quickly, though they may miss the creative and context-sensitive attacks that human red-teamers excel at. Combining human and automated red-teaming is the current best practice.

Rather than aligning a fully capable model, capability control limits what the model can do. This is a defense-in-depth approach: even if alignment fails, limiting capabilities bounds the potential harm. Approaches include:

• Output filtering: Blocking specific types of content post-generation
• Input filtering: Rejecting certain queries before generation
• Knowledge removal: Training models without certain dangerous information
• Tool restrictions: Limiting what external tools the model can access

Capability control is a complementary defense layer rather than a complete solution. It addresses the concern that alignment may fail in some cases by so that failures are less consequential. However, capability control alone is insufficient because it reduces the model's usefulness along with its potential for harm. The most effective implementations of capability control are targeted: they remove specific dangerous capabilities while preserving as much general usefulness as possible.

One important application of capability control is "information hazard" management: some information, like detailed synthesis routes for chemical or biological weapons, is dangerous enough that no legitimate use case justifies a model being able to provide it. For these narrow categories of catastrophic risk, capability control through training-time knowledge removal or inference-time filtering provides an important safety layer that complements alignment training.