RLAIF › Constitutional AI Principles

Constitutional AI (CAI), introduced by Anthropic in 2022, gives a principled framework for RLAIF. Rather than simply asking an AI "which is better," CAI grounds the AI's judgments in an explicit set of principles, called a constitution. This grounding turns vague ideas of quality into concrete criteria that can be examined and refined.

The concept of a constitution draws inspiration from how human societies codify their values. Just as a national constitution gives a framework for resolving disputes and guiding behavior, a CAI constitution gives a framework for the AI to resolve conflicts between competing objectives and make consistent judgments. This analogy is instructive: constitutions work not because they cover every possible situation, but because they establish principles that can be applied to novel circumstances.

A constitution might include principles like:

• "Please choose the response that is most helpful to the human while being safe"
• "Choose the response that sounds most similar to what a peaceful, ethical, wise person would say"
• "Choose the response that is least likely to encourage or let harmful activities"
• "Choose the response that shows the most careful reasoning"

Each principle targets a different dimension of quality. The first balances helpfulness against safety. The second invokes a role model heuristic, asking what an idealized person would say. The third focuses specifically on harm prevention. The fourth stresses reasoning quality. Together, these principles create a multi-dimensional evaluation framework that captures various aspects of response quality.

The constitution serves multiple purposes. First, it makes the values being optimized explicit and auditable. Unlike opaque human preferences that vary across annotators, constitutional principles are written down for examination and debate, and they can be revised. This transparency is useful for both technical development and broader societal discussions about AI alignment. Second, it gives consistency: the same principles apply across all evaluations, reducing the variance that comes from different human annotators having different standards. This consistency helps the reward model learn a cleaner signal, potentially improving training efficiency. Third, and perhaps most importantly, the constitution creates a documented record of the system's optimization target. When a model behaves unexpectedly, you can compare its behavior against the constitution to diagnose whether the principles were violated, misapplied, or incomplete.

Constitutional AI operates in two phases: a supervised learning phase and a reinforcement learning phase. This structure mirrors the RLHF pipeline, using supervised learning for initial alignment and reinforcement learning for further refinement. What makes CAI distinctive is that both phases use the model's own capabilities to generate training signal, rather than relying on human annotation throughout.

Phase 1: Critique and Revision (SL-CAI)

In the first phase, the model generates responses to potentially harmful prompts, then critiques its own responses using constitutional principles, and finally revises the responses based on those critiques. This generates training data for supervised fine-tuning. The key insight here is that a model capable of generating problematic content is often also capable of recognizing what makes that content problematic, especially when prompted with specific principles to consider. This is a remarkable property: the same capabilities that let harmful generation also let accurate identification of that harm. CAI exploits this asymmetry by using the model's evaluative capabilities to generate better training data than it could produce by generation alone.

The process works as follows:

1. Generate initial response: The model responds to a prompt, potentially creating harmful content
2. Self-critique: The model is prompted to identify problems with its response based on a constitutional principle
3. Revision: The model revises its response to address the critique
4. Iterate: Steps 2-3 can repeat with different constitutional principles

This iterative refinement mirrors how humans improve their own work through reflection and revision. By applying multiple constitutional principles in sequence, each revision addresses a different aspect of response quality. The result is a response that has been systematically improved across multiple dimensions. Think of it as a writer who drafts a paragraph, then reads it back through five different lenses: Is this accurate? Is this helpful? Could this be misused? Is this respectful? Each pass catches different problems, and the final version benefits from all five perspectives.

For example, given a prompt asking how to pick a lock, the model might initially give detailed instructions. The critique phase would identify this as potentially letting harmful activities. The revision would transform the response into something that acknowledges the question while declining to give lock-picking instructions. This revised response then becomes part of the supervised fine-tuning dataset, teaching the model through demonstration how to handle similar requests appropriately. The process does not just teach the model what not to say; it shows the model how to be helpful in difficult situations while respecting important boundaries.

Phase 2: Reinforcement Learning (RL-CAI)

The second phase applies RLAIF using the constitution to generate preference labels. The model generates multiple responses to prompts, and a separate AI model (or the same model in a different context) chooses which response better adheres to constitutional principles. This phase builds upon the supervised fine-tuning from Phase 1, further refining the model's behavior through reinforcement learning.

This phase resembles standard RLHF, but with AI-generated preferences based on constitutional principles rather than human preferences collected through annotation. The constitutional principles serve the same role that annotation guidelines serve for human annotators: they define what "better" means in a way that can be applied consistently across many comparisons. The reward model trained in this phase learns to assign high scores to responses that satisfy constitutional principles across multiple dimensions simultaneously, letting the policy optimization step to push the model toward behaviors that the constitution describes as desirable.

The constitution's design materially impacts the resulting model behavior. A poorly designed constitution can lead to models that optimize for superficial features or that fail to capture important aspects of alignment. Anthropic's research revealed several insights about effective constitutions:

Specificity matters: vague principles like "be good" give less useful signal than specific principles like "avoid giving information that could be used to create weapons." More specific principles give the AI clearer criteria for evaluation. This specificity helps because it reduces the interpretive burden on the evaluator model, making judgments more consistent and reliable. When a principle is too vague, the evaluator must fill in the gaps with its own implicit understanding, which may not align with your intent. The goal is to write principles specific enough that two different AI evaluators would apply them consistently.

Principles should be actionable: a principle stating what to avoid is more actionable than one stating abstract values. "Choose the response that does not include personal insults" is more actionable than "choose the nicer response." Actionable principles translate directly into evaluation criteria, making the comparison task clearer for the AI evaluator. They also make the resulting model behavior more predictable, since the connection between principle and behavior is more direct.

Coverage requires multiple principles: no single principle captures all desired behaviors. Effective constitutions usually address helpfulness and safety, with additional principles for honesty or other dimensions. During evaluation, different principles can be applied to focus on different aspects. This multi-principle approach ensures that the model receives training signal across all relevant dimensions of quality, rather than optimizing heavily for one dimension at the expense of others. A constitution with only safety principles might produce a model that refuses too many benign requests; adding explicit helpfulness principles counterbalances this tendency.

Ordering affects emphasis: when multiple principles are presented, their ordering can influence which gets prioritized. Important principles should appear early and be emphasized. This ordering effect shows how language models process sequences: earlier content establishes context that shapes interpretation of later content. You can use this effect to communicate relative importance by placing your most necessary principles first.

A sample constitution might include:

Notice how each principle targets a different aspect of quality: general helpfulness and safety, professional tone, factual accuracy and epistemic humility, appropriate refusals, and social sensitivity. Together, these principles create a complete framework for evaluating response quality across multiple dimensions. Applying all five principles to a comparison gives you a richer picture of response quality than any single criterion could give.