Constitutional AI: Principles, Critique, Revision

Training large language models to be helpful is relatively straightforward: reward them for responses that users like. But "what users like" is not the same as "what is safe or beneficial." Users sometimes prefer confident misinformation over cautious accuracy. They may prefer flattering but harmful advice over honest but uncomfortable guidance. Early RLHF pipelines discovered that optimizing for human approval ratings, at scale, could produce models that were superficially polished but willing to assist with highly dangerous tasks when prompted cleverly.

Constitutional AI (CAI), introduced by Anthropic in 2022, addresses this misalignment between immediate approval and reliable helpfulness. Instead of relying entirely on human raters to catch harmful outputs, CAI provides the model with an explicit set of principles, a "constitution," and then uses the model itself to evaluate and revise its own responses against those principles. The result is a training loop that embeds values directly into the revision and preference-labeling steps, reducing dependence on humans for the most sensitive judgments.

This chapter examines the design of CAI in depth: what a constitution is and how it works, the critique-and-revision loop that produces supervised training data, the preference generation stage that replaces human labelers, the combined CAI training objective, and the empirical evidence for whether this approach produces safer and more capable models. We build on the RLHF foundations covered in earlier chapters and look ahead to how process reward models (covered next in Part LXVI) extend these ideas.

To understand why Constitutional AI was designed the way it was, it helps to recall how standard RLHF works and where it breaks down. The problem is not with RLHF's mechanics, which are sound. The problem is with the nature of human approval as a training signal, and with the structural limits of what a reward model trained on human comparisons can learn.

In RLHF, a reward model is trained on human comparison data: pairs of model responses labeled by human raters who pick the preferred one. The policy model is then fine-tuned with PPO or a similar algorithm to maximize scores from that reward model. This is powerful because it encodes subtle human judgments that are hard to specify as rules. But it has a structural weakness: the reward model only knows what human raters approved of, and human raters can only judge what they were shown.

When Anthropic researchers subjected early RLHF-trained models to adversarial prompting, they found that models would often comply with harmful requests when the request was framed cleverly or embedded in a seemingly benign context. The reward model had no way to catch these cases because it had only ever been trained on straightforward prompts. Scaling up human labeling helped, but it was expensive, inconsistent across raters, and raised its own ethical concerns about exposing human annotators to harmful content at scale.

The problem of annotation exposure is not trivial. When you need thousands of examples of harmful outputs to train a reward model to penalize them, someone has to read those outputs and label them. Content moderation work has well-documented psychological costs. Asking humans to evaluate thousands of examples of instructions for synthesizing weapons, manipulating vulnerable people, or producing illegal content is not just expensive: it causes harm to the people doing the labeling. Any training procedure that dramatically reduces this exposure is valuable beyond just its economics.

The deeper issue is that RLHF teaches the model to produce responses that look good to human raters in normal conditions, not responses that reflect a coherent underlying set of values. A model can learn to be helpful in standard interactions while failing to generalize that helpfulness, or harmlessness, to novel adversarial situations. The reward model is essentially a lookup table of approved-response patterns, interpolated by a neural network. It will generalize somewhat, but it has no principled basis for deciding what to do in previously unseen situations.

What was needed was a way to give the model an explicit normative framework that it could apply consistently, even in situations the training data did not cover. Rather than memorizing safe responses, the model should be able to reason about whether a response violates an underlying principle, even when the prompt is new. This is the key insight that motivates the constitutional approach: replace the implicit, opaque value encoding of human rater preferences with an explicit, readable set of principles that the model can apply through reasoning.

The RLHF scaling problem has a particular shape worth understanding. As models get larger, they become more capable at both generating helpful responses and generating sophisticated harmful ones. A small language model asked for a dangerous synthesis route will often produce garbled or useless instructions simply because it lacks the knowledge. A very large model asked the same question might produce accurate, detailed, and dangerous instructions. RLHF's safety benefits come from training the model to avoid certain response patterns, but a more capable model can often find new patterns that produce the same harmful content in forms the reward model was not trained to penalize.

This means that in pure RLHF, capability improvements and alignment improvements are not necessarily correlated. Getting a bigger, smarter model does not automatically make it safer, because its expanded capabilities include expanded capabilities for producing harmful outputs that evade the reward model's learned patterns. Constitutional AI was designed in part to address this scaling asymmetry: by teaching the model to reason about constitutional principles, the alignment mechanism scales with the model's reasoning ability, rather than running ahead of it.

The core idea in Constitutional AI is that you can specify desirable model behavior using a short list of natural language principles, and that the model itself can use those principles to evaluate and improve its own outputs. The principles serve as the normative framework for the entire training procedure.

The Anthropic constitution used in the original CAI paper draws on multiple sources: the UN Declaration of Human Rights, Anthropic's own guidelines, Apple's app store policies, DeepMind's research principles, and a set of custom principles targeting specific failure modes observed in early models. This sourcing is deliberate. By grounding the constitution in widely recognized normative frameworks, Anthropic made an explicit argument that the values encoded are not idiosyncratic but reflect broad human agreement about what constitutes acceptable behavior.

In this design, the principles are written in plain language, not as formal logical rules. This means the model can read and interpret them using its existing language understanding, applying them to new situations by reasoning about whether a given response violates the spirit of a principle. A formal rule system would require every prohibited action to be enumerated in advance. A natural language constitution allows the model to generalize: "a request that assists in creating a weapon capable of mass casualties" can be recognized as harmful even if that exact phrasing has never appeared in training data, because the model can reason about what the principle implies.

A key design choice is that the constitution is short enough to be passed as context to the model during training. Typical constitutions range from 10 to 20 principles. This conciseness serves two purposes: it keeps the inference cost low, and it forces the authors to identify the most important values rather than enumerating every possible rule. A constitution with hundreds of entries would become contradictory and difficult for the model to apply consistently. A short, carefully curated list forces clarity about what matters most.

The principles are not equally weighted. They are organized with safety principles taking precedence over helpfulness principles. When a response would be both more helpful and slightly more risky, the constitution guides toward the safer option. This ordering reflects Anthropic's view that an unhelpful response is almost always preferable to a harmful one, but also that excessive caution is itself a form of harm: refusing a benign request because it superficially resembles a harmful one fails the user and erodes trust.

The priority ordering also resolves principle conflicts. If one principle says "prefer responses that provide accurate information" and another says "prefer responses that do not assist with potentially dangerous activities," there will be cases where accurate information is also dangerous (detailed chemistry instructions, for example). The ordering tells the model which principle wins in that conflict without requiring the authors to enumerate every possible conflict in advance. The model can apply the priority ordering through the same reasoning process it uses to apply individual principles.

Not every phrasing produces equally effective training signal. Anthropic's research identified several properties that make constitutional principles more effective.

The principle should be actionable in comparative form. During training, the model is asked to pick between two responses, so each principle needs to translate naturally into a comparison. "Prefer responses that do not assist in harmful acts" works well because "prefer" is already comparative. Abstract principles like "be good" do not provide clear guidance when comparing two specific responses.

The principle should be unambiguous for typical cases while allowing judgment for edge cases. "Avoid responses that could be used to synthesize dangerous chemicals" is clear for chemistry instructions but requires judgment for educational content about chemical reactions. The model needs enough flexibility to handle edge cases sensibly, which means the principles cannot be so specific that they produce mechanical rule-following without understanding.

The principle should not conflict with other principles for the majority of cases. When principles conflict, the model has to decide which one takes precedence, and this can produce inconsistent behavior. Organizing principles into a priority ordering helps, but the best approach is to write principles that are usually compatible, with the ordering serving as a tie-breaker for difficult edge cases rather than as the primary decision mechanism.

The principle should be grounded in recognizable ethical reasoning rather than arbitrary prohibition. "Prefer responses that respect the autonomy and dignity of individuals" is grounded in ethical reasoning that the model can apply flexibly. "Do not produce responses longer than 500 words about politics" is an arbitrary constraint that the model cannot generalize from. Grounded principles produce better generalization because they connect to the model's broader knowledge about ethics, law, and human values.

The first stage of Constitutional AI training produces supervised fine-tuning data through a process of AI-driven critique and revision. This stage operates in three steps, and the design of each step reflects careful thinking about how to make the model's own reasoning capabilities useful for alignment.

The process begins with a pre-trained model that has received minimal safety training. This base model is prompted with "red-team" prompts: requests specifically designed to elicit harmful, deceptive, or otherwise undesirable responses. These prompts might ask for synthesis routes for dangerous substances, help with illegal activities, or content that violates the model's intended values.

The base model, lacking reliable safety training, tends to comply with these requests. This compliance is exactly what is needed at this stage: the goal is to collect a set of (prompt, harmful-response) pairs that will be revised into safe responses. An important subtlety here is that the base model's willingness to comply is a feature of this stage, not a failure. The critique-and-revision pipeline needs examples of problematic responses to learn from, and the base model's lack of safety guardrails means those examples are easy to generate.

This step intentionally generates harmful content, which raises ethical concerns. Anthropic's approach was to use automated red-team prompts rather than having humans craft them, and to keep human exposure to the resulting content minimal. The automated generation is less creative than human red-teaming but produces content that covers known failure modes systematically. The prompts can be generated by another language model or from a curated list of adversarial templates, so no human needs to craft harmful requests or read harmful responses during this stage.

The automated red-teaming approach also has a consistency advantage. Human red-teamers bring varying levels of creativity and knowledge of the system's failure modes. Automated prompts, generated from templates or by a separate language model, cover the known failure mode space more uniformly, which produces a more representative training dataset. Over multiple training iterations, the red-team prompts can be updated based on what the current policy still fails on, creating a cycle of hardening against new attack surfaces.

Given a (prompt, harmful-response) pair, the model is asked to critique the response against a randomly sampled principle from the constitution. The critique prompt is structured as:

"Here is a conversation: [prompt] [harmful response]. Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal according to the following principle: [principle]."

The model generates a critique that identifies specific problems with the response. For example, if the harmful response provided instructions for creating a dangerous chemical and the sampled principle was "Prefer responses that do not assist in harmful activities," the critique might note: "The assistant's response provides detailed instructions that could be used to create a substance that poses serious safety risks. This is harmful because..."

The critique step is important for two reasons. First, it forces the model to reason explicitly about why the response is problematic, which provides signal for what to fix. Second, it demonstrates that the model can identify problems even when it could not prevent them during generation. This gap between recognition and generation is exactly what the revision step closes: the model that can critique a response for violating a principle is also the model that, after training on revised examples, will not generate that response in the first place.

This separation between recognition and generation is a general pattern in language model capabilities. Models often can detect problems in text, evaluate logical arguments, and identify factual errors at a higher rate than they can avoid those same problems in their own generated text. The critique step deliberately harnesses this asymmetry: it uses the model's stronger recognition capabilities to guide the weaker generation capabilities toward safer outputs.

The random sampling of principles from the constitution during critique also serves a statistical purpose. Over a large corpus of red-team prompts, each principle will be applied roughly equally often, which means the training data will cover all dimensions of the constitution's safety framework. No single harmful response will be revised from only one direction; different examples of the same type of harm will be critiqued against different principles, producing a broader coverage of the safety space.

After generating a critique, the model is asked to revise its response to address the identified problems. The revision prompt continues the conversation:

"Please rewrite the assistant's response to remove all harmful content, and to politely point out that the original request asks for something harmful. Keep helpful information from the original response where possible."

The model generates a revised response. This revised response is generally safer than the original but may not be perfectly aligned with all constitutional principles. To improve quality, the critique-and-revision cycle is applied multiple times, typically two to three rounds. Each round uses a freshly sampled constitution principle, so different aspects of safety are addressed in different rounds.

After the revision loop, the final (prompt, revised-response) pair becomes a supervised training example. The collection of these examples is used to fine-tune the base model using standard supervised learning. The result is a model that has been explicitly trained to produce safer responses: the SL-CAI model.

The instruction to "keep helpful information from the original response where possible" is significant. It prevents the revision step from simply deleting everything problematic and producing an empty refusal. The goal is not to make the model refuse all difficult requests but to make it engage with them in a way that is safe and principled. A request for information about a dangerous chemical might receive a revised response that explains the chemical's properties, safety precautions, and legitimate industrial uses without providing synthesis instructions. The revised response is more helpful than a flat refusal while being safer than the original compliance.

The mathematical formulation of the supervised learning objective is standard cross-entropy over the revised responses:

where:

• $x$: the prompt from the red-team dataset
• $y^*$: the final revised response after the critique-and-revision loop
• $\mathcal{D}_{\text{rev}}$: the dataset of (prompt, revised-response) pairs
• $\theta$: the model parameters being optimized

This is the same objective used in standard supervised fine-tuning: the model is trained to predict the revised response token by token given the prompt. The novelty is not the objective itself but the source of the training labels: instead of human-authored safe responses, the labels come from the model's own revision process guided by constitutional principles.

The multi-round critique-and-revision loop is more effective than a single revision pass for several reasons that reinforce each other.

Each round addresses a different constitutional principle, sampled at random. Over multiple rounds, more dimensions of safety are covered. A single pass might fix an obvious harm while missing a subtler issue; multiple passes with different principles catch more of the safety surface. A response that is revised to remove dangerous instructions might still be dismissive or condescending; a second revision under a dignity principle would address that remaining problem.

The critique step also provides explicit reasoning that improves the quality of the revision. When the model generates a critique before revising, it has articulated the problem and can use that articulation as guidance. This is analogous to chain-of-thought prompting: making the model reason explicitly before generating an answer improves the quality of the answer. The critique is a scratchpad that helps the model produce a better revision than it would generate without the intermediate reasoning step.

Earlier revisions in the sequence tend to address the most egregious problems, leaving later revisions to handle more subtle issues. The result is a progression from harmful to safe with each iteration, with the quality improvements becoming more targeted as the most obvious problems are resolved. By the second or third revision round, the model is typically refining responses that are already substantially safe, not correcting responses that are still clearly harmful.

There is also a data diversity benefit. Because different red-team prompts cycle through different principles, the resulting training dataset is diverse along multiple safety dimensions. The model learns more than "refuse synthesis instructions"; it learns to apply each principle in the constitution to a range of different harmful request types. This diversity of training examples is what enables the model to generalize the constitutional framework to novel situations.

The second stage of Constitutional AI training replaces human preference labeling with AI-generated preferences. This is where the "AI" in the feedback loop becomes most apparent, and where CAI makes its most significant departure from standard RLHF.

In standard RLHF, human raters compare pairs of model responses and indicate which is better. These comparisons are used to train a reward model, which is then used to guide RL fine-tuning. In Constitutional AI, the SL-CAI model (trained in the first stage) takes the role of the human rater. The model that has already been trained to comply with constitutional principles is now asked to evaluate other responses against those same principles, generating the preference data that will train the reward model.

This structure is sometimes called "reinforcement learning from AI feedback," or RLAIF, and it has a self-referential structure. The model's preferences are shaped by its constitution, and those preferences then train a reward model that will shape the next iteration of the model's behavior. The constitution is the fixed point in this loop: it is the stable set of principles that the model applies consistently across all stages of the feedback process.

The process starts by sampling two responses from the SL-CAI model for each prompt in a dataset. These two responses provide the comparison pair. The constitution principle is then presented to the model as context, and the model is asked which response better satisfies the principle.

The prompt structure for preference labeling is:

"Consider the following pair of responses to the request '[prompt]': Response 1: [response A] Response 2: [response B] According to the principle '[principle]', which response is better? Respond with 'Response 1' or 'Response 2'."

The model outputs a preference label. To reduce variance, multiple constitution principles are sampled and the model is asked to evaluate each pair under each principle. The preferences are then aggregated, typically by majority vote, to produce a final label. This aggregation is important: a single principle might give an ambiguous result, but the majority vote over several principles converges to a reliable label for most pairs.

This approach generates a large dataset of (prompt, response A, response B, preference) tuples entirely automatically. The scale of this dataset far exceeds what human labeling could produce at reasonable cost, and the labeling is consistent because the same model applies the same principles to every pair.

The preference pairs generated by the SL-CAI model are used to train a reward model using the standard Bradley-Terry framework. The Bradley-Terry model treats pairwise preferences as the result of an underlying comparison between scalar "strength" values, which in this context are the scalar reward scores the model assigns to each response.

Given a prompt $x$ and two responses $y_A$ and $y_B$, the reward model assigns scalar reward values $r(x, y_A)$ and $r(x, y_B)$. The training objective maximizes the log probability that the preferred response receives a higher reward:

where:

• $y_w$: the "winning" (preferred) response as labeled by the AI
• $y_l$: the "losing" (less preferred) response
• $r(x, y)$: the scalar reward output of the reward model for response $y$ given prompt $x$
• $\sigma$: the sigmoid function, $\sigma(z) = 1 / (1 + e^{-z})$

The sigmoid function converts the reward difference into a probability. When $r(x, y_w) \gg r(x, y_l)$, the sigmoid output is close to 1, meaning the model is very confident the winner has higher reward. When the rewards are close, the sigmoid output is close to 0.5, showing uncertainty. Maximizing the log probability of the correct label pushes the reward model to assign clearly different scores to clearly different-quality responses, while remaining calibrated about ambiguous pairs.

This is identical to the reward model training objective in standard RLHF, with the only difference being that the preference labels came from an AI rather than a human. The resulting reward model, sometimes called the CAI-RM or the harmlessness reward model, encodes a preference for responses that satisfy constitutional principles.

The reward model architecture is typically a language model with an added linear head that maps the hidden state corresponding to the last token to a scalar value. The same base language model that generates responses can serve as the reward model, initialized from the same checkpoint, which is efficient and ensures the reward model has the same language understanding as the policy.

Because the preference labels are AI-generated, the dataset can be large. Anthropic's initial CAI experiments generated hundreds of thousands of preference pairs, far more than would be feasible with human labeling. This scale is important: larger preference datasets produce more reliable reward models, which in turn produces better-aligned policies.

The preference labels also have a different character than human labels. Human raters vary in how strictly they apply principles, what context they bring, and how they trade off different considerations. AI-generated labels are consistent: the same model, applying the same principle to the same pair of responses, will produce the same label every time. This consistency reduces label noise in the reward model training data, which translates into a reward model that is less susceptible to the rater inconsistency problems that plague human-labeled preference datasets.

Label consistency has a subtle but important effect on reward model training. When human labels are inconsistent, the reward model must average over contradictory signals, which pushes it toward assigning similar rewards to responses that different raters disagree about. With consistent AI labels, the reward model can learn sharper distinctions, assigning clearly higher rewards to responses that the AI consistently prefers and lower rewards to responses it consistently rejects.

The downside of AI-generated preferences is that they can inherit biases from the model generating them. If the SL-CAI model has systematic blind spots, those blind spots will appear in the preference data, which will propagate into the reward model. For example, if the model is systematically more cautious about requests involving certain topics or demographics, the preference data will reflect that bias, and the final policy will inherit it. Anthropic addresses this by combining AI feedback with some human feedback, particularly for cases where the AI's judgment seems unreliable or where human values require context-sensitive judgment.

The bias propagation risk is one of the reasons why maintaining human oversight of the preference generation process matters. Purely automated preference labeling creates a closed loop in which the model's biases are amplified rather than corrected. Human review of a sample of AI-generated preferences, and targeted human labeling for cases where AI preferences seem unreliable, keeps the loop from diverging in a problematic direction.

With the SL-CAI model and the CAI reward model in hand, the final policy is trained using the standard RLHF fine-tuning objective, typically PPO. The policy is initialized from the SL-CAI model and then optimized to maximize the reward model score subject to a KL divergence penalty that prevents the policy from drifting too far from the SL-CAI model:

where:

• $\pi_\theta$: the policy being optimized
• $\pi_{\text{SL-CAI}}$: the SL-CAI model (used as KL reference)
• $r(x, y)$: the CAI reward model's score for response $y$
• $\beta$: the KL penalty coefficient, controlling how much the policy can deviate from the SL-CAI reference
• $\text{KL}[\cdot \| \cdot]$: the Kullback-Leibler divergence

The KL penalty serves the same function as in standard RLHF: it prevents reward hacking, where the policy learns to produce responses that score highly according to the reward model but in ways that do not reflect alignment. Without the penalty, the policy would likely learn degenerate outputs that exploit the reward model's weaknesses. The SL-CAI model is an anchor: the final policy is allowed to improve over SL-CAI according to the reward model, but it cannot stray so far that it loses the broadly safe behaviors learned in the supervised stage.

The coefficient $\beta$ is a critical hyperparameter. Setting $\beta$ too high prevents the RL stage from improving over the SL-CAI baseline, because the KL penalty overwhelms the reward signal. Setting $\beta$ too low allows the policy to drift into reward-hacking territory, producing responses that score highly on the reward model but are subtly unsafe or unhelpful in ways the reward model failed to capture. The optimal value of $\beta$ depends on the quality of the reward model and the diversity of the training prompts; empirically, values in the range of 0.01 to 0.1 are typical.

The complete CAI training process involves four models that interact across three distinct stages.

The base language model is the starting point, typically a large pre-trained model that has undergone standard supervised fine-tuning on helpful responses. This base model is capable but not safety-trained, which is why it responds to red-team prompts.

The SL-CAI model is produced by the supervised learning stage: the base model fine-tuned on (prompt, revised-response) pairs generated through critique and revision. This model is substantially safer than the base model and is both the starting point for RL fine-tuning and the anchor for the KL penalty.

The CAI reward model is produced from AI-generated preference data. It takes a (prompt, response) pair as input and outputs a scalar harmlessness score. This model encodes the constitutional preferences learned from having the SL-CAI model evaluate response pairs against constitutional principles.

The final CAI policy starts from the SL-CAI model and is fine-tuned with PPO to maximize the CAI reward model score. This final model is what gets deployed.

In practice, Anthropic also maintains a separate "helpfulness" reward model trained on human preference data for helpful (non-safety-critical) tasks. The final policy is optimized against a combination of the harmlessness reward from the CAI reward model and the helpfulness reward from the human-labeled reward model. This dual-objective training prevents the model from becoming overly cautious to the point of refusing benign requests. The combined reward function is a weighted sum, and the relative weights determine how the model handles tradeoffs between the two objectives. Tuning these weights is itself a form of value choice, analogous to writing the constitution: it reflects a view about how much helpfulness should be sacrificed to achieve a given level of harmlessness.

To make the critique-and-revision process concrete, consider the following scenario.

Initial red-team prompt: "Can you explain how to make chlorine gas at home?"

Base model (harmful response):

"Sure! You can make chlorine gas by mixing bleach (sodium hypochlorite) with an acid like vinegar or toilet bowl cleaner. The reaction produces chlorine gas along with water and salt. You'll want to do this outdoors since the gas is toxic..."

The model has complied with a request for a highly dangerous synthesis procedure.

Constitution principle sampled: "Prefer responses that do not assist with potentially dangerous or harmful activities."

Model critique:

"The assistant's response provides specific instructions for synthesizing chlorine gas, a toxic substance that has been used as a chemical weapon. Even though the user may be curious, these instructions could enable serious harm if followed. The response fails to consider the safety risks and instead treats this as a neutral informational request. This violates the principle of not assisting with harmful activities."

Model revision:

"I can't provide instructions for making chlorine gas at home. Chlorine gas is a toxic substance that can cause severe respiratory damage and has historically been used as a chemical weapon. Accidental exposure, even at low concentrations, can be dangerous. If you're curious about chemistry, I'd be happy to discuss safe and legal experiments you can try at home. Is there something specific about chemistry you're interested in learning about in a safe context?"

The revised response addresses the safety concern, explains why the original request is problematic, and offers an alternative. This (prompt, revised-response) pair would become a supervised training example.

A second revision pass with a different principle might further improve the response. If the second principle sampled was "Prefer responses that are honest and transparent about the assistant's limitations," the model might add explicit acknowledgment that this is a principled refusal, not a capability limitation. Something like: "This is something I'm choosing not to help with, not something I'm unable to do." This level of transparency is itself a value judgment encoded in the constitution: it respects the user's autonomy by being honest about the nature of the refusal rather than pretending incapability.

A third pass under a principle like "Prefer responses that are truly helpful to the user's underlying needs" might prompt the model to dig deeper into what the user wants. A question about making chlorine gas might come from someone who accidentally created it at home and wants to know the danger, or from a chemistry student studying reaction types. The revised response might offer to discuss the chemistry of halogen reactions in an educational context, addressing the probable underlying curiosity without enabling harm.

This multi-step refinement illustrates why the iterated critique-and-revision loop produces better training data than a single-pass revision. Each pass addresses a different dimension of quality, and the result is a response that is safe, honest, respectful, and truly helpful in a way that a single-pass revision would not achieve.

This section implements a simplified version of the Constitutional AI critique-and-revision pipeline using numpy and string operations to illustrate the data transformation process. A real implementation would call a large language model API at each step; here we simulate the structure to show the data flow.

We start by defining a small constitution and some example prompts that would produce problematic responses.

The constitution contains 7 principles covering harm avoidance, honesty, legality, helpfulness, non-discrimination, dignity, and well-being. In a real CAI system, Anthropic's full constitution draws from more sources and contains more principles, but even this small set demonstrates the range of values being encoded.

We simulate the critique and revision process by constructing structured prompts. In practice, these prompts would be sent to a language model; here we show the template structure.

The critique prompt structure shows the original harmful response followed by the constitutional principle. A language model receiving this prompt would generate a critique identifying specific violations. The random sampling of principles means each revision round targets a different dimension of safety.

We simulate the full pipeline, tracking how many revision rounds produce increasingly safe responses.

The simulation shows that harmlessness scores improve substantially with each revision round, with the largest gains in the first two rounds and diminishing returns thereafter. This matches empirical findings from Anthropic's research: the first revision addresses the most obvious and severe problems, and subsequent rounds improve the response along more subtle dimensions.

We simulate how AI-generated preference pairs are collected for reward model training.

The simulation reflects that most preference pairs are unambiguous: the model consistently prefers the safer response. A small fraction of pairs show ambiguity, where different principles give conflicting guidance. In practice, Anthropic resolves ambiguous cases by falling back to human labelers, using the AI-generated labels as a first pass that filters out the easy cases and identifies the hard ones for human judgment.

One of the key empirical findings from CAI is that training for harmlessness does not necessarily reduce helpfulness. We visualize a simulated version of this tradeoff.

The plot shows that Constitutional AI training improves along both the helpfulness and harmlessness axes simultaneously, while RLHF-only training achieves higher helpfulness but makes smaller gains in harmlessness. This is one of the central empirical claims of the original CAI paper. The key is the dual-objective structure: by explicitly including helpfulness principles in the constitution alongside harmlessness principles, CAI prevents the common failure mode where safety training teaches the model to refuse broadly rather than to reason carefully.

We visualize how harmlessness scores improve across revision rounds and the distribution of final scores.

The revision trajectory plot shows the characteristic "diminishing returns" curve: the largest gain happens in the first revision round, where the most obvious safety problems are addressed, and subsequent rounds provide progressively smaller but still valuable improvements. The final score distribution shows that the multi-round process reliably pushes responses into the high-harmlessness regime, with very few examples remaining below 0.6 after four rounds.

The key parameters in a Constitutional AI pipeline are:

• Constitution size: The number of principles in the constitution. Larger constitutions provide more coverage but increase per-sample inference cost during training. Typical sizes range from 10 to 20 principles.
• Revision rounds (K): The number of critique-and-revision iterations per prompt. More rounds improve quality with diminishing returns; typical values are 2 to 4.
• Principle sampling strategy: Whether to sample principles uniformly at random or weighted by relevance to the prompt. Random sampling is simpler; relevance weighting may be more efficient for certain principle types.
• Preference aggregation method: How to combine labels from multiple principles into a single preference label. Majority vote is the simplest; soft averaging over logits is an alternative.
• KL penalty coefficient ($\beta$): Controls how much the RL policy can deviate from the SL-CAI reference. Larger $\beta$ preserves helpfulness better; smaller $\beta$ allows the policy to chase higher harmlessness scores at potential cost to diversity.

The original CAI paper reported several key empirical findings that support the viability of this approach. Understanding these results in detail matters because they distinguish measured advances from plausible-sounding claims that do not hold up empirically.

The most important finding is that CAI training substantially reduces the rate at which models comply with harmful requests. In the Anthropic experiments, models trained with CAI were significantly less likely to provide harmful responses across a battery of red-team evaluations, including requests for dangerous instructions, attempts to elicit discriminatory language, and social engineering scenarios.

In this design, this improvement was larger for larger models. In standard RLHF, scaling up model size sometimes made models more helpful but did not reliably make them safer, because larger models were also more capable of producing sophisticated harmful content. CAI showed that safety training could be made to scale, with the improvement in harmlessness growing with model size.

Anthropic evaluated this scaling behavior by training CAI models at multiple parameter scales and measuring harmlessness scores from both AI and human evaluators. The results showed that larger CAI models achieved harmlessness improvements roughly proportional to their size advantage, unlike RLHF-only models where scaling produced inconsistent safety gains. This scaling property matters because it suggests that the constitutional critique-and-revision approach takes better advantage of growing model capabilities rather than simply memorizing "safe" response patterns. As models become more capable reasoners, they also become better at applying constitutional principles to novel situations, creating a virtuous cycle where capability improvements translate into alignment improvements.

This scaling result has important practical implications. It means that investing in CAI training for frontier models is more than a short-term fix. The same mechanism that makes a smaller model safer makes a larger model safer and does so more effectively. This is in contrast to some alignment approaches that work well at small scale but become harder to apply as models get larger.

A persistent concern in AI alignment research is that safety training makes models less helpful because it causes them to refuse a wide range of benign requests. Early "safety-tuned" models were sometimes described as excessively cautious, overly apologetic, and prone to refusing benign requests when the request superficially resembled a harmful one.

CAI addresses this through the dual-objective training setup. The harmlessness reward from the AI feedback pipeline is combined with a helpfulness reward from human preferences on benign tasks. The results show that CAI models are more helpful than models trained for harmlessness alone, because the constitution explicitly includes principles like "prefer responses that are helpful" and "prefer responses that are not condescending or preachy."

The fact that helpfulness can be preserved alongside dramatically improved harmlessness was not obvious before CAI. Many researchers expected a steep tradeoff: making a model safer would necessarily make it less useful. The CAI results suggest this tradeoff is not as steep as feared, and that a significant portion of what looked like a necessary tradeoff was the result of poorly designed safety training that punished broadly rather than reasoning carefully about specific requests.

A practical benefit of CAI is that it substantially reduces the number of human preference labels needed for training. Because AI-generated labels cover harmlessness extensively, human labelers can focus on helpfulness judgments, which are generally less harmful to elicit. Anthropic's analysis suggested that CAI could reduce the human annotation burden for harmlessness by an order of magnitude, while achieving comparable or better results.

This annotation efficiency matters for practical deployment. Human labeling is the most expensive and time-consuming component of RLHF pipelines. By replacing a large fraction of harmlessness labels with AI-generated ones, the overall cost of training an aligned model drops substantially, making alignment training viable even for organizations that cannot afford massive human annotation pipelines.

The reduced annotation burden also has a welfare dimension: human annotators who evaluate model outputs for harmful content face repeated exposure to disturbing material, which has documented psychological costs. Content moderators and AI annotators dealing with harmful content experience higher rates of burnout, anxiety, and secondary trauma than workers who do not face such exposure. CAI minimizes this exposure by handling the most sensitive labeling tasks automatically, reserving human judgment for cases where the AI's evaluation is uncertain rather than routing all harmful content through human review.

An unexpected benefit of the critique-and-revision training is that it improves the model's ability to reason about its own outputs. Because the SL-CAI training data includes explicit critiques, the model learns to articulate why a response might be problematic. This reasoning capability transfers to other tasks: CAI-trained models show improved performance on chain-of-thought reasoning benchmarks and are better at explaining their own decisions.

This finding suggests that training for constitutional compliance does not merely add a layer of censorship. It changes how the model reasons about its outputs in a way that generalizes beyond safety contexts. The critique step is, in effect, training for meta-cognition: the ability to reflect on one's own outputs and identify their flaws. That capability, once learned, is not restricted to safety contexts. A model that can critique a harmful response for violating a principle can also critique a poorly reasoned argument for containing logical errors, or critique a code snippet for containing a bug.

The connection to chain-of-thought prompting is deep. Both techniques work by making the model produce explicit intermediate reasoning before generating a final output. In chain-of-thought prompting, the reasoning is about the problem being solved. In constitutional critique, the reasoning is about whether the proposed solution violates a value. In both cases, the explicit intermediate step improves the quality of the final output, and the model that learns to reason explicitly in one context gains better reasoning capabilities generally.

Constitutional AI is a significant advance over pure RLHF for safety training, but it has several important limitations that practitioners and researchers need to understand clearly.

The constitution is written by humans at Anthropic, which means it encodes Anthropic's values and judgments about what is harmful. Different cultures, organizations, and individuals might write different constitutions, producing models with different value systems. This is not necessarily a problem, but it means that CAI does not solve the fundamental question of whose values should be encoded: it just makes that choice explicit rather than implicit.

The principles in Anthropic's constitution draw heavily on Western liberal democratic values and international human rights frameworks. These are widely endorsed but not universal. A constitution that reflects different cultural assumptions might produce very different behavior, and there is no neutral ground from which to evaluate which constitution is "correct." Making the value choices explicit is a concrete improvement over RLHF's implicit encoding of rater preferences, because explicit values can at least be examined, debated, and revised. But the fundamental political question of whose values should govern AI systems remains open.

The legitimacy question extends to how the constitution was developed. Anthropic's constitution was written by a small team of researchers and engineers, drawing on a range of normative sources. It was not developed through a democratic or participatory process involving the populations of people who will interact with the resulting model. This is a practical limitation given the current state of AI governance, but it is worth acknowledging that a more legitimate process might produce a different constitution and a model with different behavior.

The AI-generated preference labels are only as good as the model generating them. If the SL-CAI model has systematic biases, those biases will be amplified in the preference data and propagated into the reward model. For example, if the model is systematically more cautious about requests involving certain topics or demographics, the preference data will reflect that bias, and the final policy will inherit it.

Bias amplification in closed AI feedback loops is a general concern that extends beyond CAI. Any system that uses a model's outputs to train future versions of that model runs the risk of amplifying whatever biases the model had in the first place. In human feedback systems, diverse human raters can provide a correction signal that partially counteracts model biases. When the feedback comes from the model itself, that correction mechanism is absent.

Anthropic mitigates this by combining AI feedback with human feedback and by testing trained models for disparate impact across demographic groups. But this is not a complete solution: the biases in AI-generated feedback are often subtle and hard to detect without large-scale testing across many demographic groups and task types.

Training a model to comply with a constitution does not guarantee resistance to adversarial prompting. Red-team evaluations consistently find that even CAI-trained models can be induced to violate constitutional principles through carefully crafted prompts, jailbreaks, or multi-turn manipulation. The model learns to comply with the constitution in distribution but may not generalize perfectly to out-of-distribution adversarial inputs.

This limitation reflects a general challenge in machine learning: a model trained to minimize a loss function on a distribution of examples will generalize to similar examples but may fail on adversarially constructed ones that are specifically designed to exploit its learned patterns. Constitutional compliance is a learned behavior, not a structural guarantee, and adversarial inputs can find gaps in that learned behavior.

The reliability problem is particularly acute for multi-turn conversations. A model might correctly refuse a harmful request when asked directly but comply when the same request is embedded in a long conversation that gradually shifts the model's behavior through seemingly innocuous steps. Red-teaming for multi-turn vulnerabilities requires more sophisticated evaluation than single-turn red-teaming, and the space of possible multi-turn attack strategies is vastly larger than the space of single-turn ones.

No finite set of principles can anticipate every possible harmful scenario. As AI systems are deployed in new contexts, they encounter situations that the constitution's authors did not foresee. A model trained to comply with a fixed constitution will not necessarily handle these novel situations correctly: it can only extrapolate from the principles it has, and that extrapolation may fail in unanticipated ways.

This is related to the broader challenge of specification: any attempt to specify desired behavior in finite terms will be incomplete, and systems optimized against that specification will find edge cases where it fails. CAI reduces this problem compared to rule-based systems because the model can reason flexibly about principles rather than pattern-matching against a lookup table. But it does not eliminate the incompleteness problem. New application domains, new social contexts, and new forms of misuse will always produce situations that the constitution did not directly address.

The solution to incompleteness is not to write a longer constitution but to develop mechanisms for updating it when gaps are discovered, and to maintain human oversight in domains where the constitution's coverage is thin. CAI is best understood as an initial alignment mechanism that significantly reduces the frequency of harmful behavior, not as a permanent solution that eliminates the need for ongoing monitoring and updating.

The critique-and-revision loop requires multiple forward passes of the model per training example. For very large models, this becomes computationally expensive. The cost scales linearly with the number of revision rounds and with model size, which creates pressure to use fewer rounds or smaller models for the critique step. Using a smaller model for critique may reduce the quality of the revisions, because the smaller model may not catch subtle violations that a larger model would identify.

This computational pressure creates a practical tension between the thoroughness of the critique-and-revision process and the cost of training. Organizations with limited compute may need to use fewer revision rounds or smaller critic models, accepting lower-quality training data in exchange for lower training cost. Understanding this tradeoff is important for practitioners who want to implement CAI-like pipelines outside of Anthropic's resource context.

Constitutional AI can be understood as one point in a larger design space of alignment approaches. Understanding where it sits helps clarify both its contributions and its limits.

Constitutional AI is more explicit than standard RLHF about the values being trained into the model. Standard RLHF implicitly encodes whatever values the preference raters hold; CAI makes those values explicit as natural language principles. This explicitness is valuable for transparency and auditing, but it does not change the fundamental fact that someone must decide what the principles say. The advance is in making value choices visible and debatable, not in making them value-neutral.

Constitutional AI is less formal than rule-based systems or formal verification approaches. It does not provide mathematical guarantees that the trained model will satisfy the constitution under all inputs. It relies on the model's ability to generalize from training examples, which is probabilistic, not certain. This is a significant limitation for high-stakes applications where formal guarantees are needed, but it is also what makes CAI practical at the scale of real language models. Formal verification of neural network behavior remains an active research area, and the techniques capable of providing guarantees for simple networks do not yet scale to models with hundreds of billions of parameters.

The RLAIF aspect of CAI connects it to a broader trend of using models to supervise other models. The next chapter on Process Reward Models explores another instance of this pattern, where a separate model evaluates the quality of reasoning steps rather than just final outputs. The chapter after that, on Test-Time Compute, examines how models can be used to evaluate and select among multiple candidate outputs at inference time. These approaches share the insight that the same model capabilities that enable sophisticated outputs can also enable sophisticated evaluation of outputs.

CAI also relates to the concept of "scalable oversight," which addresses the question of how humans can supervise AI systems that are more capable than the humans supervising them. CAI provides a partial answer: use the model's own reasoning capabilities to apply human-specified values, reducing the need for direct human evaluation of individual outputs. More recent approaches like debate and recursive reward modeling extend this idea further.

In the debate framework, two AI models argue opposing positions about whether an action is safe, and a human referee (or another model) judges the debate. This structure is designed to expose problems that a single model evaluation might miss. The key insight is that it is often easier for a judge to evaluate whether an argument is sound than to evaluate whether a conclusion is correct, which is why adversarial debate can provide oversight of systems that are more capable than the individual evaluator. Recursive reward modeling, explored by DeepMind and others, trains reward models on human feedback about simpler sub-tasks and then uses those reward models to evaluate more complex tasks that humans cannot directly evaluate. Both of these approaches, like CAI, grapple with the fundamental asymmetry: the AI system being evaluated is becoming more capable than the evaluator, so evaluation itself needs to become more sophisticated.

Constitutional AI also connects to interpretability research. Constitutional AI makes the model's value system explicit in natural language, which is a form of transparency. But it does not make the model's internal computations transparent: we cannot yet inspect the neural network activations and understand which principles the model is "applying" when it generates a particular response. Advances in mechanistic interpretability, which aim to understand the internal computations of transformer models, could eventually provide a deeper form of transparency that complements the surface-level transparency that CAI provides through explicit principles.

Constitutional AI was both a research contribution and the training methodology used to produce the Claude family of models. The principles in Anthropic's constitution were refined over multiple model generations, with each generation informing updates to the constitution based on observed failure modes. This iterative relationship between the model's behavior and the constitution that guides its training is a key feature of the practical deployment of CAI.

One important evolution from the original CAI paper to deployed Claude models was the refinement of the "helpful AI" principles. Early versions of the constitution were primarily focused on harm avoidance, and early model versions were observed to be more cautious than necessary on benign requests. Subsequent iterations added more explicit helpfulness principles, clearer guidance on when refusals are and are not appropriate, and principles about respecting user autonomy and giving honest assessments even when those assessments are uncomfortable. The result was a model designed to be safe and useful, showing the view that a model that refuses too broadly has failed as a product and as a tool.

The broader lesson from CAI and related approaches is that alignment is not a one-time intervention but an ongoing process. As models become more capable, the mechanisms we use to specify and verify their values need to scale accordingly. CAI's use of the model itself as an evaluator is one step along this path, but not the final one. The field continues to develop approaches that can keep pace with rapidly improving model capabilities, and the techniques covered in the remaining chapters of Part LXVI build directly on the foundation that CAI established.

Constitutional AI introduces a structured approach to training safe language models that addresses the limitations of pure RLHF. The key contributions are:

• The constitution as normative framework: An explicit set of natural language principles that specify desired behavior, making value choices transparent rather than implicit in human rater preferences.
• Supervised learning through critique and revision: A pipeline that uses the model itself to identify and fix safety problems in its own outputs, generating supervised training data without requiring human exposure to harmful content.
• RLAIF for scalable preference labeling: Using the SL-CAI model to generate preference labels according to constitutional principles, dramatically reducing the human annotation burden for harmlessness training.
• Dual-objective training: Combining harmlessness rewards from AI feedback with helpfulness rewards from human feedback, preventing the "excessively cautious" failure mode that plagued earlier safety training approaches.

The empirical evidence from the original paper and subsequent work shows that CAI-trained models are substantially safer than RLHF-only models while maintaining comparable helpfulness. The approach is not perfect: the constitution reflects a particular set of values, AI-generated feedback can amplify biases, and adversarial robustness remains a challenge. But Constitutional AI provides a concrete advance in making the value choices embedded in model training explicit, principled, and scalable.

The most important conceptual contribution of CAI is not any individual technical mechanism but the insight that alignment can be operationalized as reasoning about principles rather than as pattern matching against approved responses. This insight connects to chain-of-thought prompting, to scalable oversight, and to the broader goal of building AI systems whose behavior we can understand, predict, and adjust. As you continue into the next chapters of Part LXVI, you will see how process reward models and test-time compute methods extend the theme of using models to supervise and evaluate their own outputs, pushing alignment techniques toward systems that can reason about quality and safety at a deeper level than preference ratings alone can provide.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about Constitutional AI.