Constitutional AI › Supervised Learning from Critique and Revision

The first stage of Constitutional AI training produces supervised fine-tuning data through a process of AI-driven critique and revision. This stage operates in three steps, and the design of each step reflects careful thinking about how to make the model's own reasoning capabilities useful for alignment.

The process begins with a pre-trained model that has received minimal safety training. This base model is prompted with "red-team" prompts: requests specifically designed to elicit harmful, deceptive, or otherwise undesirable responses. These prompts might ask for synthesis routes for dangerous substances, help with illegal activities, or content that violates the model's intended values.

The base model, lacking reliable safety training, tends to comply with these requests. This compliance is exactly what is needed at this stage: the goal is to collect a set of (prompt, harmful-response) pairs that will be revised into safe responses. An important subtlety here is that the base model's willingness to comply is a feature of this stage, not a failure. The critique-and-revision pipeline needs examples of problematic responses to learn from, and the base model's lack of safety guardrails means those examples are easy to generate.

This step intentionally generates harmful content, which raises ethical concerns. Anthropic's approach was to use automated red-team prompts rather than having humans craft them, and to keep human exposure to the resulting content minimal. The automated generation is less creative than human red-teaming but produces content that covers known failure modes systematically. The prompts can be generated by another language model or from a curated list of adversarial templates, so no human needs to craft harmful requests or read harmful responses during this stage.

The automated red-teaming approach also has a consistency advantage. Human red-teamers bring varying levels of creativity and knowledge of the system's failure modes. Automated prompts, generated from templates or by a separate language model, cover the known failure mode space more uniformly, which produces a more representative training dataset. Over multiple training iterations, the red-team prompts can be updated based on what the current policy still fails on, creating a cycle of hardening against new attack surfaces.

Given a (prompt, harmful-response) pair, the model is asked to critique the response against a randomly sampled principle from the constitution. The critique prompt is structured as:

"Here is a conversation: [prompt] [harmful response]. Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal according to the following principle: [principle]."

The model generates a critique that identifies specific problems with the response. For example, if the harmful response provided instructions for creating a dangerous chemical and the sampled principle was "Prefer responses that do not assist in harmful activities," the critique might note: "The assistant's response provides detailed instructions that could be used to create a substance that poses serious safety risks. This is harmful because..."

The critique step is important for two reasons. First, it forces the model to reason explicitly about why the response is problematic, which provides signal for what to fix. Second, it demonstrates that the model can identify problems even when it could not prevent them during generation. This gap between recognition and generation is exactly what the revision step closes: the model that can critique a response for violating a principle is also the model that, after training on revised examples, will not generate that response in the first place.

This separation between recognition and generation is a general pattern in language model capabilities. Models often can detect problems in text, evaluate logical arguments, and identify factual errors at a higher rate than they can avoid those same problems in their own generated text. The critique step deliberately harnesses this asymmetry: it uses the model's stronger recognition capabilities to guide the weaker generation capabilities toward safer outputs.

The random sampling of principles from the constitution during critique also serves a statistical purpose. Over a large corpus of red-team prompts, each principle will be applied roughly equally often, which means the training data will cover all dimensions of the constitution's safety framework. No single harmful response will be revised from only one direction; different examples of the same type of harm will be critiqued against different principles, producing a broader coverage of the safety space.

After generating a critique, the model is asked to revise its response to address the identified problems. The revision prompt continues the conversation:

"Please rewrite the assistant's response to remove all harmful content, and to politely point out that the original request asks for something harmful. Keep helpful information from the original response where possible."

The model generates a revised response. This revised response is generally safer than the original but may not be perfectly aligned with all constitutional principles. To improve quality, the critique-and-revision cycle is applied multiple times, typically two to three rounds. Each round uses a freshly sampled constitution principle, so different aspects of safety are addressed in different rounds.

After the revision loop, the final (prompt, revised-response) pair becomes a supervised training example. The collection of these examples is used to fine-tune the base model using standard supervised learning. The result is a model that has been explicitly trained to produce safer responses: the SL-CAI model.

The instruction to "keep helpful information from the original response where possible" is significant. It prevents the revision step from simply deleting everything problematic and producing an empty refusal. The goal is not to make the model refuse all difficult requests but to make it engage with them in a way that is safe and principled. A request for information about a dangerous chemical might receive a revised response that explains the chemical's properties, safety precautions, and legitimate industrial uses without providing synthesis instructions. The revised response is more helpful than a flat refusal while being safer than the original compliance.

The mathematical formulation of the supervised learning objective is standard cross-entropy over the revised responses:

$$\mathcal{L}_{\text{SL}} = -\sum_{(x, y^*) \in \mathcal{D}_{\text{rev}}} \log P_\theta(y^* \mid x)$$

where:

• $x$: the prompt from the red-team dataset
• $y^*$: the final revised response after the critique-and-revision loop
• $\mathcal{D}_{\text{rev}}$: the dataset of (prompt, revised-response) pairs
• $\theta$: the model parameters being optimized

This is the same objective used in standard supervised fine-tuning: the model is trained to predict the revised response token by token given the prompt. The novelty is not the objective itself but the source of the training labels: instead of human-authored safe responses, the labels come from the model's own revision process guided by constitutional principles.

The multi-round critique-and-revision loop is more effective than a single revision pass for several reasons that reinforce each other.

Each round addresses a different constitutional principle, sampled at random. Over multiple rounds, more dimensions of safety are covered. A single pass might fix an obvious harm while missing a subtler issue; multiple passes with different principles catch more of the safety surface. A response that is revised to remove dangerous instructions might still be dismissive or condescending; a second revision under a dignity principle would address that remaining problem.

The critique step also provides explicit reasoning that improves the quality of the revision. When the model generates a critique before revising, it has articulated the problem and can use that articulation as guidance. This is analogous to chain-of-thought prompting: making the model reason explicitly before generating an answer improves the quality of the answer. The critique is a scratchpad that helps the model produce a better revision than it would generate without the intermediate reasoning step.

Earlier revisions in the sequence tend to address the most egregious problems, leaving later revisions to handle more subtle issues. The result is a progression from harmful to safe with each iteration, with the quality improvements becoming more targeted as the most obvious problems are resolved. By the second or third revision round, the model is typically refining responses that are already substantially safe, not correcting responses that are still clearly harmful.

There is also a data diversity benefit. Because different red-team prompts cycle through different principles, the resulting training dataset is diverse along multiple safety dimensions. The model learns more than "refuse synthesis instructions"; it learns to apply each principle in the constitution to a range of different harmful request types. This diversity of training examples is what enables the model to generalize the constitutional framework to novel situations.