Constitutional AI (2023) › Training Process

The Constitutional AI training methodology implemented this self-supervision approach through carefully designed procedures that systematically built the model's ability to apply constitutional principles. The training began with creating a constitution, a curated set of principles that would guide model behavior. These principles needed to be clear enough that a language model could understand and apply them, while broad enough to cover important alignment considerations.

The first phase of training, supervised learning with self-critique, transformed how models were fine-tuned. Instead of directly fine-tuning on human-labeled examples, the process worked as follows: given a prompt, the model would generate an initial response. Then, using a special self-critique prompt, the model would evaluate whether this response followed the constitutional principles. If the model identified problems, such as the response being unhelpful, potentially harmful, or dishonest, it would generate a revised response that addressed these issues. The revised responses, having been improved through principled self-evaluation, would then be used for supervised fine-tuning.

This self-critique procedure could be iterated multiple times. A model might critique its initial response, generate a revision, critique that revision, and generate a further improvement. The process continued until the model determined that its response adequately followed the constitutional principles, or until a predetermined number of iterations was reached. Each iteration of self-critique and revision built the model's understanding of how to apply principles in practice, creating a dataset of responses that had been systematically improved through principled reasoning.

The self-critique prompts were designed to make the evaluation process explicit. A typical prompt might structure the critique as: "Review the following response according to the constitutional principle: [principle text]. Identify any ways the response violates this principle, then rewrite the response to better follow the principle." This explicit framing encouraged the model to engage in principled reasoning rather than simply pattern-matching from training data. The model learned to follow rules, reason about ethical considerations, and apply principles to novel situations.

The second phase, RLAIF, extended the self-supervision to preference learning. In this phase, for each prompt, the model would generate multiple candidate responses. The model would then compare these responses and determine which better followed the constitutional principles. This created preference pairs similar to those used in RLHF, but generated by the model itself rather than human annotators. The model might evaluate responses on multiple dimensions: helpfulness, harmlessness, honesty, and adherence to specific constitutional principles.

These AI-generated preferences could be used to train a reward model, which learned to predict which responses the model would prefer according to constitutional principles. The reward model could then guide reinforcement learning fine-tuning. This provided feedback on whether responses aligned with the constitution. This created a form of reinforcement learning that didn't require human annotators, while still benefiting from the structured feedback that preference learning could provide. The reward signal came from the model's own evaluation of adherence to principles, creating a self-consistent training loop.

An important aspect of the training process was how constitutional principles were selected and refined. Anthropic drew principles from multiple sources: human rights documents, ethical frameworks, explicit statements about desired model behavior, and critiques of harmful responses. The constitution could evolve over time, with new principles added to address specific concerns or to incorporate feedback about model behavior. The process of defining principles remained a human responsibility, requiring careful consideration of values, trade-offs, and potential unintended consequences.

The training methodology demonstrated that models could learn to apply principles even when those principles were stated in natural language rather than formal rules. Language models' ability to understand and reason about textual principles made it possible to express alignment goals in relatively intuitive terms, though the principles still needed to be clear and specific enough for consistent application. This natural language framing made it easier for humans to understand and modify the alignment goals, compared to approaches that encoded preferences implicitly in reward models.