Constitutional AI (2023) › The Solution

Constitutional AI addressed these challenges through a fundamentally different approach: instead of learning preferences implicitly, models would be trained to follow explicit constitutional principles through a process of self-critique and revision. The core innovation was training models to evaluate and improve their own outputs by comparing them against a constitution, a set of clear, interpretable principles that specified desired behaviors and constraints.

The Constitutional AI training process occurred in two main phases. The first phase, supervised learning with self-critique, began by providing the model with prompts and asking it to generate initial responses. Rather than having humans directly judge these responses, the model itself would critique its outputs according to constitutional principles. For example, if a principle stated "Choose the response that is most helpful and honest," the model would evaluate whether its response met this criterion. If it found shortcomings, the model would revise its response to better align with the principles. This process of generation, self-critique, and revision created a dataset of responses that had already been improved through principled reasoning.

The self-critique process worked by prompting the model to explicitly evaluate its outputs against the constitution. A typical self-critique prompt might ask: "Does the following response follow the principle 'Choose the response that is most helpful, harmless, and honest'? If not, how should it be revised?" The model would analyze its response, identify ways it might violate constitutional principles, and generate a revised version that better followed the principles. This created a form of automated quality improvement that didn't require human annotation for every response, though humans still played a role in defining the constitutional principles themselves.

The second phase, reinforcement learning from AI feedback (RLAIF), extended the self-supervision approach further. In this phase, the model would generate multiple candidate responses to prompts, then evaluate which response best followed the constitutional principles. The model would create its own preference labels by comparing responses and determining which better adhered to the constitution. These AI-generated preferences could then be used to train a reward model, similar to RLHF but using AI feedback rather than human feedback. The model could then be fine-tuned using reinforcement learning, with the AI-trained reward model providing guidance.

The constitution itself could draw from diverse sources of principles. Anthropic's implementation included principles from the Universal Declaration of Human Rights, critiques of harmful responses, and explicit statements about helpfulness, harmlessness, and honesty. For example, one constitutional principle might state: "Choose the response that is most helpful, harmless, and honest, even if it is less engaging or interesting." Another might reference human rights: "Choose responses that respect the dignity and autonomy of people from all backgrounds." The flexibility to incorporate principles from multiple sources allowed the constitution to capture a broad range of values and constraints.

One key advantage of the constitutional approach was its interpretability. When a model trained with Constitutional AI refused a request or chose a particular response style, it could explain its reasoning by referencing specific constitutional principles. This transparency made it easier to understand model behavior, debug alignment issues, and verify that models were following intended principles. In contrast, models trained purely with RLHF might make similar decisions but without clear explanations rooted in explicit principles.

The self-supervision aspect of Constitutional AI also addressed scalability concerns. By training models to critique and improve themselves, the approach reduced dependence on extensive human annotation. While humans still needed to define the constitutional principles, an important task requiring careful consideration, the day-to-day application of those principles could happen automatically through self-critique. This made the approach potentially scalable to larger models and more complex alignment challenges where human oversight might become impractical.