Constitutional AI › AI Feedback for Preference Learning

The second stage of Constitutional AI training replaces human preference labeling with AI-generated preferences. This is where the "AI" in the feedback loop becomes most apparent, and where CAI makes its most significant departure from standard RLHF.

In standard RLHF, human raters compare pairs of model responses and indicate which is better. These comparisons are used to train a reward model, which is then used to guide RL fine-tuning. In Constitutional AI, the SL-CAI model (trained in the first stage) takes the role of the human rater. The model that has already been trained to comply with constitutional principles is now asked to evaluate other responses against those same principles, generating the preference data that will train the reward model.

This structure is sometimes called "reinforcement learning from AI feedback," or RLAIF, and it has a self-referential structure. The model's preferences are shaped by its constitution, and those preferences then train a reward model that will shape the next iteration of the model's behavior. The constitution is the fixed point in this loop: it is the stable set of principles that the model applies consistently across all stages of the feedback process.

The process starts by sampling two responses from the SL-CAI model for each prompt in a dataset. These two responses provide the comparison pair. The constitution principle is then presented to the model as context, and the model is asked which response better satisfies the principle.

The prompt structure for preference labeling is:

"Consider the following pair of responses to the request '[prompt]': Response 1: [response A] Response 2: [response B] According to the principle '[principle]', which response is better? Respond with 'Response 1' or 'Response 2'."

The model outputs a preference label. To reduce variance, multiple constitution principles are sampled and the model is asked to evaluate each pair under each principle. The preferences are then aggregated, typically by majority vote, to produce a final label. This aggregation is important: a single principle might give an ambiguous result, but the majority vote over several principles converges to a reliable label for most pairs.

This approach generates a large dataset of (prompt, response A, response B, preference) tuples entirely automatically. The scale of this dataset far exceeds what human labeling could produce at reasonable cost, and the labeling is consistent because the same model applies the same principles to every pair.

The preference pairs generated by the SL-CAI model are used to train a reward model using the standard Bradley-Terry framework. The Bradley-Terry model treats pairwise preferences as the result of an underlying comparison between scalar "strength" values, which in this context are the scalar reward scores the model assigns to each response.

Given a prompt $x$ and two responses $y_A$ and $y_B$, the reward model assigns scalar reward values $r(x, y_A)$ and $r(x, y_B)$. The training objective maximizes the log probability that the preferred response receives a higher reward:

$$\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma\left( r(x, y_w) - r(x, y_l) \right) \right]$$

where:

• $y_w$: the "winning" (preferred) response as labeled by the AI
• $y_l$: the "losing" (less preferred) response
• $r(x, y)$: the scalar reward output of the reward model for response $y$ given prompt $x$
• $\sigma$: the sigmoid function, $\sigma(z) = 1 / (1 + e^{-z})$

The sigmoid function converts the reward difference into a probability. When $r(x, y_w) \gg r(x, y_l)$, the sigmoid output is close to 1, meaning the model is very confident the winner has higher reward. When the rewards are close, the sigmoid output is close to 0.5, showing uncertainty. Maximizing the log probability of the correct label pushes the reward model to assign clearly different scores to clearly different-quality responses, while remaining calibrated about ambiguous pairs.

This is identical to the reward model training objective in standard RLHF, with the only difference being that the preference labels came from an AI rather than a human. The resulting reward model, sometimes called the CAI-RM or the harmlessness reward model, encodes a preference for responses that satisfy constitutional principles.

The reward model architecture is typically a language model with an added linear head that maps the hidden state corresponding to the last token to a scalar value. The same base language model that generates responses can serve as the reward model, initialized from the same checkpoint, which is efficient and ensures the reward model has the same language understanding as the policy.

Because the preference labels are AI-generated, the dataset can be large. Anthropic's initial CAI experiments generated hundreds of thousands of preference pairs, far more than would be feasible with human labeling. This scale is important: larger preference datasets produce more reliable reward models, which in turn produces better-aligned policies.

The preference labels also have a different character than human labels. Human raters vary in how strictly they apply principles, what context they bring, and how they trade off different considerations. AI-generated labels are consistent: the same model, applying the same principle to the same pair of responses, will produce the same label every time. This consistency reduces label noise in the reward model training data, which translates into a reward model that is less susceptible to the rater inconsistency problems that plague human-labeled preference datasets.

Label consistency has a subtle but important effect on reward model training. When human labels are inconsistent, the reward model must average over contradictory signals, which pushes it toward assigning similar rewards to responses that different raters disagree about. With consistent AI labels, the reward model can learn sharper distinctions, assigning clearly higher rewards to responses that the AI consistently prefers and lower rewards to responses it consistently rejects.

The downside of AI-generated preferences is that they can inherit biases from the model generating them. If the SL-CAI model has systematic blind spots, those blind spots will appear in the preference data, which will propagate into the reward model. For example, if the model is systematically more cautious about requests involving certain topics or demographics, the preference data will reflect that bias, and the final policy will inherit it. Anthropic addresses this by combining AI feedback with some human feedback, particularly for cases where the AI's judgment seems unreliable or where human values require context-sensitive judgment.

The bias propagation risk is one of the reasons why maintaining human oversight of the preference generation process matters. Purely automated preference labeling creates a closed loop in which the model's biases are amplified rather than corrected. Human review of a sample of AI-generated preferences, and targeted human labeling for cases where AI preferences seem unreliable, keeps the loop from diverging in a problematic direction.