Constitutional AI › CAI Training: Combining the Two Stages

With the SL-CAI model and the CAI reward model in hand, the final policy is trained using the standard RLHF fine-tuning objective, typically PPO. The policy is initialized from the SL-CAI model and then optimized to maximize the reward model score subject to a KL divergence penalty that prevents the policy from drifting too far from the SL-CAI model:

$$\mathcal{J}_{\text{PPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)} \left[ r(x, y) - \beta \cdot \text{KL}\left[\pi_\theta(y \mid x) \,\|\, \pi_{\text{SL-CAI}}(y \mid x)\right] \right]$$

where:

• $\pi_\theta$: the policy being optimized
• $\pi_{\text{SL-CAI}}$: the SL-CAI model (used as KL reference)
• $r(x, y)$: the CAI reward model's score for response $y$
• $\beta$: the KL penalty coefficient, controlling how much the policy can deviate from the SL-CAI reference
• $\text{KL}[\cdot \| \cdot]$: the Kullback-Leibler divergence

The KL penalty serves the same function as in standard RLHF: it prevents reward hacking, where the policy learns to produce responses that score highly according to the reward model but in ways that do not reflect alignment. Without the penalty, the policy would likely learn degenerate outputs that exploit the reward model's weaknesses. The SL-CAI model is an anchor: the final policy is allowed to improve over SL-CAI according to the reward model, but it cannot stray so far that it loses the broadly safe behaviors learned in the supervised stage.

The coefficient $\beta$ is a critical hyperparameter. Setting $\beta$ too high prevents the RL stage from improving over the SL-CAI baseline, because the KL penalty overwhelms the reward signal. Setting $\beta$ too low allows the policy to drift into reward-hacking territory, producing responses that score highly on the reward model but are subtly unsafe or unhelpful in ways the reward model failed to capture. The optimal value of $\beta$ depends on the quality of the reward model and the diversity of the training prompts; empirically, values in the range of 0.01 to 0.1 are typical.

The complete CAI training process involves four models that interact across three distinct stages.

The base language model is the starting point, typically a large pre-trained model that has undergone standard supervised fine-tuning on helpful responses. This base model is capable but not safety-trained, which is why it responds to red-team prompts.

The SL-CAI model is produced by the supervised learning stage: the base model fine-tuned on (prompt, revised-response) pairs generated through critique and revision. This model is substantially safer than the base model and is both the starting point for RL fine-tuning and the anchor for the KL penalty.

The CAI reward model is produced from AI-generated preference data. It takes a (prompt, response) pair as input and outputs a scalar harmlessness score. This model encodes the constitutional preferences learned from having the SL-CAI model evaluate response pairs against constitutional principles.

The final CAI policy starts from the SL-CAI model and is fine-tuned with PPO to maximize the CAI reward model score. This final model is what gets deployed.

In practice, Anthropic also maintains a separate "helpfulness" reward model trained on human preference data for helpful (non-safety-critical) tasks. The final policy is optimized against a combination of the harmlessness reward from the CAI reward model and the helpfulness reward from the human-labeled reward model. This dual-objective training prevents the model from becoming overly cautious to the point of refusing benign requests. The combined reward function is a weighted sum, and the relative weights determine how the model handles tradeoffs between the two objectives. Tuning these weights is itself a form of value choice, analogous to writing the constitution: it reflects a view about how much helpfulness should be sacrificed to achieve a given level of harmlessness.