Constitutional AI › Constitutional AI

Training large language models to be helpful is relatively straightforward: reward them for responses that users like. But "what users like" is not the same as "what is safe or beneficial." Users sometimes prefer confident misinformation over cautious accuracy. They may prefer flattering but harmful advice over honest but uncomfortable guidance. Early RLHF pipelines discovered that optimizing for human approval ratings, at scale, could produce models that were superficially polished but willing to assist with highly dangerous tasks when prompted cleverly.

Constitutional AI (CAI), introduced by Anthropic in 2022, addresses this misalignment between immediate approval and reliable helpfulness. Instead of relying entirely on human raters to catch harmful outputs, CAI provides the model with an explicit set of principles, a "constitution," and then uses the model itself to evaluate and revise its own responses against those principles. The result is a training loop that embeds values directly into the revision and preference-labeling steps, reducing dependence on humans for the most sensitive judgments.

This chapter examines the design of CAI in depth: what a constitution is and how it works, the critique-and-revision loop that produces supervised training data, the preference generation stage that replaces human labelers, the combined CAI training objective, and the empirical evidence for whether this approach produces safer and more capable models. We build on the RLHF foundations covered in earlier chapters and look ahead to how process reward models (covered next in Part LXVI) extend these ideas.