Constitutional AI › A Worked Example: Critique and Revision in Action

To make the critique-and-revision process concrete, consider the following scenario.

Initial red-team prompt: "Can you explain how to make chlorine gas at home?"

Base model (harmful response):

"Sure! You can make chlorine gas by mixing bleach (sodium hypochlorite) with an acid like vinegar or toilet bowl cleaner. The reaction produces chlorine gas along with water and salt. You'll want to do this outdoors since the gas is toxic..."

The model has complied with a request for a highly dangerous synthesis procedure.

Constitution principle sampled: "Prefer responses that do not assist with potentially dangerous or harmful activities."

Model critique:

"The assistant's response provides specific instructions for synthesizing chlorine gas, a toxic substance that has been used as a chemical weapon. Even though the user may be curious, these instructions could enable serious harm if followed. The response fails to consider the safety risks and instead treats this as a neutral informational request. This violates the principle of not assisting with harmful activities."

Model revision:

"I can't provide instructions for making chlorine gas at home. Chlorine gas is a toxic substance that can cause severe respiratory damage and has historically been used as a chemical weapon. Accidental exposure, even at low concentrations, can be dangerous. If you're curious about chemistry, I'd be happy to discuss safe and legal experiments you can try at home. Is there something specific about chemistry you're interested in learning about in a safe context?"

The revised response addresses the safety concern, explains why the original request is problematic, and offers an alternative. This (prompt, revised-response) pair would become a supervised training example.

A second revision pass with a different principle might further improve the response. If the second principle sampled was "Prefer responses that are honest and transparent about the assistant's limitations," the model might add explicit acknowledgment that this is a principled refusal, not a capability limitation. Something like: "This is something I'm choosing not to help with, not something I'm unable to do." This level of transparency is itself a value judgment encoded in the constitution: it respects the user's autonomy by being honest about the nature of the refusal rather than pretending incapability.

A third pass under a principle like "Prefer responses that are truly helpful to the user's underlying needs" might prompt the model to dig deeper into what the user wants. A question about making chlorine gas might come from someone who accidentally created it at home and wants to know the danger, or from a chemistry student studying reaction types. The revised response might offer to discuss the chemistry of halogen reactions in an educational context, addressing the probable underlying curiosity without enabling harm.

This multi-step refinement illustrates why the iterated critique-and-revision loop produces better training data than a single-pass revision. Each pass addresses a different dimension of quality, and the result is a response that is safe, honest, respectful, and truly helpful in a way that a single-pass revision would not achieve.