Jailbreaking › Worked Example: A Defends Against B

To make the attack-defense dynamics concrete, let's trace through a single jailbreak scenario end-to-end and analyze why each defense layer succeeds or fails against it.

The attack scenario: An attacker wants a model to explain how to synthesize a controlled substance. They try a crescendo attack: starting with general chemistry questions, escalating to specific precursor chemicals, then asking about synthesis steps once the conversation has established an educational framing.

Layer 1: Input classifier. The first turn asks about "common laboratory chemicals." The input classifier scores this as benign (low jailbreak signal score: no override language, no roleplay framing). It passes through. The second turn asks about synthesis reactions. Still scored as benign. By turn 4, the questions are more specific, but a single-turn classifier still evaluates each turn in isolation and sees only a chemistry question without explicit harm signals.

Layer 2: The model itself. At turn 5, the model's own safety training triggers on the explicit synthesis request. However, because the previous turns have established an educational context and the model has already provided related information, the safety training is in tension with the helpfulness training. In an older model, this tension might be resolved toward compliance; in a well-tuned recent model, the refusal behavior should still trigger because the final turn's request is unambiguous.

Layer 3: Output classifier. Even if the model generates a partial response before its safety training kicks in and revises it, the output classifier provides a second line of defense. Any output containing synthesis instructions should be flagged and blocked regardless of how the jailbreak was constructed.

Layer 4: Monitoring. The sequence of turns is logged. The crescendo pattern (escalating specificity across turns) may be detected by a pattern-matching monitor that flags conversations with certain topic trajectories. This generates an alert for human review even if the layers above successfully blocked the final output.

This layered structure illustrates defense in depth. No single layer is sufficient: the input classifier misses crescendo attacks, the model's safety training can be in tension with helpfulness in the middle of a manipulated conversation, and even output classification can be evaded by sophisticated attacks. But each layer catches a different class of attack, and attacks that defeat all layers simultaneously are much harder to construct.