Jailbreaking › Summary

Jailbreaking exposes the gap between learned safety behaviors and true alignment. Key takeaways:

• Safety training is statistical, not logical. Models learn to refuse harmful requests for inputs similar to training data, but novel phrasings, encodings, or framings can bypass learned refusals. The underlying knowledge is not deleted; only the expression pathway is suppressed.
• Manual techniques exploit roleplay framing, instruction hierarchy manipulation, token encoding, crescendo attacks, and competing objectives. These are the cheapest attacks to attempt and the most common in the wild.
• GCG (Greedy Coordinate Gradient) formulates jailbreaking as discrete optimization, finding adversarial suffixes that push model representations toward affirmative response regions. The resulting suffixes are universal (work across behaviors) and transfer across models.
• PAIR (Prompt Automatic Iterative Refinement) uses an LLM attacker to iteratively refine jailbreaks against a target model, requiring only black-box API access. TAP extends this with tree-structured search for more efficient attacks.
• Defenses include input filtering, perplexity-based filtering, output filtering, adversarial fine-tuning, constitutional AI, randomized smoothing, and prompt hardening. Each involves trade-offs between attack block rate and false positive rate on legitimate queries.
• No defense achieves full robustness. The attack success rate can be reduced but not eliminated. Defense in depth, continuous red teaming, and monitoring are the realistic operational posture.
• Multi-turn and multi-modal attacks expand the attack surface beyond single-turn text, requiring defenses that reason about conversational context and handle multiple input modalities.
• The alignment tax and robustness tax are practical costs of safety mechanisms: capability degradation from over-refusal and performance overhead from defensive layers. These taxes must be accepted or mitigated based on the specific deployment's threat model.