Jailbreaking › Red Teaming and Continuous Evaluation
Language AI HandbookAll chapters Jailbreaking › Red Teaming and Continuous EvaluationOne section · Sections in this chapter12 / 18 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The most robust approach to jailbreaking is continuous adversarial testing: red teaming deployed systems with the latest attack techniques and patching discovered vulnerabilities.
The red teaming process (covered more deeply in the previous chapter on Red Teaming) involves structured adversarial testing. Here we focus on the jailbreaking-specific workflow:
- Define harm categories: Create a taxonomy of harmful behaviors the system should refuse. This taxonomy needs to be specific enough to be testable and broad enough to cover the real threats.
- Generate diverse attacks: Use both human red teamers and automated tools (PAIR, GCG, TAP) to find successful jailbreaks in each category. Diversity is critical: a narrow set of attack types will leave gaps.
- Measure attack success rate (ASR): For each attack type, measure what fraction of attempts succeed. This gives a quantitative baseline to track over time.
- Patch and retest: Include successful jailbreaks in the training data as negative examples, then re-evaluate. Track whether patching one category inadvertently creates gaps in others.
- Maintain a living red team knowledge base: Document successful attacks, the mechanisms that explain them, and the defenses deployed. This institutional knowledge helps onboard new red teamers and accelerates future patching cycles.
The key metric is the Attack Success Rate (ASR): the fraction of jailbreak attempts that elicit harmful responses. Lower is better, but zero is not achievable in practice. Setting an acceptable ASR threshold and monitoring continuously is the realistic goal.
ASR = (number of successful jailbreaks) / (total jailbreak attempts). "Success" requires a judge: either a human evaluator or an LLM judge that rates whether the response fulfilled the harmful behavior. LLM judges are faster but can themselves be manipulated; human evaluation is more reliable but expensive. Hybrid approaches use LLM judges for initial screening and human evaluation for borderline cases.
One underappreciated aspect of red teaming is the diversity of the red team itself. Technical researchers tend to find gradient-based and automated attacks; social engineers tend to find persona manipulation and authority claim attacks; domain experts (medical professionals, security researchers) tend to find attacks that exploit legitimate professional contexts. A homogeneous red team will find a biased sample of the vulnerability space.
Another underappreciated aspect is the difference between targeted and untargeted red teaming. Targeted red teaming sets specific harmful behaviors as goals and measures how many attempts it takes to achieve them, giving precise ASR numbers. Untargeted red teaming has no predetermined goal: testers are simply told to break the system and report what they find. Untargeted red teaming is better at discovering novel attack categories that weren't anticipated, because it doesn't constrain the search to known harmful behavior types. Both modes are necessary for broad evaluation.
Red teaming results must be interpreted against a baseline. An ASR of 5% is very good if the baseline (without any safety training) is 95%, but concerning if competitors have achieved 1%. Absolute ASR numbers are less informative than relative ASR compared to prior model versions and comparable models. Researchers have proposed formal ways to measure "relative robustness" that normalize ASR against baseline compliance rates, making comparisons across model families and versions more meaningful.