Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Jailbreaking › Jailbreak Success Rates Across Techniques

Michael BrenndoerferJanuary 13, 20263 min read

Language AI HandbookAll chapters

Part 60 of 67Safety and Security

Chapter 459Jailbreaking › Jailbreak Success Rates Across Techniques

Reading section 7 of 18 · Jailbreak Success Rates Across Techniques · 3 minSection 7 of 18 · 3 min

Section 7 of 18 · 3 min read

Jailbreaking › Jailbreak Success Rates Across TechniquesOne section · Sections in this chapter7 / 18 ↓

Sections in this chapter18

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Empirical research has measured the effectiveness of different jailbreak categories across different models and time periods. Understanding these patterns helps prioritize defense efforts.

In[9]:

Code

# Attack success rate data (approximate, based on published research)
# Rows: attack types; Columns: model generations
# Sources: Wei et al. (2023), Zou et al. (2023), Chao et al. (2023)

attack_types = [
    "Direct Request",
    "Roleplay Framing",
    "Token Encoding",
    "Crescendo",
    "Competing Objectives",
    "GCG Suffix",
    "PAIR (Automated)",
]

# Attack success rates (%) across three model generations
# Early (GPT-3.5 era), Mid (GPT-4 era), Recent (GPT-4o era)
# These represent average ASR on harmful behavior benchmarks

early_asr = [12.0, 68.0, 52.0, 55.0, 48.0, 85.0, 72.0]
mid_asr = [5.0, 38.0, 28.0, 32.0, 25.0, 67.0, 54.0]
recent_asr = [3.0, 18.0, 12.0, 20.0, 14.0, 45.0, 35.0]

Out[10]:

Visualization

Grouped bar chart comparing jailbreak attack success rates across 7 attack types for three model generations, showing declining effectiveness over time. — Attack success rate (ASR) across jailbreak techniques and three model generations. All categories show declining effectiveness over time as safety research has matured, but gradient-based attacks (GCG suffix) and automated attacks (PAIR) maintain relatively higher success rates even against recent models. Direct requests remain very low throughout, confirming that basic safety training handles naive attempts well.

Three patterns stand out in the data. First, all attack categories have become less effective over time as safety research has improved. Second, gradient-based and automated attacks maintain higher success rates than manual techniques, because they can systematically explore the input space rather than relying on human creativity. Third, even against the most recent models, success rates above zero persist for most attack types, confirming that no current defense achieves full robustness.

The persistence of gradient-based attacks even against heavily hardened models is particularly telling. GCG and its variants can find adversarial inputs that evade safety training because the safety training objective and the adversarial optimization objective are operating at different scales: safety training covers a finite distribution of training examples, while gradient-based attacks can probe the continuous space between those examples.