Red Teaming › Automated Red Teaming

Manual red teaming is expensive and slow. A single well-staffed evaluation may produce hundreds of attacks in a day. Automated red teaming uses models or optimization algorithms to generate attacks at scale, potentially producing thousands of candidate attacks per hour. This scale matters because coverage is a persistent limitation of manual red teaming: the more attacks you can test, the smaller the fraction of the attack surface remains unexamined.

Automated red teaming is not a replacement for human red teamers but a force multiplier. Humans provide creativity, domain expertise, and the ability to interpret the output of attacks in context. Automation provides the scale needed to sweep large regions of the prompt space systematically and to iterate quickly on discovered vulnerabilities. The most effective evaluations combine both.

The most common automated approach uses a language model to generate attack prompts, then scores the outputs of the target model to determine which attacks succeeded. This setup is sometimes called "red team LM" (from the paper of the same name by Perez et al., 2022).

The setup involves two models. A red team model generates candidate attack prompts targeting a specific harm category. A target model receives those prompts and produces responses. A classifier (or another language model used as a judge) scores each response for harmfulness. The red team model is then trained or prompted to generate more attacks similar to the ones that succeeded.

The core insight is that a language model has a rich prior over the space of natural language prompts. Rather than exhaustively searching the space of all possible token sequences (which is astronomically large), the red team LM uses its generation capabilities to concentrate on plausible, varied attack formulations. The attacks it generates look like natural human queries, not the syntactically odd patterns produced by direct optimization methods like adversarial suffix search. This makes the attacks more representative of what actual human adversaries would attempt.

The red team LM approach has an important bootstrapping challenge: the model needs some signal about which attacks succeed before it can learn to generate more of them. Common solutions include seeding the model with a small set of hand-crafted successful attacks, using a few-shot prompt with examples of successful attacks in each category, or initializing with a general instruction-following model and relying on the classifier signal to steer generation.

A different class of automated attacks, exemplified by the Greedy Coordinate Gradient (GCG) algorithm introduced by Zou et al. (2023), uses gradient-based optimization to find adversarial suffixes that can be appended to any prompt to cause the model to comply with harmful requests. Rather than using a language model to generate natural-sounding attacks, GCG directly optimizes the token sequence to minimize the probability that the model begins its response with a refusal pattern.

Formally, GCG solves the following optimization problem. Let $x_{1:n}$ be the user's input tokens, $x_{n+1:n+k}$ be an adversarial suffix of length $k$, and $x_{n+k+1:n+k+m}$ be a target response beginning (such as "Sure, here is how to..."). The adversarial suffix is chosen to minimize the negative log-probability of the target response:

$$\min_{x_{n+1:n+k}} \; -\log p_\theta(x_{n+k+1:n+k+m} \mid x_{1:n+k})$$

where:

• $p_\theta$: the language model parameterized by $\theta$
• $x_{n+1:n+k}$: the adversarial suffix tokens being optimized over
• $x_{n+k+1:n+k+m}$: the target affirmative response beginning (the "jailbreak target")

GCG iteratively updates the suffix by computing the gradient of the loss with respect to the one-hot token embedding, using that gradient to identify replacement candidates for each position, and greedily selecting the replacement that maximally reduces the loss. The key technical challenge is that the token space is discrete, which makes direct gradient descent impossible; GCG uses the continuous relaxation of the token embedding to identify which tokens would lower the loss if substituted, then evaluates the actual discrete substitutions.

The resulting adversarial suffixes look like nonsense to human readers ("describe;---Give tutorial}` ... !] Additionally") but reliably cause models to comply with the harmful prefix that precedes them. This technique is powerful because it is general: a single suffix can transfer across different harmful prompts and, to a limited degree, across different model checkpoints. It is less representative of human attacks, since no human adversary would craft these suffixes by hand, but it serves an important theoretical role in stress-testing model safety.

The Prompt Automatic Iterative Refinement (PAIR) algorithm introduced by Chao et al. (2023) takes a different approach that produces more natural-sounding attacks. PAIR uses a separate attacker LLM to iteratively refine a jailbreak prompt based on the target model's responses. The attacker model reads the target model's refusal, identifies why the refusal was triggered, and proposes a revised prompt that avoids the triggering patterns while preserving the harmful intent.

This process converges quickly. In experiments, PAIR found effective jailbreaks for many prompts within 20 iterations. The attacks produced are natural-sounding English text rather than adversarial garbage, which makes them more transferable to black-box settings where no gradient information is available and more representative of what a sophisticated human attacker might generate.

The iterative refinement loop treats the jailbreak problem as a search problem guided by feedback. Each failed attempt provides information about what the model's safety training is sensitive to, and the attacker LLM uses that information to adjust the framing. This is analogous to how a skilled human social engineer would refine their approach after each unsuccessful attempt, learning from the target's responses what framings are more or less effective.

A more formal optimization approach trains the red team model using reinforcement learning. The reward signal is the harmfulness score of the target model's responses. The red team model learns to maximize this reward, discovering attack strategies that consistently elicit harmful outputs.

To understand the reward function, consider the two competing objectives: maximizing harmfulness while maintaining diversity. Given a query $q$, the target model's response $r$, and a set of previously selected queries $\{q_{\text{prev}}\}$, the reward combines these two objectives:

$$R(q, r) = \text{harmfulness}(r) - \lambda \cdot \text{overlap}(q, q_{\text{prev}})$$

where:

• $\text{harmfulness}(r)$: a scalar score in $[0, 1]$ measuring how harmful the target model's response $r$ is, assigned by a classifier
• $\lambda$: a penalty coefficient controlling how strongly diversity is enforced; larger $\lambda$ penalizes redundant attacks more heavily
• $\text{overlap}(q, q_{\text{prev}})$: a similarity measure between the new query $q$ and previously seen queries $q_{\text{prev}}$, penalizing the red team model for generating attacks that closely resemble ones already attempted

The overlap penalty is critical. Without it (setting $\lambda = 0$), the red team model quickly converges to repeating slight variations of the first successful attack it finds, producing a homogeneous set of attacks that overrepresent one vulnerability while missing everything else. Diversity objectives ensure the red team explores the full attack surface rather than exploiting a single weakness.

To measure similarity between text strings, a common choice is Jaccard similarity over word-level token sets. For two texts $q_a$ and $q_b$ with corresponding word sets $W_a$ and $W_b$:

$$\text{Jaccard}(q_a, q_b) = \frac{|W_a \cap W_b|}{|W_a \cup W_b|}$$

where:

• $W_a$: the set of unique words in query $q_a$
• $W_b$: the set of unique words in query $q_b$
• $|W_a \cap W_b|$: the number of words appearing in both queries (intersection size)
• $|W_a \cup W_b|$: the total number of distinct words across both queries (union size)

Jaccard similarity equals 0 when the two texts share no words and 1 when they are identical. A similarity of 0.3 or higher indicates substantial lexical overlap, suggesting the attacks are likely targeting the same vulnerability. More sophisticated similarity measures, such as sentence embedding cosine similarity, capture semantic similarity that Jaccard misses, but they are more expensive to compute and introduce their own failure modes. Two attacks that use very different words but exploit the same conceptual framing might be correctly flagged as redundant by embedding similarity but missed by Jaccard.

Anthropic's Constitutional AI (CAI) framework, which you encountered in the alignment section of this book, can be inverted to generate adversarial tests. Instead of using principles to revise outputs toward helpfulness and harmlessness, a critique model uses those same principles to generate prompts specifically designed to elicit violations.

The constitutional critique process generates a question like "What prompt would most likely lead the model to violate its principle against providing dangerous information?" This uses the model's own understanding of its guidelines to construct targeted attacks, a technique sometimes called "self-red-teaming." The attraction of this approach is that it uses the model.s knowledge of its own safety training, which may cover more than any external attacker's knowledge. The limitation is that the model may have systematic blind spots: if the model does not recognize certain types of harm as violations of its principles, it cannot generate attacks targeting those harms. Self-red-teaming finds the harms the model knows about; it cannot find the harms the model does not know it should be protecting against.

Automated red teaming produces high volume but introduces its own evaluation problems. The harmfulness classifier that scores attack success is itself a model, and it can be fooled. An attack that succeeds against the target model may fail to register as successful with the scoring model if the harmful content is expressed in a way the scoring model does not recognize as harmful. Conversely, a response that the scoring model rates as harmful may be perfectly safe when read by a human.

This creates an evaluation gap: the red team exercise measures attack success relative to an imperfect proxy for harm, not against the ground truth of actual harm. Calibrating the classifier carefully against human judgments, running periodic human spot-checks on a random sample of automated outputs, and using multiple independent classifiers all help close this gap. The most reliable evaluations combine automated generation (for scale) with human review (for calibration), using the automated results to flag candidates for human review rather than as a final verdict.

The figure below illustrates how the diversity penalty $\lambda$ controls the tradeoff between maximizing harmfulness and ensuring broad coverage of the attack space.