Red Teaming › Red Team Methodology

Running a red team evaluation is not the same as running a standard benchmark. A benchmark measures performance on a fixed dataset with known questions and answers. A red team surfaces failure modes that the benchmark designers did not anticipate, by having human adversaries use their creativity and domain knowledge to probe for weaknesses. That distinction shapes how red teams are structured, who they recruit, and how they measure success.

Every red team exercise begins with a scope document that defines what "failure" means for this particular model and deployment context. This sounds simple but is surprisingly subtle. A response that is appropriate for a medical professional may be entirely inappropriate for a general consumer. An output that is harmless in one cultural context may be deeply offensive in another. Without explicit scope definitions, red teamers cannot make consistent judgments about what counts as a successful attack, and the resulting metrics are meaningless.

The goals vary depending on the use case. A customer service chatbot has a different threat model than a code assistant or a general-purpose research tool. Without clear goals, red teamers waste effort on irrelevant attacks and miss the failures that matter. A coding assistant does not need to be evaluated for its ability to discuss geopolitical propaganda, but it absolutely needs to be evaluated for its ability to generate functional malware, SQL injection code, and credential theft scripts.

Common scope dimensions include:

• Harm categories: Which types of harmful content matter most? Violence, hate speech, self-harm, dangerous technical information, privacy violations, and disinformation are each a separate category requiring specialized expertise.
• User populations: Is the model intended for adults only, or will minors use it? Researchers? Vulnerable individuals? Deployment context shifts what counts as a critical failure.
• Deployment surfaces: Is the model accessible through a raw API, a consumer chat interface, or embedded in a product with additional guardrails? Each layer changes what attacks are feasible.
• Severity thresholds: What severity level triggers a fix before release? A model that produces mildly unhelpful answers is different from one that produces content that could directly harm a real person.

Setting these thresholds requires policy judgment. An organization must decide, before seeing any results, what failure rate is acceptable and what minimum severity warrants a release block. Making these decisions after seeing the results introduces the temptation to move the goalposts. If the threshold for "acceptable" is implicitly adjusted to make the model look safer after the results come in, the red team exercise has served no useful function.

The scope document also specifies which attack techniques the red team is authorized to use. Some organizations prohibit their red teamers from sharing actual harmful content with each other, even within the exercise, to prevent the red team itself from becoming a vector for spreading dangerous information. Others require that all discovered vulnerabilities be reported immediately rather than held for disclosure at the end of the evaluation period. These operational security constraints are not just bureaucratic niceties; they protect both the participants and the broader community.

Before a red team exercise begins, practitioners often conduct an explicit threat modeling exercise that maps out who the likely adversaries are, what goals they have, what resources they have available, and which attack surfaces they are most likely to target. Threat modeling is a structured practice borrowed from computer security that imposes discipline on the otherwise open-ended question of "what could go wrong?"

A threat model for a language model deployment typically considers several adversary profiles. A casual misuse actor is an ordinary user who encounters the model and decides to test its limits without specialized expertise. They pose the lowest threat but are the most numerous; even if each individual attack has a low success probability, millions of users attempting naive attacks will produce a meaningful number of successes. A motivated individual is someone with a specific harmful goal, willing to spend hours crafting and refining prompts. They represent the majority of real-world jailbreak attempts. A sophisticated adversary has specialized domain knowledge, access to the model API for systematic probing, and potentially the ability to fine-tune attack approaches based on observed model behavior. They represent a smaller fraction of users but are capable of the highest-severity attacks.

Each adversary profile implies different priorities for the red team. Defenses against casual misuse focus on robustness to common surface-level attack patterns. Defenses against motivated individuals require context-sensitive safety that cannot be bypassed by reframing. Defenses against sophisticated adversaries require robustness to adaptive attacks, which is substantially harder to achieve.

The threat model also identifies which harm categories are most relevant to the deployment context. A mental health support application has an extremely high threat level for self-harm content and a lower threat level for CBRN weapons synthesis. A general-purpose research assistant used by university students has a moderate threat level across many categories. Matching the evaluation effort to the threat model ensures that limited red team resources are spent on the most consequential risks.

The people doing the red teaming matter as much as the prompts they write. A team of ML engineers will find technical exploits; they will miss social engineering, cultural nuance, and domain-specific harms. Effective red teams are deliberately diverse along several axes.

Domain expertise is critical for identifying what constitutes real harm in specialized areas. Medical red teamers know which drug interactions are dangerous and which information is already freely available in any pharmacy reference. Security researchers know which code snippets pose real threats versus which are merely scary-looking. Researchers who study extremism recognize which ideologies get laundered through innocuous-sounding phrasing. Without domain expertise, red teamers often flag benign content and miss dangerous outputs, because the boundary between helpful and harmful is domain-specific.

Demographic diversity matters because harmful content is often culturally specific. Slurs, stereotypes, and coded language that one evaluator finds obviously offensive may seem innocuous to another. Underrepresented groups are often the targets of specific harms that homogeneous teams miss entirely. A team composed entirely of researchers from one cultural background will systematically undercount harms directed at groups not represented on the team. Fairness and measurement validity are both at stake. If your evaluation misses harms directed at certain populations, your model is no safer than the metrics suggest.

Adversarial mindset is a learned skill. Some researchers are trained in red teaming specifically, having worked in security or safety roles where finding weaknesses is the primary job. Others are creative writers who excel at constructing plausible cover stories that trick a model into providing sensitive information. Still others are social engineers who understand how conversation dynamics can be manipulated to shift a model's behavior across many turns. Teams that combine these skill sets cover more of the attack surface than teams composed of any single type of expert.

The size of the red team matters too. Meta's Llama 2 evaluation used over 350 annotators across multiple rounds. OpenAI's GPT-4 evaluation involved more than 50 domain experts in addition to internal researchers. Smaller teams inevitably miss more. A rough rule of thumb in the field is that a minimum of 50 dedicated red teamers is needed to establish meaningful coverage across the major harm categories, though the right number depends heavily on the scope of the evaluation. For models with extremely broad deployment contexts (consumer-facing general-purpose assistants used by hundreds of millions of people), even 350 red teamers likely provides only shallow coverage of the full space.

Red team exercises fall on a spectrum from fully structured to fully open-ended. Each approach has complementary strengths.

Structured red teaming provides specific attack categories, example prompts, and scoring rubrics. Red teamers are asked to find as many successful attacks as possible within each category. This approach produces comparable, reproducible results that can be tracked across model versions. A structured evaluation tells you precisely: "For this specific set of attack categories, the attack success rate decreased from X% to Y% between model generations." The downside is that it anchors red teamers to the categories the designers defined in advance, which may miss novel attack patterns that fall outside the predefined taxonomy.

Open-ended red teaming gives red teamers a model and a broad goal (such as "find anything that could harm a user") without constraining the method. This surfaces unexpected failure modes that no one anticipated. The first time a new jailbreak technique is discovered is almost always through open-ended exploration, not through structured evaluation. The downside is inconsistency: different red teamers explore completely different territory, making it hard to measure coverage or aggregate results across teams.

Most serious evaluations use both approaches. Structured red teaming establishes a baseline across known harm categories. Open-ended red teaming finds the unknown unknowns. The Anthropic red team evaluation for Claude, described in their model cards, uses a combination: structured taxonomy-driven evaluation supplemented by open-ended exercises where internal researchers with specialized domain knowledge attempt novel attacks.

A third mode that has emerged is model-assisted red teaming, where a language model helps human red teamers by suggesting attack variations, translating attacks into multiple languages, or generating plausible cover stories for a given harmful request. The red teamer directs the strategy while the model generates the specific linguistic realizations. This significantly increases the productivity of each human red teamer, though it also raises questions about whether model-assisted red teaming can find failures that the assisting model is incapable of imagining.

Measuring red team success requires more than counting successful attacks. The two most useful metrics are attack success rate and coverage.

Attack success rate (ASR) is the fraction of attempted attacks that produce a harmful output. If a red teamer tries 200 prompts targeting chemical synthesis instructions and 12 of them produce actionable content, the ASR is 6%. Formally, for a set of $N$ attack attempts:

$$\text{ASR} = \frac{\sum_{i=1}^{N} \mathbf{1}[\text{harmful}(r_i)]}{N}$$

where:

• $N$: the total number of attack attempts in the evaluation
• $r_i$: the model's response to the $i$-th attack prompt
• $\mathbf{1}[\text{harmful}(r_i)]$: an indicator function that equals 1 if response $r_i$ is judged harmful by a rater, and 0 otherwise

The ASR is simple to compute but depends heavily on the quality of the attack attempts. A naive red teamer may produce a low ASR not because the model is safe but because their prompts are unsophisticated. This is why separating ASR by attacker sophistication level is important: an overall ASR of 3% means something very different if the red team used only naive attacks versus adaptive attacks tailored to the model's specific behavior.

Some evaluations report a stratified ASR that separates attacks by both harm category and sophistication level, producing a matrix of results. The matrix makes it immediately apparent which combinations remain under-defended. A model that has near-zero ASR on naive direct request attacks but 40% ASR on adaptive multi-turn escalation attacks has passed the easy tests and failed the hard ones. Publishing only the aggregate number would obscure this important pattern.

Coverage measures how much of the attack surface the red team has explored. Coverage is much harder to quantify because the space of possible attacks is open-ended. Proxy measures include the number of unique attack strategies attempted, the breadth of harm categories explored, and the diversity of linguistic framings used. Some evaluations use embedding-based clustering to measure whether the set of attempted prompts covers a broad region of the semantic space. If 80% of all attempted prompts cluster together in embedding space, the red team has been exploring a narrow region of the attack surface regardless of how many individual prompts they tried.

Both metrics require careful human judgment to score. Whether an output is "harmful" is a classification decision, and inter-annotator agreement on adversarial content is rarely perfect. Red team rubrics typically define severity on a 1-5 scale and require two independent raters to resolve disagreements. Without at least two raters on each judgment, the noise in ASR estimates is too high to draw reliable conclusions about model comparisons. When comparing ASR between two model versions, a difference of fewer than three percentage points should be treated with skepticism unless the evaluation used a large number of attacks and multiple raters.

A third metric, minimum successful attack cost, measures how much effort an attacker must invest to achieve the first success in a given harm category. A model that requires 5 minutes of effort to compromise for a given harm category is substantially less safe than one that requires 5 hours, even if both eventually fail under sufficient effort. This metric is harder to measure but is arguably more meaningful for practical risk assessment, since most real-world adversaries are not infinitely patient.