Red Teaming › Limitations and Practical Challenges

Red teaming is necessary but not sufficient for AI safety, and its limitations deserve careful attention. Understanding these limitations helps practitioners design better evaluations and avoid false confidence from passing red team tests.

The space of possible attacks is infinite. Even the most thorough red team exercise samples a tiny fraction of it. Failure modes that no red teamer thought to try remain undiscovered until they appear in the wild. This is not an argument against red teaming; it is an argument for treating red team results as a lower bound on failure rates, not an upper bound. A model that passes all red team tests has not been proven safe; it has passed the tests its evaluators designed.

This coverage problem is particularly acute for novel attack techniques. When a new jailbreak technique circulates on social media, it is often because a creative user discovered something that no red teamer tried. The model's defenses are necessarily backward-looking: they address what has already been found. The gap between what the red team found and what adversarial users will eventually discover is unknown and unknowable.

The coverage problem does not have a technical solution. Automated red teaming can increase the volume of attacks tested, but volume does not equal coverage of the conceptual space. A billion near-identical prompts provide less coverage than a thousand carefully designed prompts spanning diverse attack mechanisms and harm categories.

Whether a given output is harmful requires human judgment, and human judgment varies. Two annotators examining the same response may reach different conclusions, especially for borderline cases involving satire, dark fiction, or dual-use technical information. Inter-annotator agreement on adversarial content is typically lower than on standard annotation tasks.

This subjectivity propagates into the metrics. Attack success rates are averages over many independent judgments, each made by a human with their own background, expertise, and cultural context. Reported ASR numbers should be interpreted with error bars proportional to annotator disagreement, but most published red team reports do not provide this uncertainty estimate. When comparing ASR across model versions, even a 2-3 percentage point change may be within the noise of annotator disagreement rather than a reliable signal of improved safety.

The subjectivity problem also interacts with the incentive structure of red teaming. An organization that publishes a red team report has an incentive to make its model look as safe as possible. This does not require outright dishonesty; it can manifest through selection of evaluation criteria that the model performs well on, framing of results to emphasize improvements over a weaker baseline, or choice of a harm threshold that classifies borderline outputs as safe. External auditing of red team methodology and results, by parties without a stake in the model's commercial success, addresses this concern but is rarely conducted in practice.

Red teaming and jailbreaking exist in a feedback loop. When red teams publish their findings, they provide a roadmap for more sophisticated adversaries. The specific attack patterns documented in published red team reports are quickly incorporated into community jailbreak databases and adopted by users who were previously less sophisticated.

This creates pressure to keep red team findings confidential, which conflicts with the scientific value of open publication. The field has not settled on a consensus for how much detail to publish, when to publish it, and how to structure responsible disclosure for AI safety findings. The tension between transparency (which helps the field improve) and operational security (which prevents attackers from directly reusing documented techniques) is a real dilemma with no clean resolution.

Defenders face an asymmetric problem. An attacker needs to find only one successful attack; a defender needs to block all possible attacks. Publishing red team findings helps both sides, but it helps attackers more in the short run (they immediately learn new attack techniques) and helps defenders more in the long run (the research community can develop systematic mitigations). The optimal disclosure timing depends on the severity of the harm and the maturity of available mitigations.

Automated red teaming is efficient but introduces its own failure modes. The harmfulness classifier that scores attack success may be systematically wrong in certain domains. An automated system trained to flag explicit harmful content will miss attacks that achieve harmful goals through technically clean language. The classifier might also flag false positives, marking safe responses as harmful because they contain surface-level keywords without context.

More fundamentally, automated systems optimize for metrics. If the classifier is miscalibrated, automated red teaming will optimize for the miscalibrated metric, not for actual harm reduction. Human oversight of automated red team results is essential, not optional. The automation provides scale; the human oversight provides calibration to actual harm.

A less obvious limitation of automated red teaming is that it tends to find attacks that exploit the model's current safety training deficits rather than fundamental conceptual vulnerabilities. If the safety training has a blind spot for a particular kind of harm, the automated system will find many attacks that exploit that blind spot, creating the illusion of broad coverage while concentrating on a single failure mode. Periodically auditing the diversity of successful attacks, not just counting them, helps detect this failure mode.

Red teaming focuses on preventing specific harmful outputs. It says relatively little about whether the model is aligned with human values in a deeper sense. A model could pass every red team test and still systematically mislead users in subtle ways, amplify existing biases, or behave unpredictably under distribution shift. Red teaming also cannot detect harms that emerge from aggregate behavior across millions of users, where individual interactions are benign but the population-level effect is harmful.

Consider a model that provides excellent individual advice on stock investment. Every individual response passes all red team tests for accuracy and harm. But if the model systematically recommends a particular investment strategy because of patterns in its training data rather than because of reliable expertise, the aggregate effect across millions of users could cause substantial financial harm. No individual interaction looks harmful; the harm emerges at scale.

The interpretability and alignment chapters in this book cover complementary approaches that address these broader concerns. Red teaming is best understood as one layer of a defense-in-depth strategy, not as a standalone safety certificate. It provides empirical measurement of specific failure modes; interpretability provides mechanistic understanding of why failures occur; alignment training addresses the underlying objective misalignment that allows failures to exist.

A rarely discussed limitation of red teaming is the psychological toll on the people who conduct it. Professional red teamers spend extended periods deliberately searching for and evaluating disturbing content: child sexual abuse material, detailed violence, extremist propaganda, and content designed to exploit vulnerable people. The mental health effects of repeated exposure to this material are significant and well-documented in adjacent fields (content moderation, trauma research).

Organizations conducting red team evaluations have a responsibility to implement psychological support structures, including regular check-ins with mental health professionals, limits on daily exposure hours, and clear opt-out mechanisms when evaluators feel they need a break. The failure to address this issue is an ethical lapse and creates a practical problem as well, since red teams with high turnover and burnout cannot provide the consistent coverage that effective safety evaluation requires.