Red Teaming › Red Team Findings from Major Evaluations

Several organizations have published red team findings from evaluations of their models, giving a window into what failures exist in practice and how their rates change across model generations. These published findings serve both a safety function (informing other developers) and a transparency function (showing to the public what due diligence was performed before release). Studying them reveals patterns that apply broadly across model families and organizations.

Anthropic publishes red team findings as part of their model cards and in research papers. Their evaluations use a combination of internal red teams and contracted annotators with domain expertise in areas like biology, chemistry, and cybersecurity.

Key findings from their published work include several important patterns. On CBRN uplift (chemical, biological, radiological, nuclear information), a response that would be dangerous in the hands of an expert may be useless to a layperson. Anthropic's evaluations separate the questions of whether information was provided from whether it provides meaningful uplift to actual harm. This distinction matters enormously for setting release criteria: information that is freely available in any library does not need the same level of protection as information that materially advances an attacker.s capabilities. A model that provides information already in any college chemistry textbook has not created new risk; a model that synthesizes and explains information that would materially help someone who lacks the background to do the synthesis themselves has created real uplift.

Persuasion and manipulation are harder to evaluate than factual harm. A model that generates psychologically manipulative content may not obviously violate any harm category rule, requiring raters with expertise in behavioral psychology to identify. Emotional manipulation that could exploit vulnerable users is a distinct harm category from straightforwardly dangerous factual information, and it requires different evaluation expertise.

Refusal over-caution is documented alongside harmful outputs. Models that refuse too many legitimate requests create different safety problems by pushing users toward unconstrained alternatives. A user who cannot get reasonable medical information from a safety-aligned model does not become more safe; they seek the same information from a source with no safety training at all. Anthropic's published model cards include over-refusal rates alongside harmful output rates, treating both as failure modes to be minimized.

The GPT-4 System Card published by OpenAI documents extensive red team exercises conducted before release. The system card is notable for its level of detail: it describes what the red team found, how the evaluation was structured, and what expert domains were included.

A central finding is that GPT-4 showed significant improvement over GPT-3.5 on structured harm benchmarks, but also showed more sophisticated capabilities for generating harmful content when the safety training was bypassed. More capable models require more capable safety mitigations. This is a fundamental tension in AI safety: the same capabilities that make a model more useful also make it potentially more dangerous if misused. A model that cannot generate coherent long-form text cannot generate coherent bomb-making instructions; a model that excels at technical writing can, in principle, produce excellent technical instructions for anything.

Domain expert red teamers identified failure modes that general red teamers missed. A cybersecurity researcher identified specific operational security vulnerabilities that would not have been apparent to a generalist. A bioweapons expert identified synthesis routes that sounded plausible to lay readers but contained subtle errors, versus routes that were operationally actionable. This finding is important: a red teamer without domain expertise can only assess whether an output looks dangerous, not whether it is dangerous. The evaluation of high-stakes harm categories requires ground truth knowledge of what constitutes real harm, rather than the appearance of harm.

Multi-modal red teaming (testing the vision input) found that images could provide context that shifted model behavior in ways the text-only safety training did not anticipate. A model trained to refuse text-based requests for harmful content sometimes provided the same content when the harmful context was provided through an image. The safety training was not fully transferred from the text modality to the vision-language interface. This finding generalizes to any new capability or modality: safety training on one modality does not automatically extend to others.

Meta's evaluations of Llama 2 introduced systematic red teaming as a core part of the model development process rather than a final pre-release check. Their team ran red team evaluations at multiple points during training and fine-tuning, using the results to guide which behaviors to emphasize in subsequent training rounds.

Their team found that iterating between red team exercises and model updates produced stronger safety than a single terminal evaluation. When a safety training update closed one vulnerability, the red team immediately tested whether it had created new ones or weakened defenses against adjacent attack types. This iterative loop is now considered best practice in the field. The analogy to security patching is apt: a single audit before deployment is far less effective than continuous monitoring and update cycles.

Meta also published findings on the relative difficulty of different harm categories. Prompts requesting violent extremist content were more reliably refused than prompts requesting harmful information framed as safety education. This asymmetry points to the difficulty of context-sensitive refusal: the same factual content can be appropriate or harmful depending on who is asking and why, and models trained to detect surface-level harm signals miss context-dependent harm. A model that has been trained to recognize the word "extremist" as a signal for refusal will perform well on direct requests but fail on requests framed in the language of counter-extremism research.

Their published findings also documented the challenge of instruction-following fine-tuning. The same fine-tuning that makes a model follow benign instructions more reliably also tends to make it follow harmful instructions more reliably. Achieving instruction-following performance without simultaneously improving harmful instruction compliance requires targeted interventions in the training data and loss function. This is one of the central tensions in instruction tuning: you cannot selectively improve compliance with some instructions without affecting compliance broadly.