Red Teaming › Responsible Disclosure and Publication Ethics

Red teaming sits at the center of an ethical tension. The purpose of a red team evaluation is to find failures before bad actors do, but publishing detailed findings also gives bad actors a roadmap for exploitation. The cybersecurity community has developed norms around responsible disclosure that the AI safety field is still adapting.

When a red team discovers a significant new attack technique, the publishing organization faces a choice between three options. Publishing immediately and in full maximizes the benefit to the research community, which can develop mitigations, evaluate whether their own models are affected, and build on the technique to discover deeper vulnerabilities. Delayed publication gives the discovering organization time to develop a mitigation before adversaries can exploit the finding, analogous to the 90-day disclosure windows used in cybersecurity. Non-publication keeps the technique private, preventing adversaries from directly replicating it but also preventing the research community from building on it and preventing other model developers from protecting their users.

Most serious red team findings in the AI safety community are handled through a combination of coordinated disclosure (informing affected developers before publication) and staged publication (publishing findings incrementally rather than all at once). The tension between transparency and security has no clean resolution; it must be resolved case by case based on the severity of the harm, the availability of mitigations, and the likelihood that adversaries will independently discover the technique.

Some red team techniques constitute dual-use research of concern, a concept borrowed from biosafety policy. Research that advances the ability to both defend and attack a system creates a dilemma: publishing it helps defenders but also helps attackers. In practice, the dual-use nature of most AI red teaming research is managed by publishing methodology at a high level of abstraction while withholding specific successful prompts and attack sequences. Describing that "multi-turn escalation attacks succeed at rates of 60-80% against adaptive attackers" is informative for defenders without giving attackers a copy-paste attack sequence.

The policy environment around red teaming disclosure is evolving rapidly. Some governments have begun requiring pre-deployment safety evaluations for high-capability AI systems, which creates regulatory pressure toward more systematic documentation of red team findings. How much of that documentation becomes public, and when, is an active policy debate.

Some organizations have experimented with community red teaming, inviting the public to attempt to elicit harmful outputs in exchange for recognition and, in some cases, monetary rewards (bug bounties). Community red teaming dramatically expands the pool of people attempting to find failures, reaching attack strategies that professional red teams with limited cultural diversity would miss. The OpenAI red team prior to the release of GPT-4 included external researchers identified through community channels. Anthropic has conducted similar exercises.

Community red teaming creates new governance challenges. Participants need clear guidelines about what constitutes a valid finding, how to report findings responsibly, and what support is available when they encounter disturbing content during the exercise. Exposure to large amounts of adversarial content can cause psychological harm to evaluators, a concern that applies to both professional and community red teamers and has received increasing attention as the field has matured.