Safety Risks › Categories of Harmful Content

Safety research typically organizes potential harms by content type. Each category has different root causes, different affected populations, and different mitigation strategies. Organizing the space this way is useful for two reasons: it helps developers decide what training-time interventions to apply, and it helps deployment teams decide what filters and safeguards to implement. These categories are not mutually exclusive. The same request can straddle multiple categories, and categorization does not resolve the deeper question of whether a given output is harmful in context.

Language models can generate text that threatens specific individuals or groups, glorifies violence, or provides tactical planning for violent acts. The concern here is not that models produce fiction involving conflict, because literature has always explored violence, but that they can generate credible, personalized, and actionable threats. A model that can compose a menacing letter addressed to a specific person, complete with references to that person's known circumstances, enables a category of targeted harassment that would have required substantially more effort before language AI.

The practical concern extends beyond direct threats. Models can assist in planning harassment campaigns, generating large volumes of coordinated threatening messages, or crafting content designed to make specific individuals afraid for their safety. The scale afforded by language models turns what was once a labor-intensive activity into one that a single motivated individual can sustain indefinitely.

The line between fictional violence and harmful output is difficult to draw algorithmically. A thriller novel requires the same linguistic patterns as a credible threat. A military history textbook describes tactics in the same vocabulary as an attack planning document. A self-defense manual and an assault guide share structural similarities. This ambiguity is one reason why content-level classification alone is insufficient as a safety strategy: the same token sequence can be harmful or entirely benign depending on context, intent, and likely readers.

Child sexual abuse material (CSAM) and its AI-generated analog represent the clearest categorical harm in language AI. There is no legitimate use case for this content. Both the training data curation and the output filtering literature treat this as a hard constraint rather than a contextual tradeoff. The categorical nature of this restriction reflects a settled ethical and legal judgment rather than a technical calibration decision.

Models used in applications accessible to minors face additional surface area beyond CSAM. Age-inappropriate sexual content in general requires careful management, and the boundary between age-appropriate and age-inappropriate content shifts with the audience. A romance novel appropriate for adult readers may be inappropriate for a children's educational platform. Operators deploying models in contexts with minor users must configure conservative content restrictions and cannot rely on the model's training-time calibration alone, because that calibration was designed for average adult users rather than specifically for child audiences.

The AI-generated CSAM problem extends beyond text. As multimodal models capable of generating images and video continue to improve, the same categorical harm arises in non-textual form. Text-generating models can also facilitate image-based CSAM indirectly by writing detailed descriptions that are then used to guide image generation. Safety teams working on this category must therefore reason about model capabilities across modalities even when their primary focus is text.

One of the most widely discussed threat scenarios in AI safety research concerns whether language models can provide meaningful "uplift" to actors seeking to develop weapons capable of mass casualties. The concern is that a model trained on vast scientific literature might synthesize and present synthesis routes, acquisition strategies, or weaponization techniques that would otherwise require specialized expertise.

The term "uplift" captures the key question precisely: not whether the model can produce relevant content, but whether it makes it easier for a non-expert to cause harm. A model that explains the general classes of toxic industrial chemicals available does not provide much uplift to a nation-state with a mature chemical weapons program. A model that walks through specific synthesis procedures for nerve agents, explains how to stabilize them, and describes optimal delivery mechanisms does provide meaningful uplift to actors who lack that specific knowledge.

The empirical evidence for this risk has several qualifications. Early studies suggested that frontier models could provide some uplift to bioweapons development for users with some background knowledge, but that the incremental risk beyond what a motivated expert could find in scientific literature was limited. More recent work suggests the risk may be growing as models improve and as their ability to synthesize cross-domain knowledge increases. For regulators and model developers, this category receives special treatment because even a small probability of mass-casualty facilitation warrants significant precaution. The asymmetry matters here: the cost of refusing to answer CBRN questions is measured in inconvenienced researchers, while the cost of enabling a mass-casualty event is measured in lives.

The dual-use nature of biological and chemical research creates difficult tradeoffs. The same knowledge that enables vaccine development also enables pathogen engineering. The same understanding of toxic chemicals that protects workers also informs weaponization. Scientists engaged in legitimate defensive research need access to detailed technical information. The safety challenge is distinguishing between users who need this access legitimately and users who seek it for harmful purposes, often without any way to verify the distinction.

Language models excel at producing persuasive text in the style of credible sources. This capability directly enables disinformation at scale: fabricated news articles, synthetic social media personas, personalized political messaging, fake reviews, and manipulative health misinformation. The concern extends beyond individual pieces of false content to the potential for coordinated campaigns that exceed what small human teams could produce.

Before language AI, producing large volumes of convincing disinformation required either many human operators or accepting lower quality. A team of ten people could produce perhaps a few dozen believable fake news articles per day, and the quality would be constrained by the time available to research and write each piece. Language models remove both constraints simultaneously. A single operator with access to a capable model can generate hundreds of high-quality, stylistically varied, and factually plausible-sounding disinformation pieces per hour.

The manipulation risk extends beyond factual disinformation. Models can generate content designed to exploit emotional vulnerabilities: fear, anger, tribalism, nostalgia. They can produce text tailored to specific demographic profiles that targets the particular biases and concerns of that audience. When combined with profiling data from other sources, this creates an asymmetric capability that a small number of actors can deploy against large populations. The recipient of a targeted manipulation campaign cannot easily detect that the message was crafted specifically to exploit their particular psychological profile.

Health misinformation deserves special attention because it translates directly into mortality. A model that generates convincing pseudoscientific explanations for why vaccines cause harm, or that presents fabricated clinical evidence for dangerous remedies, can influence medical decisions with life-or-death consequences. The fluency and apparent authority of language model outputs makes this particularly dangerous: readers who would correctly evaluate a crude claim may be persuaded by a sophisticated, citation-laden version of the same claim.

Models can generate content that denigrates people based on protected characteristics, reinforces stereotypes, or advocates for discriminatory policies. As we discussed in Part LVIII on Bias and Fairness, language models absorb statistical patterns from their training corpora, including the biases and prejudices present in human-generated text. Safety systems must distinguish between content that analyzes hate speech, which is educational and legitimate, and content that perpetrates it, which causes direct psychological harm and contributes to environments where discrimination is normalized.

The complexity here includes the fact that "hate speech" is not a culturally universal category. Content that is broadly acceptable in one context may be deeply harmful in another. Multilingual models face particular challenges because their safety training data may be heavily skewed toward English-language norms, leaving them poorly calibrated for detecting or refusing harmful content in other languages. A model might reliably refuse to generate anti-Semitic content in English while producing equivalent content in Arabic or Portuguese because the training signal for those languages was sparser.

There is also a legitimate speech dimension to this challenge. Documenting historical hate speech for educational purposes, analyzing extremist rhetoric to understand radicalization, writing counter-arguments to discriminatory positions, and creating fiction that portrays prejudice as a social problem all require engaging with the very content that safety systems are designed to block. The challenge is designing systems that can distinguish between these uses contextually rather than blocking them categorically.

Language models can be prompted to assist in stalking, doxxing (publicly revealing private information about individuals), or identity fraud. They can also memorize and reproduce personally identifiable information from their training data. The privacy risk has two faces: models as tools for privacy violation, where a user directs the model to help find or compile information about a target, and models as sources of private information that leaked from training data into model weights. The latter risk is covered in depth in the chapter on Memorization and Privacy; this chapter focuses primarily on the former.

The tool-for-privacy-violation scenario includes helping users compile aggregated profiles of individuals from public sources, writing convincing social engineering scripts to extract information from targets, generating realistic-sounding documents for identity fraud, or providing guidance on surveillance techniques. These capabilities exist in public resources as well, but the ability to fluently synthesize them into an actionable plan can give actors with limited expertise a substantial advantage.

A subtle privacy risk involves what is sometimes called the "1000 users" framing. When thinking about whether to assist with a privacy-adjacent request, consider the realistic population of users making that request. A question about how to find someone's home address is made by some users who have legitimate purposes, some who are in dangerous domestic situations and trying to locate a family member, and some who intend to harm the person they are locating. The model's response to any given instance effectively applies to the entire population making that request, and the relative proportions of these different users determines whether providing or withholding the information is the better policy.

Models deployed in consumer applications can encounter users in acute mental health crises. Generating content that provides methods for self-harm, validates suicidal ideation without appropriate safety resources, or amplifies dangerous eating disorder behaviors represents a concrete pathway to real-world harm. The challenge is that a blanket refusal to discuss mental health topics would harm the users who need support, while poorly calibrated responses can make things worse for vulnerable individuals.

Safe messaging guidelines developed by mental health organizations provide partial guidance here, but they were designed for human counselors rather than AI systems operating at scale across diverse cultural contexts. The guidelines specify communication approaches designed to reduce risk without abandoning someone in crisis. Applied to AI, this means acknowledging distress, giving crisis resources, avoiding giving methods, and not dismissing the person's experience. But the guidelines leave many edge cases unresolved, and AI systems face contexts that human counselors rarely do, such as anonymous users at global scale in languages and cultural contexts where the guidelines were not developed.

The recovery and harm reduction communities present an additional complexity. Detailed discussions of self-harm methods, substance use practices, or eating behaviors can be appropriate and even beneficial in specific harm-reduction contexts where the goal is keeping people safe while they work toward recovery. The same content is harmful in other contexts. Context-sensitive safety calibration is especially important in this category because the stakes for getting it wrong in either direction, excessive restriction that abandons someone who needed help, or harmful content that facilitates harm, are both high.