Safety Risks › Misuse Scenarios

Cataloging content types is necessary but insufficient for understanding safety. The same piece of text can be harmful or benign depending on who generates it, why, and what they do with it. Safety threat modeling requires reasoning about actors, capabilities, intent, and deployment context simultaneously.

Nearly all knowledge is dual-use. Chemistry enables medicine and weapons. Biology enables vaccines and bioterrorism. Cybersecurity knowledge enables defense and attack. Cryptography enables privacy and money laundering. Language models inherit the dual-use nature of the information they contain and the skills they can apply.

This is not a new problem. Libraries have long wrestled with making dangerous information available while managing potential misuse. Publishers, search engines, and content platforms have developed their own approaches. What is new with language models is the synthesis capability: the ability to combine information from many sources into a coherent, specific, and actionable response tailored to a particular request. A library contains books about chemistry, but it does not combine them on request into a synthesis plan optimized for a specific user's situation. A language model can.

A useful framing comes from information security: the threat model. A threat model specifies who the adversary is, what resources and capabilities they have, what they want to achieve, and what barriers currently exist between them and their goal. Safety interventions can then be evaluated by how much they raise those barriers for the adversary without excessively impeding legitimate users.

A complete threat model for a deployed language model would specify multiple adversary types, ranging from curious users who stumble into sensitive territory, to low-skill opportunistic attackers, to sophisticated nation-state actors with significant resources. Each adversary type has different capabilities, different goals, and is deterred or enabled by different safety measures. A safety intervention that effectively stops curious exploration may have no effect on a sophisticated adversary who understands the safety system's architecture and can systematically probe for weaknesses.

The concept of uplift measures how much a language model increases an adversary's capability to cause harm compared to what they could achieve without the model. A model that helps a sophisticated state actor slightly improve an already well-developed bioweapons program provides little uplift but still raises serious concerns. A model that enables someone with no chemistry background to produce a dangerous substance from readily available precursors provides substantial uplift.

Uplift is closely related to counterfactual impact: would the harm have occurred anyway without the model's assistance? If the same information is freely available in Wikipedia, refusing to provide it in a different format has minimal counterfactual impact. If the model synthesizes disparate pieces of information in ways that were not previously accessible to someone without deep domain expertise, the counterfactual impact is higher.

These concepts matter because they help calibrate the tradeoffs between safety and usefulness. Restricting access to content with low counterfactual impact imposes real costs on legitimate users while providing minimal safety benefit. Every refusal is a cost paid by the population of users making similar requests, and that cost is only justified when the safety benefit is commensurate. A model that refuses to explain what SQL injection is provides no safety benefit (that information is in countless tutorials, textbooks, and security courses) while failing developers who need to understand vulnerabilities to write more secure code.

The relationship between uplift and availability is not static. As information becomes more or less widely available over time, the counterfactual impact of model-generated content changes. Synthesis routes for some compounds that were once restricted have appeared in public literature and are now widely searchable. Conversely, detailed information about emerging pathogens may become more sensitive as research progresses. Safety calibrations that were appropriate at one point in time may need to be updated as the available information changes.

One underappreciated misuse pattern involves automating tasks that were previously bottlenecked by human effort. A skilled attacker might have been able to write fifty convincing phishing emails per day. With language model assistance, the same attacker might generate fifty thousand. The harm is not that the model enables unprecedented types of attack, but that it removes the scale constraint.

This pattern applies broadly: fraud, scams, harassment campaigns, spam, coordinated inauthentic behavior on social platforms, fake reviews, and astroturfing all benefit from scale. The concern extends beyond individual harms to the degradation of information environments when malicious content floods channels faster than humans can detect or moderate it. Email spam filters learned to handle high-volume, low-sophistication attacks, but language models enable high-volume attacks that are also individually sophisticated, personalized, and harder to detect with pattern-matching approaches.

The economics of content generation shift dramatically when language models enter the picture. Producing a convincing personalized scam message that references the recipient's professional background, recent public activity, and known interests would require significant human research and writing time. A language model can produce such a message in seconds, given basic information about the target. This cost reduction translates directly into higher attack volumes and more sophisticated targeting.

The social cost of high-volume manipulation extends beyond its direct victims. When a substantial fraction of online content is machine-generated for manipulative purposes, trust in online communication degrades for everyone. People become more skeptical of messages from strangers, which imposes costs on legitimate communication, and more susceptible to manipulation by actors who exploit that cynicism in targeted ways.

At the other end of the skill spectrum, sophisticated actors may use language models in combination with other capabilities: automated red teaming to find prompt injection vulnerabilities, model-assisted social engineering, or using one model to generate adversarial inputs that bypass another model's safety filters. This category requires safety strategies that are robust to adversaries who understand how the safety systems work and are actively trying to circumvent them.

The adversarial dynamics here are related to the arms race dynamics we see in other security domains. Spam filters and spammers have been in continuous competition for decades, with each improvement in detection prompting corresponding improvements in evasion. AI safety systems face a similar dynamic: as safety training improves, adversaries who care enough will find new approaches to bypass it. This is not an argument against investing in safety, but it is an argument for treating safety as an ongoing process rather than a solved problem.

Sophisticated adversarial attacks on language models often exploit the gap between a model's training distribution and adversarial inputs. A model trained to refuse requests for harmful content will be trained on specific patterns of harmful requests. Adversaries can rephrase those requests in ways that preserve the harmful intent while avoiding the specific patterns the model was trained to recognize. This is the mechanism behind many jailbreaking techniques, which we will cover in detail in the chapter on Jailbreaking in this part.

Not all harmful use is intentional. A developer might deploy a general-purpose language model in a mental health application without appropriate safeguards. A company might use model-generated content in high-stakes decisions without human oversight. A user might follow model-generated medical advice that is subtly wrong for their individual circumstances.

These accidental or negligent misuses are often harder to address through model-level safety training than intentional malicious use. A model cannot easily distinguish between a developer who has deliberately disabled safeguards and one who simply failed to configure appropriate restrictions. Model-level safety provides a baseline, but deployment-level safeguards, product design, and organizational practices are essential for preventing accidental harm.

Automation bias is a particularly important category of accidental misuse. When a model produces confident-sounding output, users tend to trust it even when they would not trust a human expert stating the same thing with equivalent uncertainty. This asymmetric trust leads to over-reliance in high-stakes domains where language models are not reliable enough to be trusted without verification: medical diagnosis, legal advice, financial planning, and engineering safety calculations. The harm arises not from the model doing something it should not, but from users trusting it more than is warranted.

The deployment gap between a model's intended use and its actual use is also a source of accidental misuse. A model designed for creative writing assistance may end up being used for psychological counseling by users who find it accessible and non-judgmental. A model designed for code generation may end up answering questions about medication when a user asks a question it can plausibly answer. The model does not know what it was designed for; it responds to the input it receives. Safety design must account for this gap.