Safety Risks › Safety Threat Models

Structured threat modeling provides a principled way to reason about safety risks and to prioritize which interventions are worth the costs they impose on legitimate users. Several frameworks have emerged from the AI safety research community.

One standard framework evaluates potential harms along three dimensions: the probability that harm occurs, the magnitude of the harm if it does occur, and the breadth (number of people affected).

Given a potential harm $h$, the overall risk can be expressed as:

$$\text{Risk}(h) = P(h) \times M(h) \times B(h)$$

where:

• $P(h)$: the probability that harm $h$ occurs, given the realistic distribution of users and requests
• $M(h)$: the magnitude (severity) of the harm per incident, on a scale from minor inconvenience to catastrophic, irreversible damage
• $B(h)$: the breadth, capturing how many people are affected if the harm occurs

The probability $P(h)$ captures how likely a given type of harm is to occur, given the realistic distribution of users and use cases. A request for instructions on building a specific dangerous device is rare; requests for persuasive writing assistance are common. Estimating this probability correctly is difficult because it requires knowing the frequency of certain request types, the fraction of those requests made with harmful intent, and the fraction of those that translate into harm.

The magnitude $M(h)$ captures the severity of the harm per incident. A minor offense that causes embarrassment sits at one end; a harm enabling mass casualties sits at the other. The key insight is that a very low probability harm may still warrant extraordinary precaution if its magnitude is catastrophic and irreversible. Catastrophic harms that cannot be undone deserve asymmetric treatment relative to what a pure expected-value calculation would suggest, because there is no recovery path once the harm occurs.

The breadth $B(h)$ captures how many people are affected. A harm that affects only the requesting user is treated differently from a harm that affects many third parties who had no agency in the interaction. A user who harms only themselves through a model-assisted decision occupies a different moral position than a harm that affects thousands of people who never interacted with the model.

Risk in this framework is roughly proportional to the product of these three factors, which gives a principled basis for asymmetric treatment: low-probability, catastrophic, broad harms (like CBRN uplift) receive more precautionary treatment than higher-probability, lower-magnitude, narrow harms. Even when $P(h)$ is very small, if $M(h)$ is extremely large, the product $P(h) \times M(h)$ still justifies strong precautions. This is why CBRN restrictions are absolute rather than contextual: the potential magnitude is so large that even a very small probability justifies treating this as a hard constraint rather than a calibrated tradeoff.

A related quantity is counterfactual uplift, which measures how much additional harm a model enables above the baseline of what an adversary could achieve without it. Let $H_{\text{without}}$ denote the harm an adversary could achieve using only publicly available resources, and $H_{\text{with}}$ denote the harm they could achieve with model assistance. Then:

$$\text{Uplift} = H_{\text{with}} - H_{\text{without}}$$

where:

• $H_{\text{without}}$: the maximum harm achievable without model assistance, using publicly available information and tools
• $H_{\text{with}}$: the maximum harm achievable with the model as an additional resource

When $\text{Uplift}$ is near zero, the model is providing information that is already freely accessible; refusal imposes costs on legitimate users while providing minimal safety benefit. When $\text{Uplift}$ is high, the model enables harm that would not otherwise occur, justifying stronger restrictions.

This framework helps calibrate when refusal reduces harm versus when it merely inconveniences users asking for broadly available information. The practical challenge is estimating $H_{\text{without}}$ accurately. This requires knowing what information is accessible to a determined adversary, how long it would take them to find and synthesize it, and whether the model reduces that time or improves the quality of the synthesis. These are empirical questions that require ongoing research as both model capabilities and the available information changes.

Counterfactual uplift analysis also shows an important temporal dynamic. As AI capabilities improve, the gap between $H_{\text{with}}$ and $H_{\text{without}}$ may widen for some harm categories. A model that provides minimal uplift today for bioweapons development might provide substantial uplift in five years if its ability to reason over biochemical interactions improves significantly. Safety research must therefore be forward-looking, anticipating how the uplift will change as model capabilities grow.

Some users are more vulnerable to harm than others. A detailed explanation of medication dosages might be entirely appropriate for a licensed pharmacist and dangerous for an individual in a suicidal crisis. A vivid description of violent combat might be appropriate for an adult fiction reader and harmful for a child. Information about eating disorder behaviors might be valuable for a clinical researcher and harmful for someone in recovery.

Vulnerability-aware threat models explicitly account for the population of likely users rather than only the average user. In practice, this means that safety calibration should be more conservative when deployment context suggests vulnerable populations (mental health apps, platforms accessible to minors, crisis services) and can afford more latitude in professional or research contexts.

The challenge is that vulnerability is not always observable. A model cannot reliably determine whether a user is a medical professional or a person in crisis. It cannot distinguish a security researcher from an attacker. It cannot know whether a user asking about dangerous activities is doing so for protective or harmful reasons. Vulnerability-aware threat modeling therefore operates at the population level: given the realistic distribution of users likely to be making this type of request in this type of deployment context, what is the appropriate safety calibration?

This population-level reasoning sometimes produces policies that seem counterintuitive for individual interactions but are justified when the entire distribution is considered. Requiring that users explicitly acknowledge they are asking about a topic for professional reasons, even knowing that many users will falsely claim professional status, can still reduce harm if even a modest fraction of at-risk users are deterred by the friction, and if the false professional claims shift moral responsibility toward the user.

Modern deployed language models operate within a hierarchy of principals: the organization that trains the model, the business that deploys it (the operator), and the end user who interacts with it. Each level of this hierarchy has different interests, different levels of accountability, and different abilities to shape model behavior.

The model developer sets baseline safety constraints that apply everywhere. These are the hard limits that reflect the developer's judgment about what is never acceptable regardless of deployment context: CSAM, CBRN uplift, and a small number of other categorical prohibitions. The deployer (operator) customizes behavior for their specific use case, potentially enabling or restricting certain capabilities relative to the baseline. The user makes requests within the space the operator has defined.

This hierarchy creates important safety properties. An operator who deploys a model in an age-verified adult content platform can enable capabilities that would be inappropriate in a children's educational app. Conversely, an operator who deploys a customer service agent can restrict the model to topics relevant to their business, preventing the model from being used as a general-purpose assistant in ways the operator cannot oversee.

Understanding this hierarchy is essential for safety design because it clarifies who is responsible for what. Model-level safety training handles the categorical baseline. Deployment-level configuration handles context-specific constraints. User-level guardrails handle individual session management. Failures can occur at any level of the hierarchy, and attributing responsibility correctly determines where the fix should be applied.

The hierarchy also creates attack surfaces at the boundaries between levels. An operator system prompt that attempts to override model-level safety constraints will fail if the model is properly trained. A user who attempts to manipulate the model into ignoring operator-level restrictions through prompt injection will fail if the model correctly attributes the injection as a user-level input rather than an operator-level instruction. These boundary attacks are among the most important topics in AI security, and they receive detailed treatment in the chapters on Prompt Injection and Jailbreaking in this part.

One dimension that is underemphasized in simple risk frameworks is the reversibility of harm. A model response that leads to an embarrassing social media post causes harm that fades over time. A model response that provides synthesis instructions for a novel pathogen, once acted upon, causes harm that cannot be undone. The magnitude dimension of the risk framework captures severity, but it does not fully capture the qualitative difference between reversible and irreversible harm.

Irreversibility justifies a stronger form of precautionary reasoning. For reversible harms, expected value calculations are appropriate: weigh the probability against the magnitude, account for counterfactual impact, and choose the policy that maximizes expected outcomes. For irreversible catastrophic harms, something closer to a minimax approach is justified: avoid policies that could lead to catastrophic irreversible outcomes even if the probability is low, because there is no path to recovery if the worst case occurs.

This distinction maps directly onto how safety policies are structured in practice. Most safety decisions are calibrated tradeoffs: how conservative should the model be about medication advice, how explicitly can it discuss historical violence, when should it provide security vulnerability details. A small number of decisions are treated as categorical rules: never under any circumstances provide assistance with CSAM, never under any circumstances provide detailed CBRN synthesis assistance. The categorical rules exist precisely because the combination of irreversibility and catastrophic magnitude removes the calculus from expected-value reasoning.