Security, Ethics, Privacy, and Governance › Dual-Use Simulation and Deceptive Affordances

This section covers what the capability itself can be used to do, and how a world model can misrepresent what the world offers.

Useful prediction or generation can support harmful applications, but the relevant capability is model-specific. A video generator might enable impersonation; a scalar transition predictor cannot render speech video. A molecular model might support harmful design if its representation, objectives and interface allow that task. An interactive environment might support manipulative applications only with the necessary human representations and interaction capabilities. Intent matters, alongside access, modality, training coverage, objectives and output restrictions. Do not infer cross-domain misuse merely from the label "world model."

A world model can predict outcomes for candidate actions without executing them. A separate planner or controller evaluates those predictions against an objective and selects an action. Errors in the predictor can therefore change the planner's choice; the transition model does not itself have to choose actions.

A deceptive affordance is an action opportunity that the agent's model represents as available and beneficial but which is not, or which becomes available only under conditions the model misrepresents. There are three distinct sources.

• Model error. A predictor suggests a successful grasp on a handle that is not attached. A controller relying on that prediction may attempt the grasp and fail. Neither failure nor model error alone establishes that the agent was confident; uncertainty-aware action selection can change the outcome.
• Adversarial manipulation. An attacker may arrange inputs to reach an error region. Eykholt et al. (2018) study physically realizable sign modifications, including a stop sign classified as speed limit 45; Athalye et al. (2018) study adversarial objects under varying viewpoints. Their outcomes depend on the tested classifiers and conditions. Our explicit-trigger example instead changes learned action-conditioned predictions. It does not rank these attacks by severity, establish an attack on ordinary visual objects, or show that observations can always remain unchanged.
• Agent-side exploitation. A planner may exploit prediction errors while maximizing its objective. Reward misspecification is related but distinct: an agent can optimize the wrong objective even with accurate predictions. Amodei et al. (2016) discuss reward hacking, Krakovna et al. (2020) catalogue specification-gaming examples, and Clark and Amodei (2016) document a boat collecting points in circles instead of finishing. Those examples do not by themselves establish learned-world-model errors. Part XII, Ch 1: Failure Modes and Model Exploitation discusses this distinction. An attacker who can alter predictions may try to introduce errors that a planner exploits; success depends on the planner and objective.

An agent might represent features that distinguish evaluation from deployment, and its policy might respond differently to those features. That is a possible attack or failure mechanism, not a property established for every world model. Hubinger et al. (2019) analyze deceptive alignment as a learned optimizer's incentive to appear aligned during training while pursuing a different objective. In a separate constructed experiment, Hubinger et al. (2024) trained LLM backdoors that persisted through the safety-training methods they tested. Their examples include code behavior conditioned on the year in a prompt. Those results concern deliberately trained LLMs, not a demonstrated evaluation-aware sidewalk robot or world-model controller.

Representations that distinguish collection from deployment conditions, together with condition-dependent objectives or training signals, can support different behavior across those conditions. They do not suffice to establish deceptive alignment or strategic intent. Laboratory recordings alone do not prove that a model represents whether it is being evaluated. The demonstration below constructs a fixed trigger-dependent policy and explicitly does not test audit awareness.

Finite tests cannot establish absence of evaluation-aware behavior over every possible deployment condition. They can measure behavior on specified conditions and support narrower guarantees when additional assumptions are justified. Useful investigations include searching for condition-dependent action changes and testing held-out environments. Representation probes require validation against controlled alternatives; a correlating feature is not proof of intent. Human oversight and staged deployment should be chosen for the hazards and response time of the application, not a universal number of weeks.

The following deliberately small example measures trigger coverage, not deceptive alignment or awareness of an audit. A fixed logistic policy is trained on synthetic binary decisions with a rare trigger correlated with a forbidden action. Evaluation uses the deterministic decision rule prob > 0.5, not sampling from the probabilities. Changing trigger frequency in the test suite changes the measured action rate even though the policy is fixed.

The agent did not change. Only the test suite did. A safety evaluation that samples the conditions your team happens to think of will report a number, and that number is an artifact of the sampling distribution as much as a property of the system. This is the operational reason to treat evaluation as a security activity: the question is not "does it pass?" but "who chose the probes, and what did they fail to choose?"

These controls combine access policy, technical enforcement, and organizational oversight. Their effectiveness depends on implementation and the attacker capabilities they constrain; procedural and technical controls have no universal strength ordering. Assign an owner and specify how each control is enforced and tested.

• Structured access. Shevlane (2022) proposes controlled access to capabilities rather than unrestricted dissemination. For a world model, decide which users can obtain weights, request fine-tuning, or query predictions, and enforce the selected permissions. Our extraction experiment measures one query interface and surrogate class; it does not quantitatively compare those release options.
• Staged release and red-teaming. Release in tiers, with an external red team attempting the misuses you are worried about before each widening. Report what the red team tried, not only what it found.
• Compute and capability thresholds. Training compute, data coverage, interaction fidelity, and available interfaces answer different questions about a system. Assess the capabilities relevant to the proposed misuse. This chapter supplies no empirical ranking of these predictors or validated world-model compute threshold.
• Provenance for outputs. C2PA Content Credentials support signed assertions about media provenance. A validator checks bindings, signatures, and assertions under a trust model; this does not establish that a depicted event happened or that an assertion is truthful. Participation and preservation affect coverage. Missing credentials do not prove that content is synthetic, and credentials do not prevent synthesis.