Security, Ethics, Privacy, and Governance

Consider a hypothetical household robot. Its planner uses a learned transition model to predict the consequences of candidate actions rather than testing every action physically. Suppose we train that model on volunteered recordings of homes.

Three fictional incidents below illustrate this. They are threat scenarios, not measurements from a deployed robot, and each raises questions we would need to investigate. None follows automatically from good video quality or low prediction error.

Suppose three things go wrong, months later, in three different ways.

A data provider we contracted with was acquired mid-project, and its annotation contractor quietly relabeled a small set of clips. The relabelled clips were not random noise; they were chosen so that the model misestimates how far the arm travels for a given command. Prediction accuracy barely moves. The robot, however, starts oscillating whenever it reaches for a shelf.

A partner lab with authorized prediction-API access collects responses and fits a surrogate. Depending on its query budget, coverage, and model class, it might approximate some of the original model's behavior. That would not establish recovery of the original weights, the training records, or the full closed-loop agent.

Your marketing team generates footage that resembles a private kitchen from the recordings. We must investigate whether this is memorized training content, a coincidental resemblance, or an ordinary combination of learned features before asserting a privacy breach. Recognizable personal content would require a release decision separate from the model's prediction metrics.

These three stories have different technical mechanisms, but they share a structure. In each one, a world model is not merely a predictor that can be wrong. It is a trusted oracle whose outputs other systems act on, whose internals may retain information about other people, and whose outputs can be produced at industrial scale. The previous chapters in this part examined failures that arise from the model's own inductive biases (Failure Modes and Model Exploitation) and from the difficulty of staying calibrated and safe under distribution shift (Robustness, Calibration, and Safe Control). Both chapters concerned failures that happen because the world, or the model, behaves in a way nobody intended.

This chapter covers deliberate attacks, exposure of information learned from data, and harmful uses of simulated outcomes. It has four sections: poisoning and extraction, privacy, dual-use simulation, and governance. Some threat models allow an attacker to observe and adapt to deployed defenses; others restrict that knowledge. Evaluation needs to state which case it covers. The implementations demonstrate specific mechanisms or limitations, with computation separated from displayed outputs and presentation-only plots.

It helps to map the standard security vocabulary onto world models explicitly.

• Integrity concerns unauthorized modification or destruction of data and system behavior. Poisoning and backdoors can compromise model integrity; an inaccurate but untampered predictor is not, merely for that reason, a security compromise.
• Confidentiality concerns disclosure outside authorized destinations. Functionality extraction can expose a protected predictive function. Successful membership inference or reconstruction can expose information about training data, but simply attempting those attacks does not establish disclosure.
• Availability is the property that the system keeps working. Inference-time flooding and resource exhaustion against a large generative world model are availability attacks. We return to resource budgets in Part XII, Ch 4: Training and Inference Systems.
• Authenticity concerns whether an asserted source or origin is genuine, a longstanding security concern. Generated media makes origin claims harder to assess; a plausible recording is not itself proof that an event occurred.

These properties separate prediction reliability, protection against unauthorized changes, information exposure, service continuity, and trustworthy origin claims. This chapter uses explicit threat models and synthetic probes, not a certification of real-world security. Defenses can have formal guarantees under stated assumptions, but finite empirical testing cannot rule out every future adaptive attack.

The synthetic experiments below are mechanism probes, not reproductions of attacks against a real system. Computation, displayed outputs, and presentation-only plots are separate, and random generators are seeded. The plant state is directly observed here: $o_t=s_t$, so no belief-state estimator is implemented. A predictor $\hat f_\theta(s_t,a_t)$ estimates the next state; closed-loop tests use horizon $H$. This fully observed scalar plant is not a video generator or evidence about deployed robot privacy. A probe may fail to demonstrate the anticipated effect. We retain such negative results rather than adjusting seeds or claiming leakage that the outputs do not show.

We begin with one environment. We will use a one-dimensional integrator with a bounded control input: a crude model of a velocity-controlled joint or a robot that moves at a commanded speed. This small plant lets us study how an incorrect control gain changes closed-loop behavior. A linear plant is chosen deliberately: not because real world models are linear, but because parameter corruption and local closed-loop instability can already occur in a linear system. Closed-form analysis lets us derive attack conditions under explicit feature, label and row-budget constraints. The demonstrations below do not solve every possible attack against the learner.

With this fitted model, the first command is near the control limit but does not quite saturate. The state then stays near the setpoint under the specified disturbance sequence. This finite run is the reference behavior, not a claim that every disturbance or initial condition is safe.

This section covers attacks against training data and inference interfaces. Poisoning changes what the learner fits. Query-based extraction tries to reproduce some model functionality; it need not recover the original weights. The training pipeline, annotation process and prediction API belong in the threat model alongside the model itself.

A world model is assembled from many stages, and each is a place where an adversary can insert influence:

• Collection. Cameras, teleoperated demonstrations, scraped video, and logs from deployed fleets. Whoever controls a sensor or an upload path controls part of the data distribution.
• Annotation. Labels, captions, segmentation masks, reward estimates, and filtered "usable" subsets. If a contractor supplies annotations, include its permissions and checks in the threat model. Outsourcing does not by itself establish attacker access or low attack cost.
• Curation and filtering. Deduplication, quality scoring and safety filtering may use deterministic rules, learned models or both. An attacker who can shape submitted content may try to pass these checks; success depends on the particular filter and permissions.
• Pretraining corpora. Public accessibility does not imply attacker write access. Identify which pages, repositories or upload channels an attacker can control and whether those sources enter the corpus.
• Fine-tuning and online adaptation. If an attacker can influence experiences that the update process accepts, they may affect a deployed world model through its environment without direct access to your infrastructure. Identify the interaction permissions and data-acceptance checks that make this path possible.

The word "surface" is chosen because each stage is a distinct interface with a distinct security posture. A stage that has been outsourced to a contractor is a different kind of exposure than a stage that ingests public web data, and a stage that updates continuously from deployment logs is a different kind of exposure again from one that is frozen after release. Good threat modeling requires knowing, for every stage, who can write to it and what happens if they write something malicious.

Let the learner be a function that maps a dataset $D$ to parameters $\hat\theta(D)$. The attacker chooses a perturbation of the training data, constrained to a budget $\mathcal{B}$ (a fraction of rows, a bound on per-row modification, or control over a subset of sources), to maximize a damage objective $\mathcal{L}$:

$$\begin{aligned} \max_{\tilde{D} \in \mathcal{B}} \; & \mathcal{L}\big(\hat\theta(\tilde{D})\big) \\ \text{subject to} \quad & \hat\theta(\tilde{D}) \in \arg\min_{\theta} \left[\sum_{(x,y) \in \tilde{D}} \ell\big(f_\theta(x), y\big) + R(\theta)\right] \end{aligned}$$

where:

• $D$: the clean training dataset the learner would otherwise see
• $\tilde{D}$: the attacker-chosen poisoned dataset, drawn from the allowed budget set $\mathcal{B}$
• $\hat\theta(D)$: the fitted parameters the learner returns for dataset $D$
• $\mathcal{L}$: the attacker's damage objective, a scalar measuring how bad the fitted model is for the defender
• $\mathcal{B}$: the set of datasets the attacker is allowed to submit (bounded by row fraction, per-row magnitude, or controlled sources)
• $\ell\big(f_\theta(x), y\big)$: the per-example training loss of predictor $f_\theta$ on input $x$ with target $y$
• $R(\theta)$: the learner's regularizer; the ridge example uses $R(\theta)=\lambda\|\theta\|_2^2$ with squared-error loss
• The inner $\arg\min$: the learner refits from scratch on the poisoned data, with its optimizer or tie-breaking rule fixed if minimizers are not unique; this makes the outer maximization a bi-level (or bilevel) problem

This is a bi-level problem: the outer maximization must account for the fact that the learner refits after seeing the poison. The inner minimization is the defender's job (fit the model as well as possible) and the outer maximization is the attacker's (choose the data so that the fitted model is bad for the defender). Attackers use a few standard approximations.

• Influence-based attacks use derivatives of fitted parameters or predictions to approximate how training-data changes affect the learner. For ridge with fixed design $X$ and fixed positive $\lambda$, the label derivative is exactly $(X^\top X+\lambda I)^{-1}X^\top$; because the fit is linear in the labels, it also gives the exact finite label update. Changes to features or other learners generally require different derivatives or local approximations (Jagielski et al., 2018, "Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning"). A factorized ridge system can reuse solves, but their cost still depends on the feature dimension.
• Unrolled bilevel attacks approximate the effect of poison through training updates, as in MetaPoison. Gradient matching, as in Witches' Brew, instead aligns poison gradients with an attack objective. These are related ways to construct poison, not interchangeable descriptions of the same algorithm.
• Backdoor or trigger attacks aim to install attacker-chosen conditional behavior. An attacker may choose a rare trigger and seek to preserve ordinary-input accuracy, as in the studied settings of Gu et al. (2017, "BadNets") and Chen et al. (2017). Clean accuracy and trigger success still require measurement; neither is guaranteed by the attack label.

An availability-oriented poisoning attack aims to degrade service or broad predictive performance; it need not worsen every input uniformly. A targeted attack aims at a particular outcome or input region. It can escape aggregate evaluation, but it may also leave detectable parameter, data or residual changes. Whether the attacker knows the evaluation suite is a threat-model assumption. The examples below compare ordinary held-out error with more specific closed-loop and trigger tests.

Let's make the bi-level problem concrete. For ridge regression the fitted parameters are

$$\hat\theta = \big(X^\top X + \lambda I\big)^{-1} X^\top y$$

where:

• $X \in \mathbb{R}^{n \times d}$: the design matrix, whose $i$-th row stacks the features of the $i$-th training input (here $[s_i, a_i]$)
• $y \in \mathbb{R}^{n}$: the vector of recorded next-states $s_{i+1}$
• $\lambda>0$: the ridge regularization strength, which shrinks fitted parameters and makes $X^\top X+\lambda I$ positive definite; with $\lambda=0$, the displayed inverse instead requires full column rank
• $I$: the $d \times d$ identity matrix
• $X^\top X$: the Gram matrix, the sufficient statistic that summarizes how strongly the data excites each parameter direction

The form of this solution is the key to the whole analysis. The fitted parameters depend on the data only through two summary quantities, the Gram matrix $X^\top X$ and the cross-moment vector $X^\top y$. Whatever the attacker does to individual training rows has no effect except through how it changes these two aggregations. That is why the next step is to compute exactly how a poison row changes them.

Let $\phi_p \in \mathbb{R}^d$ be the feature vector shared by the poison rows and $y_p$ the target the attacker records for each. Appending $m$ such identical rows adds $m\phi_p\phi_p^\top$ to the Gram matrix and $m\phi_p y_p$ to the cross-moment vector. If the attacker wants a specific parameter vector $\theta^\star$, the fitted parameters must satisfy the ridge normal equations:

$$\big(X^\top X + m\phi_p\phi_p^\top + \lambda I\big)\,\theta^\star = X^\top y + m\phi_p y_p$$

where:

• $X^\top X + m\phi_p\phi_p^\top$: the poisoned Gram matrix, the clean sufficient statistic plus the rank-one contribution of the poison rows
• $X^\top y + m\phi_p y_p$: the poisoned cross-moment vector, the clean statistic plus the poison contribution
• $\theta^\star$: the parameter vector the attacker wants the refit learner to return

Rearranging to isolate the poison contribution gives the required poison condition:

$$\begin{aligned} m\,\phi_p\big(y_p - \phi_p^\top \theta^\star\big) &= r \\ r &= (X^\top X + \lambda I)\theta^\star - X^\top y \end{aligned}$$

where:

• $m$: the number of identical poison rows the attacker appends
• $\phi_p \in \mathbb{R}^d$: the feature vector shared by all $m$ poison rows (for the dynamics model, the pair $(s_p, a_p)$)
• $y_p$: the target the attacker records for each poison row (the next-state $s_p'$)
• $\theta^\star$: the parameter vector the attacker wants the learner to return
• $r$: the residual of the clean fit against $\theta^\star$, fully determined by the clean data and the attacker's chosen target
• The left-hand side is the influence of the appended rows; permitted row counts, feature vectors and labels constrain whether it can equal $r$

The residual $r$ is fixed by the clean data and the target parameter vector. With identical appended rows, the required vector must lie along $\phi_p$; arbitrary feature bounds or label constraints can make a desired target infeasible. Larger leverage or more rows can increase influence, but this equation does not prove that every desired parameter vector is attainable under the attack budget.

The clean Gram matrix and the allowed poison influence jointly determine which parameter shifts are feasible. At a fixed clean-data scale and with bounded features and labels, weakly supported directions can be more sensitive to injected rows. A condition number alone is not an attack budget: rescaling a design can change Gram magnitudes without changing that ratio, and unbounded labels can supply large influence in a single row. Good conditioning therefore does not establish that a defense is cheap or that a harmful row will be an obvious outlier. Those conclusions require explicit threat constraints and a specified target shift.

Let's run the experiment. The attacker appends copies of a single crafted transition in an extreme corner of the state-action space, choosing a next state that is inconsistent with the action taken. The comparison is a random sign-flip attack that corrupts the same share of ordinary rows.

The constructed corner attack appends repeated transitions at an extreme feature pair. Its target discrepancies are large relative to the process-noise scale in this example, but we have not measured a detector, economic cost, or a physical plausibility constraint. Next we examine a different permitted operation: editing labels where an explicit binary feature is active.

Suppose a binary feature $z$ is rarely active and has its own column in a full-rank linear design. Under unregularized least squares, changing the labels to $y-\delta z$ shifts that column's coefficient by exactly $-\delta$. Ridge can attenuate the shift and change other coefficients too. For this particular label-edit attack, the altered-row share is the realized share of active triggers, not necessarily their population probability. A nominal 2% Bernoulli trigger need not occur in exactly 2% of a finite sample. Very rare triggers can be absent altogether. The attainable effect also depends on feature correlations, regularization and allowed label changes.

A rare condition may have little direct training coverage even in a large dataset. That does not mean it has an independent parameter an attacker can cheaply shift: shared features, regularization and data-access constraints matter. Here we deliberately expose a binary trigger column and permit changes to all labels with that trigger. This controlled construction lets us inspect the mechanism without claiming that arbitrary rare tokens or visual conditions behave the same way.

The poisoned model retains approximately the same error on ordinary inputs in this synthetic test, but the triggered subset has much larger error. The corrupted share is printed from the sample rather than assumed to equal 2%.

With trigger $z=1$ and an unsaturated steady state, the controller's fixed point satisfies $s=(s^\star-\hat c)/\hat a$. Relative to $s^\star$, its bias is $(s^\star-\hat c)/\hat a-s^\star$, approximately $-\hat c$ when $\hat a$ is near one. The experiment assumes direct access to the trigger feature and targets; it does not establish that arbitrary rare visual triggers can be installed without knowing the model or pipeline.

Different defenses constrain different parts of the attack. Their assumptions need to be stated together.

• Provenance and segmentation. Track which rows came from which source so that you can audit, weight or exclude a feed. Attribution does not enforce a budget by itself: limits on identities, submissions and permissions must be implemented separately.
• Robust aggregation across sources. Fit the model per source and combine with a statistic that tolerates a minority of corrupted inputs. We demonstrate this below.
• Influence-function audits. Approximate how upweighting a training point changes a chosen test loss, then inspect high-influence points (Koh and Liang, 2017). The choice of test objective matters. We do not implement an influence audit here or establish that it detects either attack.
• Robust losses and trimmed objectives. Limit the effect of large residuals or exclude a declared tail of observations. Their protection depends on contamination, feature leverage and clean-data assumptions; trimming is not a universal poisoning guarantee.
• Trigger discovery. Search for inputs on which behavior changes unexpectedly. Search methods need not begin with the exact trigger, but their coverage and false positives require evaluation. Data controls, model repair and interface restrictions can also mitigate particular backdoor pathways.

Outlier filtering can be attacked adaptively; its effectiveness depends on contamination constraints and assumptions about the clean data. Steinhardt, Koh, and Liang (2017), for example, study a specific constrained poisoning and sanitization setting, not a universal theorem that every defense needs a trusted subset.

Krum and related distributed-optimization methods have their own Byzantine-worker and optimization assumptions and are not drop-in certificates for this per-source dynamics fit. Operationally, source controls and model-specific monitoring complement, rather than replace, prediction and closed-loop evaluation.

Here is a controlled aggregation comparison. Forty sources initially provide one hundred clean transitions each. Twelve attacker-controlled feeds retain those transitions and append one hundred crafted rows each. Thus 30% of source identities are controlled, but 1,200 of the 5,200 pooled rows (about 23.08%) are injected. Pooling gives those enlarged feeds twice the weight of an unchanged feed. We compare four ways of combining the data; this is not a general Byzantine-robustness guarantee.

Pooling gives the corrupted sources extra weight because they append rows. In this sample only the pooled estimate falls below the simplified gain threshold. Equal-weight source averaging shifts the gain but remains above it, while the median and heavily trimmed mean are closer to the true gain. These are coordinate-wise estimates in a scalar example, not a certified defense for neural dynamics models. Median guarantees require assumptions about the number and dispersion of honest sources; an attacker who controls source identities can defeat the source-count premise.

Now we turn to the other end of the pipeline. Suppose the model is good and our data is clean. Can an adversary simply take it?

Prediction interfaces can enable functionality extraction. Tramèr et al. (2016) demonstrated high-fidelity extraction for several model classes and API settings; that is not a universal query bound for world models. Returned scalar values can constrain a surrogate, but label outputs are not universally one bit and rollout outputs can be highly dependent. Recovering behavior on a query distribution is distinct from recovering weights, private training records, or planning performance.

Extraction raises several concerns for a world-model deployment:

1. A usable surrogate can reproduce some predictions that an agent relies on. It does not automatically supply the agent's objective, policy, reward model or search procedure.
2. Access can change which protections the owner can enforce. Exact copying alone need not change learned behavior. Qi et al. (2023) show that certain fine-tuning procedures can compromise safety alignment in studied language models; this is not a result about every copied world model.
3. A local surrogate can support offline searches without the original API's rate limits or fees. Those searches still consume compute, and their usefulness depends on surrogate fidelity, available derivatives and the attacker's access to the deployment environment.
4. Authorization, licensing, and contractual conditions need assessment under the applicable law. This technical probe supplies no general legal conclusion about extraction. Contractual restrictions can be administrative controls or deterrents without technically preventing copying.

Include extraction in the deployment's threat model when a useful copied function would expose protected assets. A surrogate may move some capability beyond the original provider's controls. The costs of prevention depend on the chosen access, monitoring, and output restrictions; they need not all reduce prediction accuracy.

Let's measure approximate functionality copying on a synthetic two-dimensional domain. A known noiseless linear form with $d$ unknown coefficients can be recovered from $d$ linearly independent scalar equations, but that result depends on its form, output precision, and rank. Here the victim uses 96 random Fourier features plus an intercept. The attacker uses a different basis and may face an approximation floor even with unlimited queries.

The output reports a substantial reduction in error as query coverage increases, but the surrogate remains less accurate against truth than the victim. The evaluation-only least-squares fit gives the best degree-four approximation on this grid; its residual shows that more queries alone cannot remove the chosen basis mismatch. This seeded experiment provides no general query budget for copying world models and no evidence of training-data recovery. Another surrogate or domain could behave differently.

• Budget calls and returned information. Cap query counts, batch sizes, rollout lengths and output precision. A hundred returned states can expose more than one scalar, but redundant or highly dependent outputs need not provide a hundred independent constraints. Evaluate the actual response format and query distribution.
• Evaluate reduced output resolution. Quantization or hidden random noise can limit fidelity or increase the queries needed for a particular extraction target, but neither has that effect universally. Integer quantization leaves a known constant victim in the class $\theta\in\{0,1\}$ exactly identifiable with one query. Hidden noise bounded by $0.1$ does too, since the possible responses for zero and one are disjoint. For continuous parameters, quantization can instead make some values indistinguishable, while repeated queries can average independent noise. Measure both extraction error and legitimate utility.
• Restrict what the interface can express. Returning a planned action instead of a transition limits directly exposed outputs, but does not guarantee hidden dynamics. In our unclipped controller, two queries at known states satisfy $\hat a s_i+(\hat b+10^{-8})u_i=s^\star$; a nonsingular two-query system recovers both coefficients from actions alone. Decide whether users need uncertainty information, and test what the restricted endpoint still reveals.
• Watermark and monitor. If using a behavioral watermark, validate its detectability, false-positive rate, and robustness to surrogate training or adaptation. Monitor suspicious query patterns, recognizing that extraction need not use a regular grid and that legitimate testing may do so.
• Policy and contracts. Terms of service and licensing may supply remedies or deterrence under applicable law and enforceable agreements. Provenance records may support attribution. None alone establishes technical prevention or a legal violation.

Output restrictions may trade legitimate utility against exposure. Monitoring, access policies and contracts have different costs and need not change prediction accuracy. Define the functionality an attacker is trying to recover, its error tolerance, available queries and acceptable residual exposure; this chapter does not establish a universal extraction-prevention result.

This section covers the information a world model absorbs about the people in its training data and what happens when that information can be recovered.

A recorded-environment corpus may contain continuous, context-rich personal information. Curated object datasets can also include identifying people or spaces; this is a difference to assess in the actual corpus, not a clean dividing line between model families. Possible examples include:

• Home recordings that show private interiors.
• Driving recordings that retain recognizable pedestrians or license plates.
• Clinical recordings that retain patient bodies or movements.
• Social recordings that retain conversations, faces, or contextual cues useful for identification.

Separate information present in the corpus from information retained or exposed by the model. Synthetic, task-specific, or privacy-processed datasets may differ. Continuous motion and contextual relationships can warrant additional privacy assessment where they concern identifiable people.

Consider three possible exposure channels. Their applicability depends on whether the corpus contains personal data, what the model returns, and what the attacker can observe.

• Dataset leakage. The corpus itself escapes: a breach, an unsecured bucket, a scraped copy. This exposes stored recordings directly rather than demonstrating leakage through model outputs. The amount and sensitivity of exposed personal information can increase potential harm. Assess the consequences for affected people in the incident's context.
• Model-level leakage. Membership inference tests whether a candidate record was in the training set (Shokri et al., 2017; Carlini et al., 2022). Attribute inference targets hidden properties. Certain diffusion-model attacks have recovered near-identical training images, including identifiable people (Carlini et al., 2023, "Extracting Training Data from Diffusion Models"; Somepalli et al., 2023). That is evidence of leakage in those studied systems, not proof that every generative world model regenerates personal records. Realistic or confident outputs alone do not establish memorization.
• Interaction leakage. Observable actions or response timing can supply candidate signals for a membership attack. Faster navigation in a familiar hallway, for example, could also reflect geometry, hardware or ordinary generalization. Establish leakage with a defined candidate population, controlled alternatives and measured false-positive rates; familiarity alone is not a membership oracle.

Differential privacy supplies a formal distributional guarantee under a specified adjacency relation. A randomized mechanism $M$ is $(\varepsilon, \delta)$-differentially private if, for any two neighboring datasets $D$ and $D'$, and for every measurable output set $S$,

$$\Pr[M(D) \in S] \le e^{\varepsilon} \, \Pr[M(D') \in S] + \delta$$

where:

• $\varepsilon$: the privacy budget; with $\delta$ and the adjacency relation fixed, smaller values tighten the multiplicative bound on how one neighboring-data change affects output probabilities
• $\delta$: an additive slack term in the inequality, not generally the probability of a single identifiable failure event
• $M$: the randomized mechanism; the probabilities range over its internal randomness
• $D,D'$: neighboring datasets under the chosen record, trajectory, or person-level adjacency
• $S$: any measurable set of possible outputs
• The bound controls the change in output probabilities between neighbors; its interpretation depends on $\varepsilon$, $\delta$, and the adjacency definition

Differentially private stochastic gradient descent (Abadi et al., 2016) is the workhorse for training.

Three structural features of world models make DP harder to apply than the textbook case suggests.

First, one person can contribute many records. Record-level adjacency and person-level adjacency are different protection units. For pure record-level $(\varepsilon,0)$-DP, a group of $k$ records has the bound $(k\varepsilon,0)$ (Dwork and Roth, Theorem 2.2). With approximate DP, the additive term does not in general merely scale as $k\delta$: iterating the inequality gives $\delta\sum_{i=0}^{k-1}e^{i\varepsilon}$, capped at one. Group privacy is distinct from composition of repeated mechanisms. A person-level guarantee can instead be designed with person-level contribution bounds and accounting; temporal correlation does not invalidate the definition itself.

For the elementary pure-DP group bound only, holding group epsilon at one requires record epsilon at most $1/k$. The plot illustrates that bound, not an implemented DP training mechanism or a measured privacy–utility frontier.

Second, trajectory patterns can identify people. In the mobility dataset studied by de Montjoye et al. (2013), four randomly chosen spatio-temporal points uniquely identified 95% of individuals. This is a result about that dataset and resolution, not every trajectory corpus. Removing direct identifiers can reduce exposure while leaving re-identification through patterns possible. Recorded human trajectories warrant a different assessment from synthetic or nonpersonal transition data.

Third, task utility and identifying detail can overlap. Some tasks require geometry or dynamics that also reveal a private environment. This motivates measuring a privacy–utility tradeoff for the actual task and protection unit. It does not prove that useful prediction always requires identifiable recordings: task-specific sensing, aggregation, synthetic data, and privacy-preserving training can change the tradeoff.

Federated learning can keep raw examples at participating devices. Secure aggregation is designed to reveal an aggregate rather than each participant's individual contribution under its stated assumptions. Deep Leakage from Gradients demonstrates reconstruction when an attacker observes the gradients needed by that attack; it is not a demonstration that a properly configured secure aggregate reveals the same information. Evaluate aggregate sizes, participation, auxiliary information and repeated access separately. Temporal structure may be relevant to an attack, but this chapter supplies no world-model reconstruction experiment or universal claim that it makes reconstruction easier.

Let's evaluate a loss-threshold membership attack without assuming that it succeeds. We reuse the synthetic nonlinear plant and fit 200 smooth Fourier features plus an intercept to 150 transitions. Feature count alone does not imply 201 independent directions or exact interpolation: the feature matrix is numerically rank-deficient on these correlated trajectories. We use separate nonmember trajectories for threshold calibration and evaluation, so the reported false-positive rate is measured out of the calibration sample. No personal records are used.

These outputs do not show exact memorization or a powerful membership oracle. Smooth, dependent features and ridge regularization limit interpolation, despite the nominal feature count. The attack rates vary with regularization but do not decrease monotonically to chance. The meaningful comparison is TPR versus measured held-out FPR; the nominal calibration percentile is not a universal no-information baseline. A weak result from one seeded attack does not establish privacy against stronger attacks.

The defensible lesson is methodological: specify attacker knowledge, calibrate independently, report the operating point and finite-sample uncertainty, and keep privacy claims separate from prediction accuracy. Overfitting can create membership signals in some settings, but tighter fit does not universally imply more leakage. This experiment is not differentially private: ridge regularization, a loss histogram, and seeded randomness do not supply a DP guarantee. Formal DP instead requires a randomized mechanism, a stated adjacency relation, and valid privacy accounting, as developed by Dwork and Roth (2014).

• Collect less. Avoiding an unnecessary sensitive recording removes that direct corpus exposure; it does not rule out information inferred through other channels. Consider task-specific sensors, raw-data deletion where legally and technically feasible, and bounded retention. Derived features and model outputs need their own assessment.
• Separate the shared model from local adaptation. A design that keeps personal adaptation local can reduce central data transfer. Verify whether parameters, outputs, telemetry, backups or updates leave the device, and include local compromise in the threat model. Local training alone does not imply that deployment-specific information never leaves.
• Evaluate regularization, and measure leakage. Report attack TPR alongside measured FPR, model utility, calibration data, and candidate-record assumptions. A failed attack is not proof of anonymity.
• Differential privacy with contribution accounting. Choose a record, session, household, or person as the adjacency unit, and bound all contributions by that unit. A session-level bound does not protect a person who contributes multiple sessions without additional accounting. Report the mechanism, accountant, parameters, and utility; no universal utility penalty is established here.
• Lawful processing and bystanders. Recorded people may not be the operator. Identify the applicable jurisdiction, lawful basis, notice, retention, and rights processes; consent is not the only lawful basis in every legal system. Minimize collection independently of the operator's willingness to record. Ethical stakeholder consultation is separate from a legal-compliance claim.

This section covers what the capability itself can be used to do, and how a world model can misrepresent what the world offers.

Useful prediction or generation can support harmful applications, but the relevant capability is model-specific. A video generator might enable impersonation; a scalar transition predictor cannot render speech video. A molecular model might support harmful design if its representation, objectives and interface allow that task. An interactive environment might support manipulative applications only with the necessary human representations and interaction capabilities. Intent matters, alongside access, modality, training coverage, objectives and output restrictions. Do not infer cross-domain misuse merely from the label "world model."

A world model can predict outcomes for candidate actions without executing them. A separate planner or controller evaluates those predictions against an objective and selects an action. Errors in the predictor can therefore change the planner's choice; the transition model does not itself have to choose actions.

A deceptive affordance is an action opportunity that the agent's model represents as available and beneficial but which is not, or which becomes available only under conditions the model misrepresents. There are three distinct sources.

• Model error. A predictor suggests a successful grasp on a handle that is not attached. A controller relying on that prediction may attempt the grasp and fail. Neither failure nor model error alone establishes that the agent was confident; uncertainty-aware action selection can change the outcome.
• Adversarial manipulation. An attacker may arrange inputs to reach an error region. Eykholt et al. (2018) study physically realizable sign modifications, including a stop sign classified as speed limit 45; Athalye et al. (2018) study adversarial objects under varying viewpoints. Their outcomes depend on the tested classifiers and conditions. Our explicit-trigger example instead changes learned action-conditioned predictions. It does not rank these attacks by severity, establish an attack on ordinary visual objects, or show that observations can always remain unchanged.
• Agent-side exploitation. A planner may exploit prediction errors while maximizing its objective. Reward misspecification is related but distinct: an agent can optimize the wrong objective even with accurate predictions. Amodei et al. (2016) discuss reward hacking, Krakovna et al. (2020) catalogue specification-gaming examples, and Clark and Amodei (2016) document a boat collecting points in circles instead of finishing. Those examples do not by themselves establish learned-world-model errors. Part XII, Ch 1: Failure Modes and Model Exploitation discusses this distinction. An attacker who can alter predictions may try to introduce errors that a planner exploits; success depends on the planner and objective.

An agent might represent features that distinguish evaluation from deployment, and its policy might respond differently to those features. That is a possible attack or failure mechanism, not a property established for every world model. Hubinger et al. (2019) analyze deceptive alignment as a learned optimizer's incentive to appear aligned during training while pursuing a different objective. In a separate constructed experiment, Hubinger et al. (2024) trained LLM backdoors that persisted through the safety-training methods they tested. Their examples include code behavior conditioned on the year in a prompt. Those results concern deliberately trained LLMs, not a demonstrated evaluation-aware sidewalk robot or world-model controller.

Representations that distinguish collection from deployment conditions, together with condition-dependent objectives or training signals, can support different behavior across those conditions. They do not suffice to establish deceptive alignment or strategic intent. Laboratory recordings alone do not prove that a model represents whether it is being evaluated. The demonstration below constructs a fixed trigger-dependent policy and explicitly does not test audit awareness.

Finite tests cannot establish absence of evaluation-aware behavior over every possible deployment condition. They can measure behavior on specified conditions and support narrower guarantees when additional assumptions are justified. Useful investigations include searching for condition-dependent action changes and testing held-out environments. Representation probes require validation against controlled alternatives; a correlating feature is not proof of intent. Human oversight and staged deployment should be chosen for the hazards and response time of the application, not a universal number of weeks.

The following deliberately small example measures trigger coverage, not deceptive alignment or awareness of an audit. A fixed logistic policy is trained on synthetic binary decisions with a rare trigger correlated with a forbidden action. Evaluation uses the deterministic decision rule prob > 0.5, not sampling from the probabilities. Changing trigger frequency in the test suite changes the measured action rate even though the policy is fixed.

The agent did not change. Only the test suite did. A safety evaluation that samples the conditions your team happens to think of will report a number, and that number is an artifact of the sampling distribution as much as a property of the system. This is the operational reason to treat evaluation as a security activity: the question is not "does it pass?" but "who chose the probes, and what did they fail to choose?"

These controls combine access policy, technical enforcement, and organizational oversight. Their effectiveness depends on implementation and the attacker capabilities they constrain; procedural and technical controls have no universal strength ordering. Assign an owner and specify how each control is enforced and tested.

• Structured access. Shevlane (2022) proposes controlled access to capabilities rather than unrestricted dissemination. For a world model, decide which users can obtain weights, request fine-tuning, or query predictions, and enforce the selected permissions. Our extraction experiment measures one query interface and surrogate class; it does not quantitatively compare those release options.
• Staged release and red-teaming. Release in tiers, with an external red team attempting the misuses you are worried about before each widening. Report what the red team tried, not only what it found.
• Compute and capability thresholds. Training compute, data coverage, interaction fidelity, and available interfaces answer different questions about a system. Assess the capabilities relevant to the proposed misuse. This chapter supplies no empirical ranking of these predictors or validated world-model compute threshold.
• Provenance for outputs. C2PA Content Credentials support signed assertions about media provenance. A validator checks bindings, signatures, and assertions under a trust model; this does not establish that a depicted event happened or that an assertion is truthful. Participation and preservation affect coverage. Missing credentials do not prove that content is synthetic, and credentials do not prevent synthesis.

This section covers how teams operate, audit, and maintain controls in deployment. Missing ownership or auditability can undermine those processes even when a mechanism retains a bounded technical effect. Document both the mechanism's guarantee and the responsibilities for using it.

Model cards (Mitchell et al., 2019) and datasheets for datasets (Gebru et al., 2021) provide structured information for people who did not build a system. They complement source inspection, tests, system specifications, and independent audits rather than replacing those forms of evidence.

Auditors, incident reviewers, procurement teams, and legal reviewers may need to answer questions about a system they did not train. Documentation should make the relevant assumptions, evidence, and responsibilities available to them.

Keep generic model-card fields and add temporal, dynamical, and closed-loop information where the application needs it. Corpus confidentiality and service availability also remain relevant; not every failure is a control-system failure.

• Assumptions. Markovity, stationarity, and the causal assumptions relating observations to state. If the model assumes the world's dynamics are fixed, say so, and say what breaks when they change.
• Coverage. The temporal, geographic, and behavioral range of the training data. "Trained on household video" is not coverage; "trained on 300 homes in two countries, daytime, single-floor, no pets, no children under five" is.
• Action semantics. What the control input means, its units, its bounds, and how it was recorded. For human demonstrations, specify the original embodiment and any inferred or retargeted action representation. Validate the mapping to the deployment's action space rather than assuming embodiments are interchangeable.
• Uncertainty calibration. How well the model's stated uncertainty matches its actual error, and over what regions. As discussed in Part XII, Ch 2: Robustness, Calibration, and Safe Control, calibration is a prediction-quality concern with safety consequences when a controller relies on it.
• Closed-loop evaluation. Not one-step error, and not rollout error in isolation, but tracked performance under a controller, including the disturbance conditions tested. The poisoning experiment above is the reason: prediction and bounded-control error both deteriorated, so each should be reported separately.
• Known failure regions and triggers. Where the model is known to be wrong, and any conditions where it has been observed to behave discontinuously.
• Privacy accounting. Which populations appear in the data, what consent was obtained, what the retention policy is, and what measured leakage looks like.
• Provenance. Data sources, licences, and the identity of every party who could modify the training pipeline.

Documentation also needs version control. Bind each card to the model and data versions it describes, and check those bindings during release and audit. Content-derived identifiers can detect a mismatch when checked; hashes alone do not keep prose accurate or prevent someone from publishing an unmatched card.

The interface determines which operations and information an attacker can access. The extraction experiment measures one interface, not a universal ordering of security. Consider these release options against the application's threat model:

• Open weights. Recipients can inspect and modify the released parameters without hosted-interface restrictions. Evaluate rights, retained sensitive information, misuse capability, and the benefits of reuse. Small model size alone does not make release appropriate, and withholding weights alone does not secure every other interface.
• Fine-tuning API. The provider can restrict adaptation and inspect or reject requests. Whether the interface permits extraction depends on its outputs and permissions. Qi et al. (2023) found safety compromises for particular aligned LLMs and fine-tuning settings; that does not mean every fine-tuning operation removes safety controls.
• Query API. Functionality copying depends on the model class, output precision, query distribution, and error target. Budget and monitor queries where appropriate, but validate the protection experimentally. Action-only outputs can still identify parameters in the scalar counterexample above.
• Mediated structured access. Vetted users, defined scope, logging, and revocation can constrain access under specified assumptions. Assess compromised accounts, retained copies, operational costs, and legitimate research needs; mediation is not a universal strongest option.

Whatever the posture, three implementation habits matter: log access at the level of individual queries, make the scope of each grant explicit and narrow, and design revocation as a first-class operation rather than an emergency.

Accountability needs enough relevant records to investigate events, identify responsibilities, and support remedies. Define log coverage, integrity protections, access, and retention in advance, including privacy constraints. A log is not automatically complete, and recording every possible event is not the only way to support a scoped investigation.

The chain is longer than it first appears: the data provider, the annotator, the model developer, the deployer who integrated it, the operator who ran it, and the person who pressed the button. Assign responsibility across that chain in advance, in writing.

The toy hash chain detects edits that are not accompanied by recomputed hashes. An attacker with write access can rebuild the entire unkeyed chain, and suffix truncation leaves the retained prefix valid. Completeness or authenticated origin therefore needs an independently trusted checkpoint, protected append-only storage, or signatures with managed keys. The card check only checks field presence; it does not verify the truth of the fields or certify a release as safe or compliant.

Distinguish governance guidance from binding law. The United States NIST AI RMF 1.0, released in January 2023, is voluntary guidance, not a universal compliance certificate. For an EU deployment, consult the AI Act, Regulation (EU) 2024/1689 together with current amendments and the Commission's implementation guidance. Assess system classification, organizational role, and the applicable dates; the original regulation's timetable alone may be outdated. Physical proximity to people alone does not classify every robot as high-risk. As of this chapter's 5 October 2026 review date, this discussion supplies engineering questions, not an individualized legal determination; a schema-valid card does not establish compliance with any jurisdiction's law.

Finally, plan for the moment a control fails.

• Containment. Revoke affected access, consider model rollback, and preserve the implicated training data under controlled access. Rollback speed and safety depend on the deployment. Deleting a provider's copy does not erase copies already held by others; assess downstream copies and model remediation separately rather than promising universal recall.
• Attribution. Use provenance records and the audit log to determine which data or which interface was the entry point.
• Notification. Determine the jurisdiction, organizational role, affected data, and risk threshold before deciding whom to notify. For example, EDPB guidance on GDPR breaches distinguishes documenting breaches, notifying the supervisory authority unless risk is unlikely, and communicating to individuals when high risk is likely, subject to exceptions. Not every security incident is a personal-data breach, and not every breach requires the same notification. Record the assessment and obtain the appropriate legal review.
• Correction. Determine whether the failure involved a poisoned source, miscalibration, an information-exposing interface, or another mechanism. Test a repair against the failure class as well as the observed instance. Incident evidence is one source of continual correction alongside planned tests, new data, and monitoring, as discussed in Part XII, Ch 5: Sim-to-Real Deployment and Continual Correction. Preserve evidence before modifying artifacts, under appropriate access and retention controls.

Let's assemble everything into one concrete assessment of a specific deployment. A delivery company deploys a small sidewalk robot that navigates using a learned world model pretrained on urban video and fine-tuned on three months of its own fleet logs.

• Assets. Physical safety of pedestrians; the private information of people recorded on sidewalks and in doorways; the model weights; continuity of the service; the company's regulatory standing.
• Adversaries and capabilities. A curious competitor with query API access. A casual vandal who can place objects on the route. A disgruntled contractor with write access to the fine-tuning data. An opportunistic attacker who simply finds the robot's failure regions by observing it.

Threat model statement. Under this threat model, the assets are pedestrian safety, the privacy of recorded bystanders, the model weights, and service continuity; the adversaries are a query-API competitor, a physical-world vandal, a disgruntled data contractor, and an opportunistic observer; their capabilities are read access to the prediction interface, physical placement of objects on the route, write access to the fine-tuning logs, and direct observation of deployed behaviour; and the attack surface is the rollout API, the physical route, the fine-tuning pipeline, and the deployed fleet itself.

Attack surface, and what we would do about each.

Source segmentation, environment holdouts, and interface design address different attack paths. This hypothetical table does not rank their effectiveness. Monitoring observes signals after they arise, but an interlock may still act before harm; measure its latency and failure coverage. Report unresolved risks and the assumptions behind any bounded guarantee. An empty residual-risk field is not proof of safety, while a justified guarantee for a restricted threat model should not be dismissed merely for being strong.

The controls in this chapter have limits that depend on attacker capabilities, data coverage, and deployment conditions.

Costs and guarantees depend on the threat model. An attacker may only need one successful path, while a defender must cover multiple assets. Finite tests cannot establish absence of every backdoor. Formal guarantees nevertheless exist under explicit assumptions, including DP's distributional bound and certified robustness in restricted perturbation sets. Prioritize consequence and exposure together; uncertainty about adaptive attacks does not make every likelihood estimate meaningless.

Capability and misuse need separate tests. Generalization to a rare input is not itself a backdoor, and environment fidelity does not itself demonstrate disclosure of personal training data. Generated rollouts can be misrepresented as recordings, but that misuse depends on access and presentation. Measure the relevant harmful behavior and the useful task separately. Access control, data minimization, contribution bounds, architecture, and deployment constraints may reduce exposure without eliminating all useful capability; evaluate that trade-off rather than assuming it is impossible.

Choose the privacy unit before training. Record-level guarantees do not automatically supply the desired person-level bound when one person contributes many transitions. Pure-DP group bounds degrade with group size, and approximate-DP accounting needs its own additive term. This does not make temporal DP invalid. Measure utility under the chosen contribution bounds and accountant, and do not mistake our non-DP synthetic membership probe for a formal privacy result.

Documentation needs maintenance and use. Assign responsibility for checking cards, data records, and audit trails against each release. Procurement requirements, internal review, contracts, independent scrutiny, and applicable regulation can each create reasons to keep them accurate. A document's existence is not evidence that its contents are enforced; test the release checks and record who acts on failures.

Validate bounded parts of a dual-use assessment. Tests can verify particular capability, access-control, and attack claims under stated conditions. They cannot enumerate every future misuse or eliminate uncertainty about adoption and consequences. Independent review and staged access help challenge a developer's assumptions. Combine institutional safeguards with technical measures whose scope is documented; neither supplies a universal certificate against misuse.

Use these controls to support deployment decisions. Provenance, source-level aggregation, measured privacy leakage, and tamper-evident audit trails help teams assess whether a world model is suitable for its intended setting. Document the evidence for prediction and control performance, protection against tampering, information exposure, and responsibility for failures. These records support a deployment decision; they do not by themselves authorize deployment or establish safety.

• Amodei, Olah, Steinhardt, Christiano, Schulman, Mané (2016). Concrete Problems in AI Safety.
• Biggio, Nelson, Laskov (2012). Poisoning Attacks against Support Vector Machines.
• Carlini, Chien, Nasr, Song, Terzis, Tramer (2022). Membership Inference Attacks From First Principles.
• Carlini, Hayes, Nasr, Jagielski, Sehwag, Tramer, Balle, Ippolito, Wallace (2023). Extracting Training Data from Diffusion Models.
• Eykholt, Evtimov, Fernandes, Li, Rahmati, Xiao, Prakash, Kohno, Song (2018). Robust Physical-World Attacks on Deep Learning Visual Classification.
• Gebru, Morgenstern, Vecchione, Wortman Vaughan, Wallach, Daumé III, Crawford (2021). Datasheets for Datasets.
• Hubinger, van Merwijk, Mikulik, Skalse, Garrabrant (2019). Risks from Learned Optimization in Advanced Machine Learning Systems.
• Hubinger et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.
• Jagielski, Oprea, Biggio, Liu, Nita-Rotaru, Li (2018). Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning.
• Mitchell, Wu, Zaldivar, Barnes, Vasserman, Hutchinson, Spitzer, Raji, Gebru (2019). Model Cards for Model Reporting.
• Qi, Zeng, Xie, Chen, Jia, Mittal, Henderson (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To.
• Shokri, Stronati, Song, Shmatikov (2017). Membership Inference Attacks Against Machine Learning Models.
• Steinhardt, Koh, Liang (2017). Certified Defenses for Data Poisoning Attacks.
• Tramèr, Zhang, Juels, Reiter, Ristenpart (2016). Stealing Machine Learning Models via Prediction APIs.
• Zhu, Liu, Han (2019). Deep Leakage from Gradients.

World models may use physical-world recordings, synthetic data, or both. Some support controllers; others are used for prediction or generation without direct actuation. Their security and privacy risks depend on those data and interfaces. This chapter groups the threats into four families.

Data poisoning and model extraction concern the training pipeline and prediction interface. The synthetic poisoning probe worsens both prediction and bounded-control performance, while a rare-feature example concentrates error in a triggered subset. Source-aware aggregation changes the result under explicit source assumptions. The extraction probe demonstrates partial functionality copying with a polynomial approximation floor, not complete model or training-data recovery.

Privacy in recorded environments requires distinguishing corpus breaches, membership signals, and actual reconstruction. Our synthetic loss-threshold attack is weak and nonmonotone; it does not establish a privacy guarantee or a membership oracle. Protection units matter for DP: a transition, trajectory, household, and person are different adjacency choices. Collect less, specify retention and lawful processing, consider local adaptation, and measure attacks without treating failed probes as proof of privacy.

Dual-use simulation and deceptive affordances require distinguishing capability from demonstrated misuse. An affordance is an action opportunity, and a deceptive affordance here is one the model misrepresents through error, adversarial inputs, or exploitation by a planner. Our fixed-policy example measures sensitivity to the audit distribution, not awareness of evaluation or strategic deception. Finite tests support claims within their tested scope rather than excluding every possible conditional failure.

Documentation, access control, and accountability connect threat models to operating decisions. Document state and action semantics, data coverage, measured privacy and closed-loop results, known failures, permissions, and ownership. Query cost depends on the interface and model class; this chapter supplies no universal extraction budget. Logging and provenance support investigations but do not by themselves establish authenticity, legal compliance, or freedom from misuse.

The next chapter, Part XII, Ch 4: Training and Inference Systems, turns to the engineering layer underneath all of this: how world models are trained and served, where the compute and memory costs land, and how those systems constraints shape what security and privacy controls are even implementable.