Security, Ethics, Privacy, and Governance › Security, Ethics, Privacy, and Governance

Consider a hypothetical household robot. Its planner uses a learned transition model to predict the consequences of candidate actions rather than testing every action physically. Suppose we train that model on volunteered recordings of homes.

Three fictional incidents below illustrate this. They are threat scenarios, not measurements from a deployed robot, and each raises questions we would need to investigate. None follows automatically from good video quality or low prediction error.

Suppose three things go wrong, months later, in three different ways.

A data provider we contracted with was acquired mid-project, and its annotation contractor quietly relabeled a small set of clips. The relabelled clips were not random noise; they were chosen so that the model misestimates how far the arm travels for a given command. Prediction accuracy barely moves. The robot, however, starts oscillating whenever it reaches for a shelf.

A partner lab with authorized prediction-API access collects responses and fits a surrogate. Depending on its query budget, coverage, and model class, it might approximate some of the original model's behavior. That would not establish recovery of the original weights, the training records, or the full closed-loop agent.

Your marketing team generates footage that resembles a private kitchen from the recordings. We must investigate whether this is memorized training content, a coincidental resemblance, or an ordinary combination of learned features before asserting a privacy breach. Recognizable personal content would require a release decision separate from the model's prediction metrics.

These three stories have different technical mechanisms, but they share a structure. In each one, a world model is not merely a predictor that can be wrong. It is a trusted oracle whose outputs other systems act on, whose internals may retain information about other people, and whose outputs can be produced at industrial scale. The previous chapters in this part examined failures that arise from the model's own inductive biases (Failure Modes and Model Exploitation) and from the difficulty of staying calibrated and safe under distribution shift (Robustness, Calibration, and Safe Control). Both chapters concerned failures that happen because the world, or the model, behaves in a way nobody intended.

This chapter covers deliberate attacks, exposure of information learned from data, and harmful uses of simulated outcomes. It has four sections: poisoning and extraction, privacy, dual-use simulation, and governance. Some threat models allow an attacker to observe and adapt to deployed defenses; others restrict that knowledge. Evaluation needs to state which case it covers. The implementations demonstrate specific mechanisms or limitations, with computation separated from displayed outputs and presentation-only plots.

It helps to map the standard security vocabulary onto world models explicitly.

• Integrity concerns unauthorized modification or destruction of data and system behavior. Poisoning and backdoors can compromise model integrity; an inaccurate but untampered predictor is not, merely for that reason, a security compromise.
• Confidentiality concerns disclosure outside authorized destinations. Functionality extraction can expose a protected predictive function. Successful membership inference or reconstruction can expose information about training data, but simply attempting those attacks does not establish disclosure.
• Availability is the property that the system keeps working. Inference-time flooding and resource exhaustion against a large generative world model are availability attacks. We return to resource budgets in Part XII, Ch 4: Training and Inference Systems.
• Authenticity concerns whether an asserted source or origin is genuine, a longstanding security concern. Generated media makes origin claims harder to assess; a plausible recording is not itself proof that an event occurred.

These properties separate prediction reliability, protection against unauthorized changes, information exposure, service continuity, and trustworthy origin claims. This chapter uses explicit threat models and synthetic probes, not a certification of real-world security. Defenses can have formal guarantees under stated assumptions, but finite empirical testing cannot rule out every future adaptive attack.