Security, Ethics, Privacy, and Governance › Limitations and Impact

The controls in this chapter have limits that depend on attacker capabilities, data coverage, and deployment conditions.

Costs and guarantees depend on the threat model. An attacker may only need one successful path, while a defender must cover multiple assets. Finite tests cannot establish absence of every backdoor. Formal guarantees nevertheless exist under explicit assumptions, including DP's distributional bound and certified robustness in restricted perturbation sets. Prioritize consequence and exposure together; uncertainty about adaptive attacks does not make every likelihood estimate meaningless.

Capability and misuse need separate tests. Generalization to a rare input is not itself a backdoor, and environment fidelity does not itself demonstrate disclosure of personal training data. Generated rollouts can be misrepresented as recordings, but that misuse depends on access and presentation. Measure the relevant harmful behavior and the useful task separately. Access control, data minimization, contribution bounds, architecture, and deployment constraints may reduce exposure without eliminating all useful capability; evaluate that trade-off rather than assuming it is impossible.

Choose the privacy unit before training. Record-level guarantees do not automatically supply the desired person-level bound when one person contributes many transitions. Pure-DP group bounds degrade with group size, and approximate-DP accounting needs its own additive term. This does not make temporal DP invalid. Measure utility under the chosen contribution bounds and accountant, and do not mistake our non-DP synthetic membership probe for a formal privacy result.

Documentation needs maintenance and use. Assign responsibility for checking cards, data records, and audit trails against each release. Procurement requirements, internal review, contracts, independent scrutiny, and applicable regulation can each create reasons to keep them accurate. A document's existence is not evidence that its contents are enforced; test the release checks and record who acts on failures.

Validate bounded parts of a dual-use assessment. Tests can verify particular capability, access-control, and attack claims under stated conditions. They cannot enumerate every future misuse or eliminate uncertainty about adoption and consequences. Independent review and staged access help challenge a developer's assumptions. Combine institutional safeguards with technical measures whose scope is documented; neither supplies a universal certificate against misuse.

Use these controls to support deployment decisions. Provenance, source-level aggregation, measured privacy leakage, and tamper-evident audit trails help teams assess whether a world model is suitable for its intended setting. Document the evidence for prediction and control performance, protection against tampering, information exposure, and responsibility for failures. These records support a deployment decision; they do not by themselves authorize deployment or establish safety.

• Amodei, Olah, Steinhardt, Christiano, Schulman, Mané (2016). Concrete Problems in AI Safety.
• Biggio, Nelson, Laskov (2012). Poisoning Attacks against Support Vector Machines.
• Carlini, Chien, Nasr, Song, Terzis, Tramer (2022). Membership Inference Attacks From First Principles.
• Carlini, Hayes, Nasr, Jagielski, Sehwag, Tramer, Balle, Ippolito, Wallace (2023). Extracting Training Data from Diffusion Models.
• Eykholt, Evtimov, Fernandes, Li, Rahmati, Xiao, Prakash, Kohno, Song (2018). Robust Physical-World Attacks on Deep Learning Visual Classification.
• Gebru, Morgenstern, Vecchione, Wortman Vaughan, Wallach, Daumé III, Crawford (2021). Datasheets for Datasets.
• Hubinger, van Merwijk, Mikulik, Skalse, Garrabrant (2019). Risks from Learned Optimization in Advanced Machine Learning Systems.
• Hubinger et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.
• Jagielski, Oprea, Biggio, Liu, Nita-Rotaru, Li (2018). Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning.
• Mitchell, Wu, Zaldivar, Barnes, Vasserman, Hutchinson, Spitzer, Raji, Gebru (2019). Model Cards for Model Reporting.
• Qi, Zeng, Xie, Chen, Jia, Mittal, Henderson (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To.
• Shokri, Stronati, Song, Shmatikov (2017). Membership Inference Attacks Against Machine Learning Models.
• Steinhardt, Koh, Liang (2017). Certified Defenses for Data Poisoning Attacks.
• Tramèr, Zhang, Juels, Reiter, Ristenpart (2016). Stealing Machine Learning Models via Prediction APIs.
• Zhu, Liu, Han (2019). Deep Leakage from Gradients.