Security, Ethics, Privacy, and Governance › Worked Example: A Threat Model for One Deployment
World Models HandbookAll chapters Security, Ethics, Privacy, and Governance › Worked Example: A Threat Model for One DeploymentOne section · Sections in this chapter7 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's assemble everything into one concrete assessment of a specific deployment. A delivery company deploys a small sidewalk robot that navigates using a learned world model pretrained on urban video and fine-tuned on three months of its own fleet logs.
- Assets. Physical safety of pedestrians; the private information of people recorded on sidewalks and in doorways; the model weights; continuity of the service; the company's regulatory standing.
- Adversaries and capabilities. A curious competitor with query API access. A casual vandal who can place objects on the route. A disgruntled contractor with write access to the fine-tuning data. An opportunistic attacker who simply finds the robot's failure regions by observing it.
Threat model statement. Under this threat model, the assets are pedestrian safety, the privacy of recorded bystanders, the model weights, and service continuity; the adversaries are a query-API competitor, a physical-world vandal, a disgruntled data contractor, and an opportunistic observer; their capabilities are read access to the prediction interface, physical placement of objects on the route, write access to the fine-tuning logs, and direct observation of deployed behaviour; and the attack surface is the rollout API, the physical route, the fine-tuning pipeline, and the deployed fleet itself.
Attack surface, and what we would do about each.
| Attack path | Control | Residual risk |
|---|---|---|
| Trigger object placed on the route, exploiting a data-poor visual condition | Test held-out environments and relevant physical perturbations; monitor predictions with a defined stop or fallback policy | Coverage remains limited; effectiveness and response time need validation |
| Contractor corrupts a fraction of fine-tuning logs to bias the navigation gain | Per-source fitting with median aggregation; track the gain parameter directly in monitoring; audit logs of who wrote which rows | A patient adversary who controls many sources can still shift the median |
| Competitor copies prediction functionality through the rollout API | Restrict outputs and query grants; validate copying attacks against the actual interface; monitor query patterns | Action-only outputs and multiple accounts can retain extraction paths; copying success is model-dependent |
| Generator reproduces recognizable pedestrians in marketing footage | Minimize personal training data; restrict generation and release; test reconstruction and review outputs; add provenance for attribution | Provenance does not prevent privacy leakage or certify consent; reconstruction tests have limited coverage |
| Policy selects a forbidden action when a specific street sign is present | Test held-out conditions and candidate triggers; validate representational probes; stage deployment with a response policy | Finite tests do not exclude every conditional failure or establish strategic intent |
Source segmentation, environment holdouts, and interface design address different attack paths. This hypothetical table does not rank their effectiveness. Monitoring observes signals after they arise, but an interlock may still act before harm; measure its latency and failure coverage. Report unresolved risks and the assumptions behind any bounded guarantee. An empty residual-risk field is not proof of safety, while a justified guarantee for a restricted threat model should not be dismissed merely for being strong.