Security, Ethics, Privacy, and Governance › Data Poisoning and Model Extraction

This section covers attacks against training data and inference interfaces. Poisoning changes what the learner fits. Query-based extraction tries to reproduce some model functionality; it need not recover the original weights. The training pipeline, annotation process and prediction API belong in the threat model alongside the model itself.

A world model is assembled from many stages, and each is a place where an adversary can insert influence:

• Collection. Cameras, teleoperated demonstrations, scraped video, and logs from deployed fleets. Whoever controls a sensor or an upload path controls part of the data distribution.
• Annotation. Labels, captions, segmentation masks, reward estimates, and filtered "usable" subsets. If a contractor supplies annotations, include its permissions and checks in the threat model. Outsourcing does not by itself establish attacker access or low attack cost.
• Curation and filtering. Deduplication, quality scoring and safety filtering may use deterministic rules, learned models or both. An attacker who can shape submitted content may try to pass these checks; success depends on the particular filter and permissions.
• Pretraining corpora. Public accessibility does not imply attacker write access. Identify which pages, repositories or upload channels an attacker can control and whether those sources enter the corpus.
• Fine-tuning and online adaptation. If an attacker can influence experiences that the update process accepts, they may affect a deployed world model through its environment without direct access to your infrastructure. Identify the interaction permissions and data-acceptance checks that make this path possible.

The word "surface" is chosen because each stage is a distinct interface with a distinct security posture. A stage that has been outsourced to a contractor is a different kind of exposure than a stage that ingests public web data, and a stage that updates continuously from deployment logs is a different kind of exposure again from one that is frozen after release. Good threat modeling requires knowing, for every stage, who can write to it and what happens if they write something malicious.

Let the learner be a function that maps a dataset $D$ to parameters $\hat\theta(D)$. The attacker chooses a perturbation of the training data, constrained to a budget $\mathcal{B}$ (a fraction of rows, a bound on per-row modification, or control over a subset of sources), to maximize a damage objective $\mathcal{L}$:

$$\begin{aligned} \max_{\tilde{D} \in \mathcal{B}} \; & \mathcal{L}\big(\hat\theta(\tilde{D})\big) \\ \text{subject to} \quad & \hat\theta(\tilde{D}) \in \arg\min_{\theta} \left[\sum_{(x,y) \in \tilde{D}} \ell\big(f_\theta(x), y\big) + R(\theta)\right] \end{aligned}$$

where:

• $D$: the clean training dataset the learner would otherwise see
• $\tilde{D}$: the attacker-chosen poisoned dataset, drawn from the allowed budget set $\mathcal{B}$
• $\hat\theta(D)$: the fitted parameters the learner returns for dataset $D$
• $\mathcal{L}$: the attacker's damage objective, a scalar measuring how bad the fitted model is for the defender
• $\mathcal{B}$: the set of datasets the attacker is allowed to submit (bounded by row fraction, per-row magnitude, or controlled sources)
• $\ell\big(f_\theta(x), y\big)$: the per-example training loss of predictor $f_\theta$ on input $x$ with target $y$
• $R(\theta)$: the learner's regularizer; the ridge example uses $R(\theta)=\lambda\|\theta\|_2^2$ with squared-error loss
• The inner $\arg\min$: the learner refits from scratch on the poisoned data, with its optimizer or tie-breaking rule fixed if minimizers are not unique; this makes the outer maximization a bi-level (or bilevel) problem

This is a bi-level problem: the outer maximization must account for the fact that the learner refits after seeing the poison. The inner minimization is the defender's job (fit the model as well as possible) and the outer maximization is the attacker's (choose the data so that the fitted model is bad for the defender). Attackers use a few standard approximations.

• Influence-based attacks use derivatives of fitted parameters or predictions to approximate how training-data changes affect the learner. For ridge with fixed design $X$ and fixed positive $\lambda$, the label derivative is exactly $(X^\top X+\lambda I)^{-1}X^\top$; because the fit is linear in the labels, it also gives the exact finite label update. Changes to features or other learners generally require different derivatives or local approximations (Jagielski et al., 2018, "Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning"). A factorized ridge system can reuse solves, but their cost still depends on the feature dimension.
• Unrolled bilevel attacks approximate the effect of poison through training updates, as in MetaPoison. Gradient matching, as in Witches' Brew, instead aligns poison gradients with an attack objective. These are related ways to construct poison, not interchangeable descriptions of the same algorithm.
• Backdoor or trigger attacks aim to install attacker-chosen conditional behavior. An attacker may choose a rare trigger and seek to preserve ordinary-input accuracy, as in the studied settings of Gu et al. (2017, "BadNets") and Chen et al. (2017). Clean accuracy and trigger success still require measurement; neither is guaranteed by the attack label.

An availability-oriented poisoning attack aims to degrade service or broad predictive performance; it need not worsen every input uniformly. A targeted attack aims at a particular outcome or input region. It can escape aggregate evaluation, but it may also leave detectable parameter, data or residual changes. Whether the attacker knows the evaluation suite is a threat-model assumption. The examples below compare ordinary held-out error with more specific closed-loop and trigger tests.

Let's make the bi-level problem concrete. For ridge regression the fitted parameters are

$$\hat\theta = \big(X^\top X + \lambda I\big)^{-1} X^\top y$$

where:

• $X \in \mathbb{R}^{n \times d}$: the design matrix, whose $i$-th row stacks the features of the $i$-th training input (here $[s_i, a_i]$)
• $y \in \mathbb{R}^{n}$: the vector of recorded next-states $s_{i+1}$
• $\lambda>0$: the ridge regularization strength, which shrinks fitted parameters and makes $X^\top X+\lambda I$ positive definite; with $\lambda=0$, the displayed inverse instead requires full column rank
• $I$: the $d \times d$ identity matrix
• $X^\top X$: the Gram matrix, the sufficient statistic that summarizes how strongly the data excites each parameter direction

The form of this solution is the key to the whole analysis. The fitted parameters depend on the data only through two summary quantities, the Gram matrix $X^\top X$ and the cross-moment vector $X^\top y$. Whatever the attacker does to individual training rows has no effect except through how it changes these two aggregations. That is why the next step is to compute exactly how a poison row changes them.

Let $\phi_p \in \mathbb{R}^d$ be the feature vector shared by the poison rows and $y_p$ the target the attacker records for each. Appending $m$ such identical rows adds $m\phi_p\phi_p^\top$ to the Gram matrix and $m\phi_p y_p$ to the cross-moment vector. If the attacker wants a specific parameter vector $\theta^\star$, the fitted parameters must satisfy the ridge normal equations:

$$\big(X^\top X + m\phi_p\phi_p^\top + \lambda I\big)\,\theta^\star = X^\top y + m\phi_p y_p$$

where:

• $X^\top X + m\phi_p\phi_p^\top$: the poisoned Gram matrix, the clean sufficient statistic plus the rank-one contribution of the poison rows
• $X^\top y + m\phi_p y_p$: the poisoned cross-moment vector, the clean statistic plus the poison contribution
• $\theta^\star$: the parameter vector the attacker wants the refit learner to return

Rearranging to isolate the poison contribution gives the required poison condition:

$$\begin{aligned} m\,\phi_p\big(y_p - \phi_p^\top \theta^\star\big) &= r \\ r &= (X^\top X + \lambda I)\theta^\star - X^\top y \end{aligned}$$

where:

• $m$: the number of identical poison rows the attacker appends
• $\phi_p \in \mathbb{R}^d$: the feature vector shared by all $m$ poison rows (for the dynamics model, the pair $(s_p, a_p)$)
• $y_p$: the target the attacker records for each poison row (the next-state $s_p'$)
• $\theta^\star$: the parameter vector the attacker wants the learner to return
• $r$: the residual of the clean fit against $\theta^\star$, fully determined by the clean data and the attacker's chosen target
• The left-hand side is the influence of the appended rows; permitted row counts, feature vectors and labels constrain whether it can equal $r$

The residual $r$ is fixed by the clean data and the target parameter vector. With identical appended rows, the required vector must lie along $\phi_p$; arbitrary feature bounds or label constraints can make a desired target infeasible. Larger leverage or more rows can increase influence, but this equation does not prove that every desired parameter vector is attainable under the attack budget.

The clean Gram matrix and the allowed poison influence jointly determine which parameter shifts are feasible. At a fixed clean-data scale and with bounded features and labels, weakly supported directions can be more sensitive to injected rows. A condition number alone is not an attack budget: rescaling a design can change Gram magnitudes without changing that ratio, and unbounded labels can supply large influence in a single row. Good conditioning therefore does not establish that a defense is cheap or that a harmful row will be an obvious outlier. Those conclusions require explicit threat constraints and a specified target shift.

Let's run the experiment. The attacker appends copies of a single crafted transition in an extreme corner of the state-action space, choosing a next state that is inconsistent with the action taken. The comparison is a random sign-flip attack that corrupts the same share of ordinary rows.

The constructed corner attack appends repeated transitions at an extreme feature pair. Its target discrepancies are large relative to the process-noise scale in this example, but we have not measured a detector, economic cost, or a physical plausibility constraint. Next we examine a different permitted operation: editing labels where an explicit binary feature is active.

Suppose a binary feature $z$ is rarely active and has its own column in a full-rank linear design. Under unregularized least squares, changing the labels to $y-\delta z$ shifts that column's coefficient by exactly $-\delta$. Ridge can attenuate the shift and change other coefficients too. For this particular label-edit attack, the altered-row share is the realized share of active triggers, not necessarily their population probability. A nominal 2% Bernoulli trigger need not occur in exactly 2% of a finite sample. Very rare triggers can be absent altogether. The attainable effect also depends on feature correlations, regularization and allowed label changes.

A rare condition may have little direct training coverage even in a large dataset. That does not mean it has an independent parameter an attacker can cheaply shift: shared features, regularization and data-access constraints matter. Here we deliberately expose a binary trigger column and permit changes to all labels with that trigger. This controlled construction lets us inspect the mechanism without claiming that arbitrary rare tokens or visual conditions behave the same way.

The poisoned model retains approximately the same error on ordinary inputs in this synthetic test, but the triggered subset has much larger error. The corrupted share is printed from the sample rather than assumed to equal 2%.

With trigger $z=1$ and an unsaturated steady state, the controller's fixed point satisfies $s=(s^\star-\hat c)/\hat a$. Relative to $s^\star$, its bias is $(s^\star-\hat c)/\hat a-s^\star$, approximately $-\hat c$ when $\hat a$ is near one. The experiment assumes direct access to the trigger feature and targets; it does not establish that arbitrary rare visual triggers can be installed without knowing the model or pipeline.

Different defenses constrain different parts of the attack. Their assumptions need to be stated together.

• Provenance and segmentation. Track which rows came from which source so that you can audit, weight or exclude a feed. Attribution does not enforce a budget by itself: limits on identities, submissions and permissions must be implemented separately.
• Robust aggregation across sources. Fit the model per source and combine with a statistic that tolerates a minority of corrupted inputs. We demonstrate this below.
• Influence-function audits. Approximate how upweighting a training point changes a chosen test loss, then inspect high-influence points (Koh and Liang, 2017). The choice of test objective matters. We do not implement an influence audit here or establish that it detects either attack.
• Robust losses and trimmed objectives. Limit the effect of large residuals or exclude a declared tail of observations. Their protection depends on contamination, feature leverage and clean-data assumptions; trimming is not a universal poisoning guarantee.
• Trigger discovery. Search for inputs on which behavior changes unexpectedly. Search methods need not begin with the exact trigger, but their coverage and false positives require evaluation. Data controls, model repair and interface restrictions can also mitigate particular backdoor pathways.

Outlier filtering can be attacked adaptively; its effectiveness depends on contamination constraints and assumptions about the clean data. Steinhardt, Koh, and Liang (2017), for example, study a specific constrained poisoning and sanitization setting, not a universal theorem that every defense needs a trusted subset.

Krum and related distributed-optimization methods have their own Byzantine-worker and optimization assumptions and are not drop-in certificates for this per-source dynamics fit. Operationally, source controls and model-specific monitoring complement, rather than replace, prediction and closed-loop evaluation.

Here is a controlled aggregation comparison. Forty sources initially provide one hundred clean transitions each. Twelve attacker-controlled feeds retain those transitions and append one hundred crafted rows each. Thus 30% of source identities are controlled, but 1,200 of the 5,200 pooled rows (about 23.08%) are injected. Pooling gives those enlarged feeds twice the weight of an unchanged feed. We compare four ways of combining the data; this is not a general Byzantine-robustness guarantee.

Pooling gives the corrupted sources extra weight because they append rows. In this sample only the pooled estimate falls below the simplified gain threshold. Equal-weight source averaging shifts the gain but remains above it, while the median and heavily trimmed mean are closer to the true gain. These are coordinate-wise estimates in a scalar example, not a certified defense for neural dynamics models. Median guarantees require assumptions about the number and dispersion of honest sources; an attacker who controls source identities can defeat the source-count premise.

Now we turn to the other end of the pipeline. Suppose the model is good and our data is clean. Can an adversary simply take it?

Prediction interfaces can enable functionality extraction. Tramèr et al. (2016) demonstrated high-fidelity extraction for several model classes and API settings; that is not a universal query bound for world models. Returned scalar values can constrain a surrogate, but label outputs are not universally one bit and rollout outputs can be highly dependent. Recovering behavior on a query distribution is distinct from recovering weights, private training records, or planning performance.

Extraction raises several concerns for a world-model deployment:

1. A usable surrogate can reproduce some predictions that an agent relies on. It does not automatically supply the agent's objective, policy, reward model or search procedure.
2. Access can change which protections the owner can enforce. Exact copying alone need not change learned behavior. Qi et al. (2023) show that certain fine-tuning procedures can compromise safety alignment in studied language models; this is not a result about every copied world model.
3. A local surrogate can support offline searches without the original API's rate limits or fees. Those searches still consume compute, and their usefulness depends on surrogate fidelity, available derivatives and the attacker's access to the deployment environment.
4. Authorization, licensing, and contractual conditions need assessment under the applicable law. This technical probe supplies no general legal conclusion about extraction. Contractual restrictions can be administrative controls or deterrents without technically preventing copying.

Include extraction in the deployment's threat model when a useful copied function would expose protected assets. A surrogate may move some capability beyond the original provider's controls. The costs of prevention depend on the chosen access, monitoring, and output restrictions; they need not all reduce prediction accuracy.

Let's measure approximate functionality copying on a synthetic two-dimensional domain. A known noiseless linear form with $d$ unknown coefficients can be recovered from $d$ linearly independent scalar equations, but that result depends on its form, output precision, and rank. Here the victim uses 96 random Fourier features plus an intercept. The attacker uses a different basis and may face an approximation floor even with unlimited queries.

The output reports a substantial reduction in error as query coverage increases, but the surrogate remains less accurate against truth than the victim. The evaluation-only least-squares fit gives the best degree-four approximation on this grid; its residual shows that more queries alone cannot remove the chosen basis mismatch. This seeded experiment provides no general query budget for copying world models and no evidence of training-data recovery. Another surrogate or domain could behave differently.

• Budget calls and returned information. Cap query counts, batch sizes, rollout lengths and output precision. A hundred returned states can expose more than one scalar, but redundant or highly dependent outputs need not provide a hundred independent constraints. Evaluate the actual response format and query distribution.
• Evaluate reduced output resolution. Quantization or hidden random noise can limit fidelity or increase the queries needed for a particular extraction target, but neither has that effect universally. Integer quantization leaves a known constant victim in the class $\theta\in\{0,1\}$ exactly identifiable with one query. Hidden noise bounded by $0.1$ does too, since the possible responses for zero and one are disjoint. For continuous parameters, quantization can instead make some values indistinguishable, while repeated queries can average independent noise. Measure both extraction error and legitimate utility.
• Restrict what the interface can express. Returning a planned action instead of a transition limits directly exposed outputs, but does not guarantee hidden dynamics. In our unclipped controller, two queries at known states satisfy $\hat a s_i+(\hat b+10^{-8})u_i=s^\star$; a nonsingular two-query system recovers both coefficients from actions alone. Decide whether users need uncertainty information, and test what the restricted endpoint still reveals.
• Watermark and monitor. If using a behavioral watermark, validate its detectability, false-positive rate, and robustness to surrogate training or adaptation. Monitor suspicious query patterns, recognizing that extraction need not use a regular grid and that legitimate testing may do so.
• Policy and contracts. Terms of service and licensing may supply remedies or deterrence under applicable law and enforceable agreements. Provenance records may support attribution. None alone establishes technical prevention or a legal violation.

Output restrictions may trade legitimate utility against exposure. Monitoring, access policies and contracts have different costs and need not change prediction accuracy. Define the functionality an attacker is trying to recover, its error tolerance, available queries and acceptable residual exposure; this chapter does not establish a universal extraction-prevention result.