Interaction Data and Passive Observation

Imagine two teams racing to build a world model for tabletop manipulation. The first team has a Franka arm in the lab. Over a month, the robot has pushed, picked, and dropped ten thousand objects, logging joint torques, gripper forces, and the resulting object poses. The second team scrapes ten million cooking and crafting videos from the internet. No robot, no actions, and no torque readings, just pixels over time.

Which team ends up with a better world model?

The larger video corpus does not settle which team has the better model; the answer depends on what the model will be asked to do. If the question is "how does the appearance of a carrot change when someone chops it," the video team has the richer evidence. If the question is "what gripper force will lift this mug without slipping," the robot logs contain the relevant commanded forces and outcomes. Passive video alone does not reveal the unchosen gripper forces or their counterfactual effects.

This is the central tension of this chapter. World models are trained on data, and the data comes from two very different sources. Interaction data is generated by an agent whose actions cause the transitions: a robot arm, a game-playing agent, a recommender system that chooses what to show. Passive observation data is generated by someone else's process that the model merely watches: YouTube clips, screen recordings, tick-by-tick financial feeds, satellite imagery. The distinction runs through nearly every design decision in a world-model pipeline, from how the data is stored, to which objectives make sense, to what the trained model is allowed to be trusted to do. It is worth slowing down over this distinction early, because confusion about it resurfaces as concrete bugs and disappointments much later.

The distinction is not about format. A video could in principle be interaction data if the camera is on the agent's own body and we log the commands. A vector of joint torques could be passive observation if it is from a machine we did not control. The distinction is about who chose the actions. If your agent chose them, you observe the joint distribution $p(s, a, s')$ under your behavior. If someone else chose them, you observe $p(s, a, s')$ under their behavior, and the two distributions can diverge sharply. The same file format, the same sensor, or the same channel of data can sit on either side of the line; what matters is the provenance of the action signal.

This matters because a model optimized for prediction may later be queried by a planner. Prediction objectives average error over a data distribution, whereas planning selects actions using particular model predictions. If training data place no mass near those queries, the transition behavior is not identified there without structural assumptions. As we will see in Part VII: Planning and Agency, a small average prediction error can coexist with a large error in the region the planner selects.

There are three interlocking questions this chapter answers.

1. What does interaction data give you that passive data cannot? Interventional signal: the effect of actions you control, which underpins causal claims and counterfactual reasoning. We introduced the causal view in Part IV: Space, Physics, and Causality; here we make it quantitative.
2. What can passive data provide at a different scale? Breadth and diversity. A large video corpus may include more objects, scenes, and unusual events than one interaction campaign, although collection and curation bias still determine which long-tail events appear.
3. How do you combine them? One useful pattern is to pretrain representations and dynamics priors on passive data, then use action-labeled interaction data to ground the model in the agent's controls. Some systems train these objectives jointly instead. Either design still has to measure whether the resulting model helps closed-loop control rather than merely improving prediction loss.

The rest of the chapter develops the vocabulary needed to reason about these three questions. We will formalize the interaction process, examine the missing state, action, and selection information that passive observations can hide, define the coverage and behavior-policy concepts that determine what a model can learn, and end with practical recipes for combining sources. Along the way we build a small numerical experiment that makes the coverage argument concrete. The goal is not to pick a winner between the two data types, but to determine which source supports a given downstream question and which conclusions require additional assumptions.

A useful frame is that each data source naturally answers a different family of questions. Appropriately controlled interaction data can answer what happens when you do this; passive data can answer what tends to happen in the observed process. The first is a causal, agent-centered question under identification assumptions. The second is a descriptive, observer-centered question. A decision-making model may benefit from both, but action-conditioned control requires some way to ground predictions in the agent's available controls.

Action-conditioned control models are commonly trained or grounded on trajectories produced by an agent, operator, or simulator. We represent an interactive environment as a Markov decision process $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $P(s' \mid s, a)$ is the transition kernel, $R(s, a)$ is the reward function, and $\gamma$ is the discount factor, as formalized in Part II: Inference, Decisions, and Control. An agent trajectory is a sequence

where:

• $\tau$: a trajectory, i.e., a sequence of states, actions, and rewards from time $0$ to terminal time $T$
• $s_t$: the state observed at time $t$
• $a_t$: the action selected at time $t$
• $r_t$: the reward received after taking action $a_t$ in state $s_t$
• $T$: the terminal time index of the trajectory

sampled by an iterative loop: observe state $s_t$, choose action $a_t \sim \beta(\cdot \mid s_t)$, receive reward $r_t$ and next state $s_{t+1} \sim P(\cdot \mid s_t, a_t)$. A dataset $\mathcal{D}$ of trajectories is a collection of such sequences. The loop is the same primitive that appears throughout the book, from filtering and control in Part II to planning in Part VII. What changes between chapters is not the loop itself but the assumptions we make about the data it produces and the questions we ask of the resulting model.

Three distinct stochastic processes produce a trajectory dataset, and conflating them is a common source of confusion:

• The environment transition kernel $P(s' \mid s, a)$: the physical or logical dynamics we want the world model to learn.
• The behavior policy $\beta(a \mid s)$: the rule used to choose actions during data collection. This may be a heuristic, a random policy, a human operator, or a previous learned agent.
• The initial-state distribution $\rho(s_0)$: where episodes begin.

A world model trained on $\mathcal{D}$ is learning to approximate $P$ under the marginal state-action distribution $d^{\beta}(s, a)$ induced by $\beta$ and $\rho$. Outside that support, internally consistent predictions can still be wrong. The environment kernel describes the dynamics; the behavior policy and initial-state distribution describe how we chose to look at those dynamics. All three matter when we assess what the dataset can identify.

Interaction data records which command was applied before each transition $(s_t, a_t, s_{t+1})$. Under an important condition, this can supply interventional evidence: the logged state or history must include the information used to choose the action, and the action-selection randomization must be independent of unobserved transition disturbances after conditioning on that information. The behavior policy is still state-dependent, since $a_t \sim \beta(\cdot\mid s_t)$; "controlled" does not mean "independent of state." If an operator uses an unlogged sensor or private intent, interaction logs can be confounded in the same way as observational data. With the condition satisfied, varying $a_t$ lets us estimate how the agent's available controls change the next-state distribution rather than relying only on observed correlations.

Three concrete consequences:

• Counterfactual reasoning. Given an action-grounded model of $P(s' \mid s,a)$ learned under the stated no-unobserved-confounding condition, the agent can compare candidate commands. Observational actions can support the same conditional effect only when an identification assumption such as conditional ignorability is justified.
• Targeted exploration. The agent controls the data it will collect. As discussed in Part VII, Ch 5: Active Perception, Dual Control, and Exploration, it can deliberately choose trajectories that reduce model uncertainty in regions the planner will visit.
• Reward and value signal. A collection system can record or construct rewards alongside interaction trajectories, enabling control-oriented objectives. The action log alone does not guarantee that reward labels are present.

Interaction can be expensive: real robots wear out, human operators require time, and simulators require engineering. The behavior policy shapes the resulting distribution. An uncertainty-seeking explorer may emphasize novel regions, while a deployed policy may spend more time in routine states. That mismatch is not inevitable, but it must be measured rather than inferred from the label "exploration data."

Offline model-based RL uses a fixed trajectory dataset, while off-policy online methods may use a growing replay buffer. Both store transitions of the form

where:

• $\mathcal{D}$: the trajectory dataset containing $N$ transitions
• $i$: the transition index
• $t$: the time index associated with transition $i$
• $N$: the number of transitions in the dataset
• $s_t^{(i)}$: the state at time $t$ for transition $i$
• $a_t^{(i)}$: the action applied in that state
• $r_t^{(i)}$: the reward received
• $s_{t+1}^{(i)}$: the resulting next state
• $\mathrm{done}_t^{(i)} \in \{0, 1\}$: a terminal indicator equal to 1 when the transition ends an episode and 0 otherwise

For a one-step supervised objective, implementations often flatten trajectories into transition records and shuffle them into minibatches. This changes how the optimizer samples the replay buffer, but it does not make transitions independent: adjacent records were produced by the same Markov chain. Stationarity keeps the marginal distribution fixed, while independence is a separate assumption. Temporal mixing and episode structure therefore affect the effective sample size even when the loss is written as an average over individual transitions.

These assumptions are worth stating explicitly because they are frequently implicit in the training code.

• Markov. $s_{t+1}$ depends on the full history only through $(s_t, a_t)$. If the observed state is only a proxy for the true state, the correct model is over a belief state, as discussed in Part II, Ch 2: Bayesian Filtering and Belief States.
• Stationary. The transition kernel does not change over time or episodes.
• Deployment coverage. The behavior-induced occupancy places enough probability near the state-action pairs the planner will query. Ergodicity of the behavior-induced Markov chain alone does not guarantee this target-relative coverage.

These are separate checks, and each can fail on its own. A process can be stationary but poorly covered, or well covered for a target while its observation is non-Markov. Naming the checks separately makes deployment failures easier to diagnose.

Interaction datasets come in two practically distinct forms, and they impose different training protocols:

• On-policy data is generated by the policy being evaluated or updated. Many policy-gradient and actor-critic procedures collect such batches. The data can support multiple optimization epochs, but after the policy changes it is no longer exactly on-policy; excessive reuse can increase the mismatch between the sampling and updated policies.
• Off-policy data is generated by a different behavior policy, often an older version of the same agent, a hand-designed explorer, or a heterogeneous pool of policies. It can be retained and reused, although stale or low-quality transitions may still be filtered. Its behavior mixture determines a coverage distribution that must be measured rather than assumed.

The distinction between the two flavors concerns whether the data-generating policy matches the policy currently being evaluated. On-policy collection reduces that immediate mismatch but requires new collection as the policy changes. Off-policy collection permits reuse, but training and evaluation must account for where the behavior distribution places mass.

The Data Curation and Synthetic Data chapter (Part VI, Ch 4) will discuss how to filter, weight, and curate these datasets at scale. For now, the key point is that "I have interaction data" is not a specificity claim: what actions, in what states, under what behavior?

Passive observation is data where the transition happened without you choosing it. The canonical example is internet video: someone else moved the world, and your model just watches. Other examples include web navigation traces recorded from real users, financial tick data recorded from markets, and telemetry from aircraft that you did not operate. The defining feature is not the modality, but the absence of your own agency in producing the transitions.

The information content of an unlabeled video differs from an interaction triplet. In the simplified model below, the video provides observations $o_1,o_2,\dots,o_T$ but not latent states, actions, or rewards:

where:

• $o_t$: the observation (image or frame) at time $t$
• $s_t$: the unobserved latent state at time $t$
• $f$: an unknown deterministic observation map from latent state to pixels; a stochastic sensor would instead require an observation distribution
• $T$: the number of time steps in the video

The latent state $s_t$ evolved under some unobserved sequence of action choices by whoever was acting in the video.

Three structural gaps open up. Without extra assumptions, labels, or grounding data, collecting more samples from the same observation process does not by itself close them.

The transition in a video is $(o_t, ?, o_{t+1})$. The action that caused it is missing. If we want a world model that a robot can query for $P(s' \mid s, a)$, we cannot directly regress from video alone. Either we

• infer the actions via an inverse dynamics model $q(a_t \mid o_t, o_{t+1})$ that predicts what action would explain the observed change, as discussed in Part VI, Ch 2: Actions, Inverse Dynamics, and Latent Actions,
• introduce a latent action $z_t$ and model $p(o_{t+1},z_t\mid o_t)$, recognizing that $z_t$ is not yet an actuator command, or
• give up on explicit actions and learn a video-only dynamics prior that is later fine-tuned with action-labeled interaction data.

Inverse or latent-action modeling is one established strategy for video pretraining, including systems such as Genie. It has two related ambiguities. If distinct real actions produce the same visible transition, an inverse predictor cannot identify which action occurred from those frames alone. When the actions themselves are latent, their coordinates are also identifiable only up to reparameterizations that preserve the predicted transitions. Either ambiguity is acceptable for outcome-oriented generation, but a model that must emit actuator commands needs action-labeled data or another grounding signal that maps latent effects to the agent's controls.

Even when we can observe an action in a video (say the robot arm in the clip is visible), the actor may have chosen it using information absent from the logged observations. The action can then correlate with latent variables such as intent or a private sensor reading. This confounds a causal interpretation unless those variables are measured, the action is conditionally randomized, or another identification strategy applies. The same warning applies to interaction logs when the collection policy uses unrecorded inputs.

This is the confounding problem studied in Part IV, Ch 3: Causal Models and Interventions. Possible responses include measuring a sufficient context, collecting randomized interventions, using a valid instrument when one exists, or limiting the claim to observational prediction. A conditional model $P(o_{t+1}\mid o_t,a_t,c_t)$ transfers only when its recorded context $c_t$ contains the relevant information and the deployment distribution is covered.

Video gives us $o_t$ and never $s_t$. Two different underlying states that look identical in the current frame may evolve very differently, because the visual observation is not Markov. A classic example: two mugs sitting on a table look the same, but one has a chip on the bottom that will make it topple with a gentle tap. The video observation does not distinguish them, but the underlying physics does.

This is the source of the "video world models need memory" intuition. To make a video-only model behave as though it has a Markov state, it can infer latent state from history; the current frame may not be enough. The recurrent state-space models in Part V, Ch 1 and the transformers in Part V, Ch 2 both build representations of history that aim to make latent transitions approximately Markov. History helps only when it contains information about the hidden factor; an unobserved property can remain ambiguous indefinitely.

In spite of these gaps, large passive corpora can offer properties that a single interaction campaign may not match:

• Breadth of scenes. A large, diverse video corpus can cover more objects, textures, lighting conditions, and viewpoints than one robot collection campaign.
• Long tails. Aggregated internet data may contain events that a single agent would observe rarely, although upload and curation bias can distort their apparent frequency.
• Visual priors. A video model can learn features for objects, depth cues, motion, and appearance. Transfer to an interactive domain is an empirical question and depends on the domain gap.
• Semantic grounding. Captions, narrations, and speech can provide weak supervision about intentions and labels that unlabeled interaction logs may lack.

For a model that must predict the consequences of explicit actuator commands, passive video is best treated as a source of representation and dynamics priors rather than a direct substitute for action-labeled transitions. Latent-action video models can still learn controllable effect spaces, but those effects must be grounded before they are interpreted as the agent's commands. We will return to this distinction in Part IX: Foundation and World-Action Models.

Dataset size alone does not say whether the state-action pairs needed at deployment are represented. Two equal-sized datasets can therefore support very different target policies. Coverage formalizes this target-relative distinction.

Coverage connects the behavior policy and exploration strategy to world-model generalization.

For a chosen time index, or for an explicitly defined time-averaged occupancy, the data distribution factorizes as

where:

• $d^{\beta}(s,a)$: the state-action occupancy distribution under the behavior policy $\beta$
• $d^{\beta}(s,a,s')$: the joint distribution of state, action, and next state under the behavior policy $\beta$
• $d^{\beta}(s)$: the state-occupancy distribution induced jointly by the initial-state distribution $\rho(s_0)$, the behavior policy $\beta$, and repeated transitions through $P$
• $\beta(a \mid s)$: the conditional probability or density of selecting action $a$ in state $s$
• $P(s' \mid s,a)$: the environment transition kernel
• $\rho(s_0)$: the initial-state distribution over starting states

The initial-state distribution is therefore not generally the same object as $d^{\beta}(s)$. It influences later state occupancy through the behavior policy and the environment dynamics. With the environment fixed, data collection can change $d^{\beta}(s,a)$ through both $\rho$ and $\beta$. Without additional structural assumptions, transition behavior remains unidentified wherever the resulting occupancy assigns no probability.

The factorization has an important implication: coverage can be improved in two different ways, and they target different gaps. Changing $\rho$ moves where the agent starts, which can flood previously unvisited regions with data even if the behavior policy is unchanged. Changing $\beta$ moves where the agent goes once it starts, which can fill in the actions and transitions reachable from any start. A careful data-collection design usually combines both, because starting broadly without a policy that spans the action space still leaves action-side coverage gaps.

Concretely: if the behavior policy is "always push with 1 N of force," no amount of that data identifies how the object behaves at 10 N without structural assumptions. If the behavior policy is "grasp the same red mug," predictions for other mugs depend on generalization beyond the observed object distribution. Causal confusion in imitation learning is related but distinct: a learner can select a predictive correlate of the expert's action rather than its cause, and the resulting policy fails after its own actions shift the observation distribution. Targeted interventions can distinguish the competing causal explanations; merely increasing sample count under the same correlations may not.

Formally, an offline model-based RL setting has a fixed dataset $\mathcal{D}$ sampled from $d^{\beta}$ and cannot collect more during offline training. The world model is trained on $\mathcal{D}$, then used inside a planner that evaluates candidate target policies. The target policy's state-action distribution $d^{\pi}$ is not generally the training distribution. Coverage is therefore one necessary part of assessing accuracy at planner queries, alongside sample size, model specification, and optimization error.

This leads to a clean diagnostic:

• In-support planning. The planner queries $(s, a)$ inside $\operatorname{supp}(d^{\beta})$, the support of the behavior-policy state-action distribution. This avoids pure support extrapolation, although finite data, partial observability, and model misspecification can still make predictions inaccurate.
• Near-support planning. The planner queries $(s, a)$ just outside $\operatorname{supp}(d^{\beta})$. Error may grow with distance from the data, but smooth growth is an inductive assumption rather than a guarantee. Any error can compound over a rollout horizon.
• Far-support planning. The planner queries $(s, a)$ far outside $\operatorname{supp}(d^{\beta})$. The data do not constrain those predictions. They may be wrong yet confident, giving the planner an opportunity to exploit model error.

The pattern is so common that modern offline MBRL often penalizes the planner for querying off-support actions, or ensembles several models and treats disagreement as a coverage signal. We'll pick this up in Part VIII: Decision-Centric Research Lineages, where ensembles and pessimistic planning are central topics. Pessimism, in this reading, is a way of encoding coverage as a preference: when the model is uncertain, prefer the plan that stays closer to the data.

Coverage can create a chicken-and-egg problem: the planner wants to visit states the dataset never saw, but the behavior policy did not collect them. One response is intervention: actively choose actions that populate the missing region. Here intervention is a data-collection strategy, using controlled action choices to distinguish outcomes that observational correlations leave ambiguous.

Three canonical strategies, revisited in Part VII, Ch 5: Active Perception, Dual Control, and Exploration, are:

• Random or entropy-maximizing exploration. For a bounded action space and a specified base measure, a uniform policy maximizes one-step action entropy. It does not guarantee uniform state coverage because the dynamics determine which states those actions reach.
• Uncertainty-driven exploration. Use a current model's uncertainty proxy, such as ensemble disagreement or estimated information gain, to choose actions. The strategy helps only when the proxy tracks epistemic error; correlated models can agree and still be wrong.
• Goal- or task-conditioned exploration. Deliberately probe regions relevant to a known task family. This concentrates coverage where it is expected to matter and leaves other regions sparse, which is acceptable only if the deployment family remains within scope.

Each strategy tries to improve coverage within a region of interest but can leave other regions sparse. Coverage is always relative to a target distribution. A random explorer may leave the task-relevant corner barely populated, while a task-conditioned explorer may miss rare events. Which failure matters more depends on the downstream queries.

The sources can be combined rather than treated as mutually exclusive. Passive data may supply broader visual and temporal variation, while appropriately logged interaction data supplies grounding in the agent's action space. Whether the combination helps depends on domain alignment, objective weighting, and downstream evaluation; neither source automatically repairs the other's weaknesses.

A common staged design is:

1. Stage 1: Video or passive pretraining. Train an encoder and a latent dynamics model on a large corpus of passive video or other observational data. Objectives can include reconstruction, contrastive learning, masked prediction, or sequence generation, as discussed in Part VI, Ch 1: Predictive and Generative Objectives. Action labels are not required for these representation or observation-prediction losses.
2. Stage 2: Action-conditioned fine-tuning. Freeze or partially unfreeze the encoder, add an action-conditioned dynamics head, and train on interaction data. The action head learns to map actions to effects; the encoder has already been shaped by video.

Passive data can teach the model what the world looks like and which changes commonly follow one another; interaction data then ties changes to the agent's recorded controls. The two objectives can complement rather than replace each other. One useful mental picture is that the encoder learns visual and temporal primitives from video, while the action-conditioned model learns how the agent's commands alter their evolution.

Another design trains both sources simultaneously with a shared backbone:

• A single latent encoder $q_\phi(z_t \mid h_t)$, parameterized by $\phi$, that summarizes history $h_t$ into a latent state $z_t$.
• A video head that predicts future latents from past latents plus a learned latent-action code inferred by inverse dynamics (no explicit action labels).
• An action head that predicts the next latent given the current latent and a commanded action $a_t$.

The two heads share the encoder, so both objectives update the representation used with actions. This can transfer useful features, but gradients can also conflict or let one source dominate. Later in Part VI, Ch 6: Pretraining, Post-Training, Scaling, and Adaptation, we will discuss how this joint regime scales and how the data mixture can be selected. Joint and sequential training should be compared under the same compute budget rather than assuming either is uniformly better.

When you have both sources, three questions must be answered.

• How to weight the losses. Video and interaction objectives can have different scales and gradient behavior. A staged warmup, normalized losses, or adaptive gradient balancing are possible choices. The coefficient is a validation parameter, not a portable numerical default.
• How to align representations. If the video encoder was trained on pixel data with 3-channel RGB and the interaction data has depth, proprioception, or force, project each modality into the same latent and use cross-modal contrastive losses to tie them together. We saw the multimodal machinery in Part III: Representing Agents and Worlds.
• Which actions to trust. Inferred actions can be ambiguous or noisy. Calibrated confidence can weight examples directly; robust losses can reduce the influence of large residuals, but they are not substitutes for uncertainty estimates. An ensemble is useful only if its disagreement is validated against held-out action labels or another grounding signal.

These questions determine how much each source shapes the final model. Over-weighting video risks weak action grounding; over-weighting a narrow interaction dataset risks losing useful visual breadth. The mixing coefficients should be selected for the deployment context rather than fixed in advance.

Passive pretraining is a strong prior when it is aligned: the pretrained dynamics should be close to the interaction dynamics. Two failure modes are common:

• Visual domain gap. Human cooking video differs from robot-camera data in viewpoint, embodiment, and dynamics. Negative transfer is possible when pretrained features emphasize the wrong invariances. Freezing, partial unfreezing, and full fine-tuning are alternatives to compare on target-domain validation data; a small learning rate is not a general guarantee.
• Action gap. Even when visual features transfer, a latent action has no intrinsic mapping to a robot's continuous torque vector. Action-labeled interaction data must identify a grounding map, whether it is initialized from scratch or adapted from another embodiment.

A practical hypothesis is that transfer should improve as the pretraining and target domains align in observations, dynamics, and relevant objects. Web-navigation traces are closer to a browser agent than generic video, and manipulation footage with matching objects and viewpoints is closer to a robot task than unrelated footage. This is a validation hypothesis, not a guarantee.

The two sources have a useful structural asymmetry.

• Appropriately controlled interaction data can be causally informative but narrow. It covers a particular action space and behavior-induced state distribution.
• Passive data can be broad but is ordinarily correlational. It may contain rich observations of a large state distribution without interventional evidence for your specific actions.

A world model used for control must reconcile these sources. The loss mixture, shared parameters, and fine-tuning schedule determine how strongly the interaction-derived action model can override passive-data priors. Broad visual coverage does not imply reliable control, and narrow interaction coverage does not imply trustworthy predictions even on support. The distinction has to be tested rather than inferred from the training recipe.

Let's make the coverage argument concrete with a tiny one-dimensional controlled system. We'll collect interaction data under a biased behavior policy and fit a deliberately misspecified linear dynamics model. The true system has a mild nonlinearity that is activated mainly where the behavior policy rarely goes. This makes the difference between interpolation and extrapolation visible. Then we'll demonstrate importance weighting as a partial remedy.

The environment has nonlinear controlled dynamics with additive Gaussian noise:

where:

• $s_t \in \mathbb{R}$: the scalar state at time $t$
• $a_t \in [-1, 1]$: the scalar action applied at time $t$
• $[a_t]_+ = \max(a_t, 0)$: the positive part of the action
• $\epsilon_t \sim \mathcal{N}(0, 0.05^2)$: independent zero-mean Gaussian process noise
• $0.9$: the state persistence coefficient
• $0.5$: the linear action coefficient
• $0.35$: the positive-action curvature coefficient

Negative actions follow a linear response; positive actions add a quadratic effect that a linear fitted model cannot represent exactly.

In this seeded experiment, the largest realized errors occur in the poorly covered positive-action region, where the fitted linear model cannot represent the true curvature. The experiment does not prove that sparse coverage always causes high error. It shows that sparse coverage can hide misspecification and leave off-support accuracy weakly tested. The estimator can appear well-behaved on the observed data while remaining unreliable in the region the target policy will visit.

If the target policy is known and its conditional action distribution overlaps the behavior policy's support everywhere, action importance weighting can estimate a target-action loss while holding the behavior state occupancy fixed:

where:

• $d^{\beta}(s)$: the state-occupancy distribution under the behavior policy
• $\pi(a\mid s)$: the target policy's conditional action distribution
• $\beta(a\mid s)$: the behavior policy's conditional action distribution
• $P(\cdot\mid s,a)$: the environment transition kernel
• $\ell(s,a,s')$: the transition loss
• $\pi(a\mid s)/\beta(a\mid s)$: the action importance weight

The equality requires $\beta(a\mid s)>0$ wherever $\pi(a\mid s)>0$, up to sets of measure zero, for states with positive $d^{\beta}(s)$. It does not correct the separate shift from $d^{\beta}(s)$ to $d^{\pi}(s)$. Correcting the full target-policy occupancy would require the joint ratio $d^{\pi}(s,a)/d^{\beta}(s,a)$, which is generally unknown.

The figure answers one narrow question: one-step squared error under a target action distribution while holding the behavior-state occupancy fixed. A real evaluation should keep four measurements separate:

• held-out one-step error at target-relevant state-action pairs;
• open-loop rollout error as predictions feed back over increasing horizons;
• coverage and uncertainty diagnostics, including whether uncertainty rises on deliberately shifted inputs; and
• closed-loop task performance and constraint violations when a planner uses the model.

Good one-step MSE does not imply good planning. A planner can repeatedly query a small biased region, exploit a localized model error, or amplify a small systematic error over a rollout. Conversely, a visually imperfect predictor can still preserve the task-relevant quantities needed for action selection.

The distinction between interaction and passive data is more than a taxonomy. It affects which questions the resulting world model can be trusted to answer and how the model should be trained, evaluated, and deployed.

The clearest limitation is the one the worked example exposes: without additional structural assumptions, no weighting scheme identifies transition behavior where the behavior distribution has zero support. Importance weighting is unbiased for its stated target only under absolute continuity and finite expectations. Even when support overlaps, density ratios can have very high or infinite variance in high-dimensional state-action spaces, making finite-sample estimates unstable. This is why many offline methods treat coverage as a planning-side concern and discourage the planner from querying the model far off support. This theme runs through Part VIII: Decision-Centric Research Lineages and Part XII: Reliable World Models.

A second limitation is hidden selection. A model trained on internet video learns from clips that were recorded, uploaded, edited, retained, and sampled by the collection pipeline. That distribution need not match the deployment environment. Standard in-distribution prediction losses may not expose this mismatch, although explicit provenance audits, held-out deployment data, and out-of-distribution diagnostics can reveal parts of it. Passive data therefore carries a selection process that is harder to inspect than an explicitly logged behavior policy.

Recent systems provide several versions of this combination: action-free video pretraining followed by control grounding, mixed action-labeled and unlabeled training, and latent-action models such as Genie. These examples establish that the sources can complement each other, not that transfer or closed-loop control will succeed automatically. The required amount of interaction data remains an empirical property of the target embodiment, task, and pretraining distribution.

A third limitation is that interaction data is not cheap just because it is "yours." Beyond the physical cost of running a robot or a user study, a pipeline may require reward design, failure labels, episode boundaries, and quality control. Which cost dominates depends on how much of this information the environment records automatically. The next chapter on Data Curation and Synthetic Data (Part VI, Ch 4) picks up this thread and discusses filtering, relabeling, augmentation, and provenance.

A fourth, more speculative limitation is that neither kind of data tells the model what it is for. A world model trained on the richest possible corpus still needs an external signal about which trade-offs matter: whether to prefer rare-event accuracy over common-case smoothness, whether to be conservative or aggressive in planning, whether to trust video priors or interaction evidence when they disagree. These are value-laden choices, and they sit outside the data itself. Treating them as engineering details rather than design decisions is one of the ways large world models quietly acquire biases that are not reflected in any of their training metrics.

The central tradeoff is conditional rather than absolute. Properly logged or randomized interaction data can identify the effects of the agent's controls, but its coverage may be narrow and collection expensive. Passive data can be broad, yet it may omit actions and inherit confounding and selection bias. A staged or joint training recipe is a hypothesis about how to combine those signals. Held-out prediction, rollout, uncertainty, and closed-loop tests must decide whether it worked.

• Interaction data records the agent's applied commands. It supports causal and counterfactual queries when the logging and action-selection assumptions rule out unobserved confounding.
• Passive observation data (video, logs, scraped signals) may lack actions and may contain confounded behavior, so it cannot generally identify $P(s' \mid s, a)$ on its own. It is useful for representation and dynamics priors, but interaction evidence is needed to identify a controllable transition function without stronger assumptions.
• The behavior policy, initial-state distribution, and dynamics determine coverage. Missing support leaves transition behavior unidentified without further assumptions; good coverage is necessary but not sufficient for reliable planning.
• Behavior policies shape observed data distributions. Action importance weighting targets a changed conditional action distribution only under absolute continuity and finite expectations, and it does not correct state-occupancy shift.
• Interventions can expand coverage through random, uncertainty-driven, or task-conditioned exploration, but action diversity alone does not guarantee state coverage.
• Passive pretraining and action-labeled fine-tuning can complement each other, as can joint objectives, but their benefit depends on domain alignment and downstream validation.
• Data support constrains what the training observations identify. Model class, finite-sample error, partial observability, uncertainty calibration, and rollout compounding still matter inside that support.

The key parameters in the worked example are:

• A_TRUE: State persistence coefficient (0.9). Determines how much current state carries forward.
• B_TRUE: Linear action coefficient (0.5). Determines the direct effect of the action.
• CURVATURE_TRUE: Positive-action curvature coefficient (0.35). Adds nonlinearity for positive actions.
• NOISE_STD: Process noise standard deviation (0.05). Controls stochasticity of transitions.
• n_episodes: Number of trajectories collected (200). More episodes increase coverage but cost more interaction.
• horizon: Steps per episode (20). It controls the number of correlated transitions collected per episode; the example evaluates one-step error rather than rollout compounding.
• behavior_biased: Behavior policy that determines the collected state-action distribution. The held-out target actions are sampled uniformly and paired with resampled behavior states.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about interaction data and passive observation.