Interaction Data and Passive Observation › Limitations and Impact

The distinction between interaction and passive data is more than a taxonomy. It affects which questions the resulting world model can be trusted to answer and how the model should be trained, evaluated, and deployed.

The clearest limitation is the one the worked example exposes: without additional structural assumptions, no weighting scheme identifies transition behavior where the behavior distribution has zero support. Importance weighting is unbiased for its stated target only under absolute continuity and finite expectations. Even when support overlaps, density ratios can have very high or infinite variance in high-dimensional state-action spaces, making finite-sample estimates unstable. This is why many offline methods treat coverage as a planning-side concern and discourage the planner from querying the model far off support. This theme runs through Part VIII: Decision-Centric Research Lineages and Part XII: Reliable World Models.

A second limitation is hidden selection. A model trained on internet video learns from clips that were recorded, uploaded, edited, retained, and sampled by the collection pipeline. That distribution need not match the deployment environment. Standard in-distribution prediction losses may not expose this mismatch, although explicit provenance audits, held-out deployment data, and out-of-distribution diagnostics can reveal parts of it. Passive data therefore carries a selection process that is harder to inspect than an explicitly logged behavior policy.

Recent systems provide several versions of this combination: action-free video pretraining followed by control grounding, mixed action-labeled and unlabeled training, and latent-action models such as Genie. These examples establish that the sources can complement each other, not that transfer or closed-loop control will succeed automatically. The required amount of interaction data remains an empirical property of the target embodiment, task, and pretraining distribution.

A third limitation is that interaction data is not cheap just because it is "yours." Beyond the physical cost of running a robot or a user study, a pipeline may require reward design, failure labels, episode boundaries, and quality control. Which cost dominates depends on how much of this information the environment records automatically. The next chapter on Data Curation and Synthetic Data (Part VI, Ch 4) picks up this thread and discusses filtering, relabeling, augmentation, and provenance.

A fourth, more speculative limitation is that neither kind of data tells the model what it is for. A world model trained on the richest possible corpus still needs an external signal about which trade-offs matter: whether to prefer rare-event accuracy over common-case smoothness, whether to be conservative or aggressive in planning, whether to trust video priors or interaction evidence when they disagree. These are value-laden choices, and they sit outside the data itself. Treating them as engineering details rather than design decisions is one of the ways large world models quietly acquire biases that are not reflected in any of their training metrics.

The central tradeoff is conditional rather than absolute. Properly logged or randomized interaction data can identify the effects of the agent's controls, but its coverage may be narrow and collection expensive. Passive data can be broad, yet it may omit actions and inherit confounding and selection bias. A staged or joint training recipe is a hypothesis about how to combine those signals. Held-out prediction, rollout, uncertainty, and closed-loop tests must decide whether it worked.