Interaction Data and Passive Observation › Combining Passive and Embodied Experience

The sources can be combined rather than treated as mutually exclusive. Passive data may supply broader visual and temporal variation, while appropriately logged interaction data supplies grounding in the agent's action space. Whether the combination helps depends on domain alignment, objective weighting, and downstream evaluation; neither source automatically repairs the other's weaknesses.

A common staged design is:

1. Stage 1: Video or passive pretraining. Train an encoder and a latent dynamics model on a large corpus of passive video or other observational data. Objectives can include reconstruction, contrastive learning, masked prediction, or sequence generation, as discussed in Part VI, Ch 1: Predictive and Generative Objectives. Action labels are not required for these representation or observation-prediction losses.
2. Stage 2: Action-conditioned fine-tuning. Freeze or partially unfreeze the encoder, add an action-conditioned dynamics head, and train on interaction data. The action head learns to map actions to effects; the encoder has already been shaped by video.

Passive data can teach the model what the world looks like and which changes commonly follow one another; interaction data then ties changes to the agent's recorded controls. The two objectives can complement rather than replace each other. One useful mental picture is that the encoder learns visual and temporal primitives from video, while the action-conditioned model learns how the agent's commands alter their evolution.

Another design trains both sources simultaneously with a shared backbone:

• A single latent encoder $q_\phi(z_t \mid h_t)$, parameterized by $\phi$, that summarizes history $h_t$ into a latent state $z_t$.
• A video head that predicts future latents from past latents plus a learned latent-action code inferred by inverse dynamics (no explicit action labels).
• An action head that predicts the next latent given the current latent and a commanded action $a_t$.

The two heads share the encoder, so both objectives update the representation used with actions. This can transfer useful features, but gradients can also conflict or let one source dominate. Later in Part VI, Ch 6: Pretraining, Post-Training, Scaling, and Adaptation, we will discuss how this joint regime scales and how the data mixture can be selected. Joint and sequential training should be compared under the same compute budget rather than assuming either is uniformly better.

When you have both sources, three questions must be answered.

• How to weight the losses. Video and interaction objectives can have different scales and gradient behavior. A staged warmup, normalized losses, or adaptive gradient balancing are possible choices. The coefficient is a validation parameter, not a portable numerical default.
• How to align representations. If the video encoder was trained on pixel data with 3-channel RGB and the interaction data has depth, proprioception, or force, project each modality into the same latent and use cross-modal contrastive losses to tie them together. We saw the multimodal machinery in Part III: Representing Agents and Worlds.
• Which actions to trust. Inferred actions can be ambiguous or noisy. Calibrated confidence can weight examples directly; robust losses can reduce the influence of large residuals, but they are not substitutes for uncertainty estimates. An ensemble is useful only if its disagreement is validated against held-out action labels or another grounding signal.

These questions determine how much each source shapes the final model. Over-weighting video risks weak action grounding; over-weighting a narrow interaction dataset risks losing useful visual breadth. The mixing coefficients should be selected for the deployment context rather than fixed in advance.

Passive pretraining is a strong prior when it is aligned: the pretrained dynamics should be close to the interaction dynamics. Two failure modes are common:

• Visual domain gap. Human cooking video differs from robot-camera data in viewpoint, embodiment, and dynamics. Negative transfer is possible when pretrained features emphasize the wrong invariances. Freezing, partial unfreezing, and full fine-tuning are alternatives to compare on target-domain validation data; a small learning rate is not a general guarantee.
• Action gap. Even when visual features transfer, a latent action has no intrinsic mapping to a robot's continuous torque vector. Action-labeled interaction data must identify a grounding map, whether it is initialized from scratch or adapted from another embodiment.

A practical hypothesis is that transfer should improve as the pretraining and target domains align in observations, dynamics, and relevant objects. Web-navigation traces are closer to a browser agent than generic video, and manipulation footage with matching objects and viewpoints is closer to a robot task than unrelated footage. This is a validation hypothesis, not a guarantee.

The two sources have a useful structural asymmetry.

• Appropriately controlled interaction data can be causally informative but narrow. It covers a particular action space and behavior-induced state distribution.
• Passive data can be broad but is ordinarily correlational. It may contain rich observations of a large state distribution without interventional evidence for your specific actions.

A world model used for control must reconcile these sources. The loss mixture, shared parameters, and fine-tuning schedule determine how strongly the interaction-derived action model can override passive-data priors. Broad visual coverage does not imply reliable control, and narrow interaction coverage does not imply trustworthy predictions even on support. The distinction has to be tested rather than inferred from the training recipe.