Interaction Data and Passive Observation › Observational Video and Internet Data

Passive observation is data where the transition happened without you choosing it. The canonical example is internet video: someone else moved the world, and your model just watches. Other examples include web navigation traces recorded from real users, financial tick data recorded from markets, and telemetry from aircraft that you did not operate. The defining feature is not the modality, but the absence of your own agency in producing the transitions.

The information content of an unlabeled video differs from an interaction triplet. In the simplified model below, the video provides observations $o_1,o_2,\dots,o_T$ but not latent states, actions, or rewards:

$$\{o_t\}_{t=1}^{T}, \qquad o_t = f(s_t)$$

where:

• $o_t$: the observation (image or frame) at time $t$
• $s_t$: the unobserved latent state at time $t$
• $f$: an unknown deterministic observation map from latent state to pixels; a stochastic sensor would instead require an observation distribution
• $T$: the number of time steps in the video

The latent state $s_t$ evolved under some unobserved sequence of action choices by whoever was acting in the video.

Three structural gaps open up. Without extra assumptions, labels, or grounding data, collecting more samples from the same observation process does not by itself close them.

The transition in a video is $(o_t, ?, o_{t+1})$. The action that caused it is missing. If we want a world model that a robot can query for $P(s' \mid s, a)$, we cannot directly regress from video alone. Either we

• infer the actions via an inverse dynamics model $q(a_t \mid o_t, o_{t+1})$ that predicts what action would explain the observed change, as discussed in Part VI, Ch 2: Actions, Inverse Dynamics, and Latent Actions,
• introduce a latent action $z_t$ and model $p(o_{t+1},z_t\mid o_t)$, recognizing that $z_t$ is not yet an actuator command, or
• give up on explicit actions and learn a video-only dynamics prior that is later fine-tuned with action-labeled interaction data.

Inverse or latent-action modeling is one established strategy for video pretraining, including systems such as Genie. It has two related ambiguities. If distinct real actions produce the same visible transition, an inverse predictor cannot identify which action occurred from those frames alone. When the actions themselves are latent, their coordinates are also identifiable only up to reparameterizations that preserve the predicted transitions. Either ambiguity is acceptable for outcome-oriented generation, but a model that must emit actuator commands needs action-labeled data or another grounding signal that maps latent effects to the agent's controls.

Even when we can observe an action in a video (say the robot arm in the clip is visible), the actor may have chosen it using information absent from the logged observations. The action can then correlate with latent variables such as intent or a private sensor reading. This confounds a causal interpretation unless those variables are measured, the action is conditionally randomized, or another identification strategy applies. The same warning applies to interaction logs when the collection policy uses unrecorded inputs.

This is the confounding problem studied in Part IV, Ch 3: Causal Models and Interventions. Possible responses include measuring a sufficient context, collecting randomized interventions, using a valid instrument when one exists, or limiting the claim to observational prediction. A conditional model $P(o_{t+1}\mid o_t,a_t,c_t)$ transfers only when its recorded context $c_t$ contains the relevant information and the deployment distribution is covered.

Video gives us $o_t$ and never $s_t$. Two different underlying states that look identical in the current frame may evolve very differently, because the visual observation is not Markov. A classic example: two mugs sitting on a table look the same, but one has a chip on the bottom that will make it topple with a gentle tap. The video observation does not distinguish them, but the underlying physics does.

This is the source of the "video world models need memory" intuition. To make a video-only model behave as though it has a Markov state, it can infer latent state from history; the current frame may not be enough. The recurrent state-space models in Part V, Ch 1 and the transformers in Part V, Ch 2 both build representations of history that aim to make latent transitions approximately Markov. History helps only when it contains information about the hidden factor; an unobserved property can remain ambiguous indefinitely.

In spite of these gaps, large passive corpora can offer properties that a single interaction campaign may not match:

• Breadth of scenes. A large, diverse video corpus can cover more objects, textures, lighting conditions, and viewpoints than one robot collection campaign.
• Long tails. Aggregated internet data may contain events that a single agent would observe rarely, although upload and curation bias can distort their apparent frequency.
• Visual priors. A video model can learn features for objects, depth cues, motion, and appearance. Transfer to an interactive domain is an empirical question and depends on the domain gap.
• Semantic grounding. Captions, narrations, and speech can provide weak supervision about intentions and labels that unlabeled interaction logs may lack.

For a model that must predict the consequences of explicit actuator commands, passive video is best treated as a source of representation and dynamics priors rather than a direct substitute for action-labeled transitions. Latent-action video models can still learn controllable effect spaces, but those effects must be grounded before they are interpreted as the agent's commands. We will return to this distinction in Part IX: Foundation and World-Action Models.