Video Generators as Candidate World Models › Formal Setting: Frames, Actions, and Conditioned Futures

A video generator used as a world model samples futures conditioned on a short history and, when available, a sequence of actions. Write a frame as

$$o_t \in \mathbb{R}^{H \times W \times C},$$

where $H$ and $W$ are pixel dimensions and $C$ is the channel count (three for RGB). This chapter examines a camera-observation setting: the physical state is hidden from the generator, while frames and any recorded controls are available. Other world models may also receive proprioception, maps, or privileged simulator state.

A history at time $t$ is a window of recent frames and recorded actions:

$$h_t = (o_{t-L+1:t},\; a_{t-L+1:t-1}),$$

Here $L$ is the context length, and the displayed full window assumes zero-indexed observations with $t\geq L-1$; for earlier steps, use the available shorter history. The action window ends at $t-1$ because those controls have already been applied. A finite history can omit an earlier event needed to predict the future; a persistent memory state or another sensor may carry information beyond this window. We keep any other condition $c$ separate from $h_t$. It may be text, a goal, or a time-varying prescribed camera trajectory; a camera command changes the observation process and is not automatically a physical action on the scene.

A generator samples a block of future frames

$$\tilde{o}_{t+1:t+K} \sim p_\theta(\tilde{o}_{t+1:t+K} \mid h_t,\; a_{t:t+K-1},\; c),$$

Here $K$ is the prediction horizon, $a_{t:t+K-1}$ is the planned sequence of world-affecting controls, and $c$ holds other conditioning. The action space $\mathcal{A}$ may be discrete (left/right/stay) or continuous (an end-effector command). Timing must be specified somewhere in the conditioning for a timed-response test; a time-ordered action sequence is one explicit way to do so. Permuting the same actions can change intermediate states and sometimes the final state. A model given only an unordered set cannot distinguish the two sequences when their consequences differ.

Even when a generator does accept actions, the conditioning distribution matters. Suppose a model was trained on a fixed behavioral policy $\pi_\beta(a_t \mid o_t)$. The conditional distribution it learns is

$$p(o_{t+1} \mid h_t, a_t) = \frac{p(o_{t+1}, a_t \mid h_t)}{p(a_t \mid h_t)}$$

where $p(a_t\mid h_t)>0$. This identity defines an observational conditional; a fitted $p_\theta$ approximates it only to the extent supported by its data and model fit. If the policy tends to push only when the object is visible, a predictor may use visibility as a shortcut and underuse the action input. For control we want the consequence of setting the action:

$$p\big(o_{t+1} \mid \operatorname{do}(a_t), h_t\big),$$

This is the next-observation distribution when the agent sets $a_t$. The observational and interventional conditionals coincide at a given history and action under consistency, conditional exchangeability (no unmeasured common cause of action and outcome after conditioning on $h_t$), and positivity for that action. Randomized action assignment can help establish exchangeability; dense observed action support alone cannot. At zero support, $p(o_{t+1}\mid h_t,a_t)$ for that action cannot be estimated from those data—it is not thereby proven unequal to the interventional quantity. Hernán and Robins, Causal Inference: What If develops these identification conditions.

Consider a hidden property $U$ that affects both a policy's choice and the object's motion. Even if both actions occur at every observed history, conditioning on the chosen action can reveal information about $U$; setting that action need not have the same effect. Positivity lets us compare actions within supported histories, while exchangeability rules out that hidden-confounding explanation. A nearly deterministic policy makes off-policy effects hard to estimate and may permit a good in-policy predictor to ignore actions. Neither excellent in-policy fit nor inevitable off-policy failure follows without testing the model.

This distinction echoes Part VI, Chapter 3, where we distinguished interaction data from passive observation. It will also frame our counterfactual probes below.