Video Generators as Candidate World Models › A Planning Interface Over Generated Futures

Let us now make the connection to decisions explicit. Suppose a generator produces action-conditioned videos. A receding-horizon planner over generated futures solves, at each step $t$,

$$A_t^* \in \arg\min_{A \in \mathcal{A}^H} \;\mathbb{E}_{\tilde{o}_{t+1:t+H} \sim p_\theta(\cdot\mid h_t,A,c)}\!\left[\sum_{j=1}^{H} \gamma^{j-1} J(\tilde{o}_{t+j}, a_{t+j-1})\right] \;+\; \lambda \, \mathcal{U}(A\mid h_t,c),$$

where $A=(a_t,\ldots,a_{t+H-1})$, $H\geq1$ is the horizon, $\gamma\in(0,1]$ discounts later costs, and $J$ scores a predicted next observation and its preceding action. The weight $\lambda\geq0$ multiplies a nonnegative estimated uncertainty penalty $\mathcal{U}(A\mid h_t,c)$. Define $g(o,a)\leq0$ as satisfying a specified safety rule. A safety filter might reject $A$ when $\Pr_{p_\theta}(\exists j\in\{1,\ldots,H\}:g(\tilde{o}_{t+j},a_{t+j-1})>0\mid h_t,A,c)>\delta$ for a chosen $\delta\in[0,1]$. This is a model-estimated chance constraint, not a guarantee of physical safety. Only the first action $a_t^*$ of the chosen sequence is executed; the planner then observes the true next frame and replans.

The summation scores a candidate action sequence by expected discounted cost. The uncertainty term can discourage actions with poorly supported predictions, but only if that estimate is informative about model error. Sample-based risk filters are similarly limited by generator calibration and sample count. Multimodal futures matter because a single plausible sample may conceal rare but costly outcomes; an expectation alone can also conceal tail risk.

This is a model-predictive-control-style interface, with a generator supplying the candidate futures. Part VII, Chapter 1 develops the broader planning setup.

The compute cost is where video-space planning differs from latent-space planning.

• Video-space rollouts generate $H$ frames or an equivalent clip for each of $M$ candidate action sequences. A frame-by-frame diffusion implementation with $N$ denoising evaluations per frame may use roughly $M H N$ network evaluations; a joint-clip implementation instead denoises a spatiotemporal representation per candidate. Autoregressive token models may need about $M H P$ sequential token predictions for $P$ tokens per frame, subject to batching, caching, and model design. Runtime and memory depend on token count, representation, network size, sampler, hardware, and horizon—not pixel resolution alone.
• Latent-space rollouts, including the feature-space prediction approach of DINO-WM in Part IX, Chapter 2, evolve a compressed state without decoding every candidate frame. This can be cheaper when the latent transition is substantially smaller than the video generator, but the factor is implementation- and benchmark-dependent. DINO-WM itself is not a small recurrent network, and visual encoding can remain a significant cost.

The comparison is not that one always wins. Video futures may make some predicted details easier to inspect, while a chosen latent might preserve the exact features a task needs. Either family can represent uncertainty, including through ensembles, at a cost determined by its architecture. Part VIII, Chapter 1 gives decision-centric examples.

The failure modes differ too. Latent-space planners can fail because their latent cannot represent some task-critical aspect of the observation (a numeric readout, a tiny object). Video-space planners can fail because the generator's pixels are right for the wrong reasons or because the perception head that computes $J$ is not robust to generated artifacts. An uncertainty penalty can help when its score is informative about task-relevant model error, even if it is not perfectly calibrated. It cannot reliably protect against confidently wrong predictions assigned a low penalty; those predictions may lead the planner to choose poor actions.

Receding-horizon feedback can limit the effect of rollout error by replanning from fresh observations. It cannot make an action-independent inner model distinguish consequences of candidate actions. For tasks that require choosing among actions, matched counterfactual branching is therefore a central diagnostic, though the test must cover action differences that matter in the task.