Cosmos and Omnimodal World Foundation Models › Multimodal Generative Dynamics

With latents in hand, we can state a conditional prediction task. A video generator can be used as a world model when its generated futures are fit for the intended dynamics and decision question; architecture or visual quality alone does not establish that suitability.

Write the conditioning context as $c$ (text, a first frame, a layout, a control signal) and the observed history as $x_{\le t}$. A video world model represents

$$p_\theta\!\left(x_{t+1:t+K} \mid x_{\le t}, c\right),$$

the distribution over the next $K$ observation frames. Compare this with the latent state-space form we have used throughout the book, $p_\theta(z_{t+1} \mid z_t, a_t)\, p_\theta(o_t \mid z_t)$. Two differences stand out and both have consequences.

First, this factorization has no separately defined Markov state $z_t$. A video transformer still has internal latent tokens and activations, but they need not be a compact, inspectable belief state. It conditions on observation history within a finite context, so object permanence across occlusion must be learned and evaluated rather than assumed. A generated car that fails to reappear after an occlusion is a temporal-consistency error; diagnosing its cause takes more than observing pixels alone. The temporal-state discussion in Part III provides a useful comparison.

Second, an explicit action variable is not required for observational prediction. Without one, a model trained on behavior data estimates futures under the behavior mixture represented in that dataset. Such samples may aid some data-generation or perception tasks after validation, but they do not by themselves isolate a proposed braking intervention from correlated observations. Adding recorded actions makes counterfactual control studies possible only with adequate coverage, causal assumptions, and measured action adherence; conditioning alone is not a causal guarantee. We return to action interfaces in the next chapter.

Diffusion and flow matching are two related ways to train continuous-latent generators; neither is simply a universally cleaner reformulation of the other. The flow construction below clarifies the number of network evaluations per solver rollout; realized latency also depends on token count, implementation, and hardware. Prediction and inpainting can share weights when the model is trained and conditioned for those tasks; they do not follow automatically from the ODE.

Let $y_1$ be a data sample (a latent video, all tokens at once) drawn from $p_{\text{data}}(\cdot\mid c)$ for context $c$, and let $y_0 \sim \mathcal{N}(0, I)$ be independent noise of the same shape. Define a straight-line interpolant indexed by $t \in [0,1]$. This teaching convention runs from noise at $t=0$ to data at $t=1$; the Predict2.5 report writes the opposite time orientation, which is equivalent after reversing $t$ and the velocity sign:

$$y_t = (1-t)\, y_0 + t\, y_1 .$$

Along any particular path, the velocity is constant and equal to $y_1 - y_0$. The conditional velocity field is therefore trivial. The marginal velocity field is the conditional expectation

$$u_t(y,c) = \mathbb{E}\!\left[\, y_1 - y_0 \;\middle|\; y_t = y,\ c \,\right],$$

and regressing a network $v_\theta(y, t, c)$ onto $y_1 - y_0$ with a plain squared error,

$$\mathcal{L}(\theta) = \mathbb{E}_{y_0, y_1, t}\!\left[\left\| v_\theta(y_t, t, c) - (y_1 - y_0) \right\|^2\right],$$

has $u_t(y,c)$ as its population squared-loss minimizer under the stated sampling construction. The network sees individual (noise, data, context) examples rather than the conditional expectation directly. Once trained, samples are produced by numerically integrating the transport equation

$$\frac{dy}{dt} = v_\theta(y, t, c), \qquad y(0) \sim \mathcal{N}(0, I),$$

from $t = 0$ to $t = 1$ with an Euler or higher-order solver. With the ideal field and exact integration, this flow transports noise to the conditional data distribution. A trained, finite-step solver approximates it. “Probability-flow ODE” has a more specific meaning in score-based diffusion; this is a flow-matching transport ODE. Fewer steps reduce network evaluations, but latency and quality do not scale by a universal factor.

Why this matters for world models specifically, beyond sample quality:

• Generation can be parallel across frames within one solver step. A $K$-frame clip may take the same number of network evaluations as a one-frame sample, but each evaluation processes more tokens and can cost much more compute and memory. This favors some offline batch workloads; it does not guarantee a latency win.
• Prediction and inpainting can share a masked generator when its architecture and training support that conditioning. Replacing known coordinates during sampling needs a noise-level-consistent procedure; repeatedly inserting clean values is not a universal sampler. Depth, segmentation, and first-frame controls may require separate encoders, branches, or post-training, and adherence remains measured rather than guaranteed.
• For a fixed-shape clip sampler, you choose $K$ when laying out that sample's token grid. One common longer-horizon design chains generated chunks conditioned on earlier output; it needs seam and consistency tests. Other architectures can use different extension strategies, so chunk seams are a design-specific risk, not a universal location for failure.

The discrete-token path has a different mix of tradeoffs, especially sequential token decoding and the opportunity to cache prior context.

$$p_\theta(x_{1:K} \mid c) = \prod_{k=1}^{K} p_\theta(x_k \mid x_{<k}, c).$$

Here $x_k$ denotes a discrete token index, not a whole frame; for Cosmos1 it can be an FSQ index. Each token is sampled from a categorical distribution conditioned on available earlier tokens and context. Generation is incremental and may reuse a KV cache, but context length, cache memory, runtime, and error accumulation bound practical horizons. Action tokens can be interleaved only when the model was designed and trained to use them. Part IX develops the foundation-model setting.

The costs are operational. Sampling token indices is sequential in this factorization, so longer sequences normally require more decoding work, although batching, caching, and serving affect wall-clock latency. Exposure bias can arise when generated histories differ from training prefixes, allowing errors to compound. A generator may synthesize plausible high-frequency detail after quantization, but it cannot infer the exact lost detail from the code alone. These are distinct failure mechanisms; quantization does not itself cause autoregressive drift.

As a first approximation, clip-oriented flow/diffusion sampling trades several full-clip network evaluations against autoregressive token-by-token latency and caching. Both families can be adapted to longer sequences or interactive workloads, with different engineering costs. The original Cosmos platform released continuous- and discrete-token video families; later releases should be described by their own interfaces rather than assumed to share that exact division.

The next worked example isolates a failure of point prediction under a multimodal future.

Suppose you train a deterministic regressor to predict a future position by minimizing squared error. The Bayes-optimal solution is the conditional mean. With a multimodal conditional distribution, that mean can lie in a low-density region between likely outcomes. A point prediction then hides the risk of each mode, even though the mean itself need not have zero probability.

Multimodal futures can arise in interaction: a pedestrian may cross or stop, a vehicle may brake or continue, and a cyclist may swerve or hold a line. Their prevalence is task-dependent. For risk-sensitive decisions, a model needs a way to represent consequential alternatives; a conditional mean alone can conceal them. That does not imply a generative model is automatically calibrated or more accurate.

We use one scalar, one conditioning variable, and a two-component mixture. It illustrates a decision risk without claiming to reproduce the full behavior of a deployed system.

Consider a single decision-relevant scalar: the pedestrian's lateral position one second in the future, measured in metres relative to the ego vehicle's path. The context is $c \in [0,1]$, how much the ego yields (0 means the ego does not yield at all, 1 means it yields fully). The true generative process has two outcomes:

• the pedestrian crosses: $y = +2.0$ m,
• the pedestrian stops: $y = -1.0$ m,

with the crossing probability increasing smoothly in $c$, and Gaussian noise of standard deviation 0.25 m on either branch. The bimodality is the whole point: both outcomes are physically plausible, and which one occurs is uncertain given the context. Reducing the process to a single number is exactly the simplification that makes it tempting and wrong.

Let's write that process down and measure how bad the conditional mean is.