Diffusion and Flow World Models › Limitations and Impact

This scalar demonstration does not expose several system-level issues that matter in deployment. Three important examples are wall-clock latency, decision-making from predictive distributions, and distribution shift during learned-model rollouts.

The first complication is sampling cost interacting with control frequency. Twenty evaluations of a small MLP and twenty evaluations of a large transformer over image-conditioned latents have very different wall-clock costs. At control rates of 10 to 100 Hz, the full perception-planning-action loop has only 100 to 10 ms per cycle. Practical responses include reducing NFE, distilling to a few-step student, shrinking the network, or predicting a short buffer of future states. The relevant comparison is therefore sample quality at a measured end-to-end latency on the target hardware, not the quality of an unconstrained offline sample.

The second complication is that stochastic predictions require distribution-aware evaluation and planning. Planning with deterministic dynamics uses one modeled successor for each state-action pair, whereas planning in an MDP reasons over transition distributions and planning in a POMDP additionally reasons over belief distributions. Sampling-based methods can approximate planning computations in any of these settings. When a diffusion model supplies samples to a separate planner, that planner generally needs an objective-aware way to aggregate downstream outcomes, such as value estimation or a tractable surrogate. Conditional generative planners may instead encode part of that decision rule through return, goal, skill, or constraint conditioning. We revisit the compute and calibration demands of these interfaces in Part VIII: Decision-Centric Research Lineages.

The third complication is distribution shift. During recursive rollout, model samples become later contexts, and prediction errors can move those contexts beyond well-covered training regions. Behavior there is not validated by in-distribution loss and can become inaccurate or poorly calibrated. Guidance changes the sampled distribution as well: extrapolative CFG weights can trade diversity for condition adherence and may worsen errors under shift, but the direction and size of that effect are empirical. Rollout calibration should therefore be tested jointly with the intended guidance setting.

No single commonly used metric is sufficient for all the uses discussed here. Wasserstein distance can have poor empirical sample complexity in high dimensions, with rates governed by the distribution's effective metric dimension (Weed and Bach, 2019). For kernels whose bound remains uniform with dimension, maximum mean discrepancy has dimension-independent square-root concentration rates (Gretton et al., 2012); in growing dimension, its test power depends on the alternative and bandwidth and can deteriorate in some regimes (Huang et al., 2023). Neither metric necessarily reflects planning relevance. Task-level transfer and closed-loop control metrics are more direct for decision making and are often more expensive and environment-specific because they require execution in a task environment, although simulator-based or offline proxies can be cheaper. Evaluation therefore needs distributional, calibration, latency, and downstream measures selected for the application, connected to Part XI: Evaluation and Understanding. Latency, shift, calibration, and planning interact, so the deployed system must be evaluated as a whole.

Diffusion and flow models add a flexible alternative to earlier probabilistic dynamics models, latent-variable models, and autoregressive sequence models. They can model continuous, multimodal conditionals without a prescribed finite mixture and can express some trajectory constraints through inpainting. Flow matching, better solvers, consistency training, and distillation can reduce evaluation counts in suitable settings, making low-latency deployment more plausible when verified by measurement. Their contribution is not the invention of distributional world modeling, but a useful family of steerable generative parameterizations.