Diffusion and Flow World Models › Controllability, Latency, and Sampling Tradeoffs

For goal-directed planning, a generative model's utility depends strongly on the control information or inference-time objective available to select among its possible futures. An unconditional sampler still has uses in forecasting, simulation, uncertainty estimation, and representation learning; a model conditioned or guided by actions, goals, or constraints can participate more directly in planning. Diffusion and flow world models offer several control mechanisms. The choice shapes behavior and may change sampling cost: direct conditioning can preserve the update schedule, while classifier-free guidance, external guidance, and inpainting add mechanism-specific work.

Conditioning is the first mechanism. Putting a state-action pair, observation history, or belief state into $c$ turns a marginal generator into a conditional predictor. In closed-loop execution, context is refreshed from the latest observation or estimate at each replanning update. In open-loop generation, a horizon is generated from its initial context. A supplied action sequence can condition the full horizon either as an explicit input or, when actions are trajectory variables, by fixing or inpainting those action entries; if the action entries remain unconstrained, the model generates them. Direct context conditioning is a modeling and training mechanism: it changes the distribution the network learns, while the numerical sampler can remain unchanged.

Classifier-free guidance (CFG, Ho and Salimans, 2022) is the second mechanism. Train the network to handle both conditional and null conditioning by randomly dropping $c$ during training. At sampling time, form an affine combination of the null and conditional predictions with $w\geq0$:

$$\tilde{\epsilon} = \epsilon_\theta(x_t, t, \varnothing) + w\,\bigl(\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \varnothing)\bigr).$$

Under this convention, $w=0$ gives the null prediction, $w=1$ gives the ordinary conditional prediction, and $w>1$ extrapolates beyond it. Increasing $w$ can strengthen adherence to the condition, but it can also reduce diversity and bias samples away from the learned conditional distribution. The useful setting is empirical: neither small $w$ nor large $w$ guarantees better physical accuracy. The guidance term scales the conditional-minus-null difference by $w$ and amplifies it beyond the ordinary conditional prediction only when $w>1$.

Classifier guidance (Dhariwal and Nichol, 2021) adds the gradient of a noisy-input classifier log likelihood to the diffusion score or reverse update. Diffuser adapts this guided-sampling pattern to control by using the gradient of a learned trajectory-return model to steer samples toward higher predicted return. The guidance model can be changed at inference time, but it introduces another source of error. An inaccurate gradient, applied repeatedly across denoising steps, can bias samples toward trajectories the guide scores highly even when the dynamics model has little supporting data there.

For trajectory diffusion, the third mechanism is direct horizon-wide inpainting. Mask the generated trajectory, reimpose known entries such as the initial state or goal after each update, and sample the rest. A standard causal one-step model instead needs an additional coordination mechanism for comparable future-time constraints. Inpainting enforces the masked values numerically, but it does not guarantee that the interpolated trajectory is dynamically feasible. Constraints outside the learned support can therefore yield a completion with correct endpoints and implausible intermediate states; feasibility must be checked separately.

Latency is the fourth consideration, and it often decides whether a generative world model is usable in a control loop. Neural-network function evaluations (NFE) are a useful cost proxy, but wall-clock time also depends on model size, batching, memory traffic, solver overhead, and hardware. The following ranges are illustrative configurations rather than guarantees:

• An ancestral DDPM may use hundreds or roughly a thousand single-stage updates.
• A deterministic DDIM schedule may use tens of selected timesteps.
• Higher-order diffusion or ODE solvers can reach useful accuracy in tens of steps on some tasks, although a multistage step can require multiple NFE.
• Rectified flows may tolerate coarse integration when their learned paths are sufficiently straight.
• Consistency models and distilled students explicitly target one-step or few-step generation.

The controlling factor inside a model-predictive control loop is wall-clock budget, not NFE alone. A controller running at 10 Hz has a 100 ms cycle for sensing, inference, planning, and action selection together. Reducing evaluations and using a smaller network can both help, but their effect must be measured on the target hardware.

There is therefore an empirical quality-latency frontier, not a universal ordering of samplers. A solver that is better at equal step count may use more network evaluations per step, while a distilled student can trade fidelity for low latency. A practical workflow is to define the wall-clock budget, benchmark candidate samplers at that budget, and then decide whether distillation is worth its training cost. We return to the planning side in Part VII, Ch 1: Sampling-Based Planning and Model Predictive Control.

The final tradeoff is about what is generated. Per-state prediction has a smaller output but repeatedly conditions on model-generated states. Trajectory generation models cross-time dependencies jointly, while multiplying output dimension by the horizon. Short-window generation with frequent replanning is one useful compromise; the best window length is task- and latency-dependent.