Cosmos and Omnimodal World Foundation Models › Summary

The Cosmos releases provide a worked example of physical-world model infrastructure: curation and tokenization in the original platform, later controlled generation, and an omnimodal language–vision–audio–action interface in Cosmos 3. These capabilities belong to different versions and require task-specific evaluation.

Key takeaways:

• Data curation shapes what the model can learn. Physical AI data can differ from typical web video in the prevalence of agentic viewpoints and action-relevant events, and in the availability of aligned sensors and continuous trajectories. The original Cosmos report describes shot splitting, filtering, VLM captioning, and distributed data preparation; corpus counts should be tied to that release.
• Tokenization mediates pixels and computation. The nominal compression triple $c_t \times c_h \times c_w$ sets a latent-grid cell count under the stated padding assumptions; additional patchification and conditioning can change generator sequence length. Continuous latents pair naturally with diffusion or flow models; discrete codes support categorical autoregression. Our toy does not establish a winner at equal rate. Causal tokenization avoids future-frame dependence; non-causal encoding may require buffering online.
• A generative model can represent multiple futures. In the toy pedestrian mixture, density at the conditional mean is about $3.3\times10^7$ times lower than density at a mode. Opposite toy decisions illustrate why event definitions and validated tail probabilities matter; they are not a deployed safety result.
• Flow and diffusion samplers can process a clip jointly at each network evaluation, with compute growing with clip size. Masked prediction requires a trained conditioning path and noise-consistent sampling. Autoregressive tokens are incremental, but context, runtime, and error accumulation limit useful horizons; action control requires training for action inputs.
• Conditioning architectures offer different controls, not guarantees. Shared-token interfaces can accept trained modality combinations; adaptive control branches can weight spatial signals. Zero initialization preserves base behavior at step zero only. Label adherence must be measured before synthetic data reuse.
• Guidance changes sampled futures. In the toy model, increasing $w$ suppresses a crossing mode and biases estimated event frequency. Using $w=1$ removes extrapolative guidance but is not proof of calibration; any risk estimate needs independent validation for its precise sampling setup.
• Guardrails must be distinguished. Documented Cosmos components screen prompts and generated visual content; physical-plausibility filters and runtime risk monitors are additional application designs, not shipped Cosmos safeguards. The toy threshold monitor is not a safety case.
• Customization needs retention tests. Zero-initialized LoRA starts from the base function, then learns an active perturbation. Our toy adapter uses 14.8% trainable parameters; its old-domain retention and real deployment cost were not measured.
• Serving budgets differ. Offline generation emphasizes throughput; interactive use has a controller-imposed latency limit. Under this chapter's illustrative full-attention formula, a 10-second 720p clip at 8x16x16 costs about 227 PFLOPs, or about 1,580 ideal GPU-hours for 10,000 such clips at an assumed $4\times10^{14}$ FLOP/s. The 20-second scenario costs about 861 PFLOPs per clip and about 5,980 ideal GPU-hours for 10,000 clips, before omitted overheads. These are not measured Cosmos costs.
• Throughput alone does not validate correctness. Physical plausibility, horizon consistency, control adherence, and calibration need separate checks. Curation continues after generation, and neural-simulator errors can bias policy evaluation.