Cosmos and Omnimodal World Foundation Models › Cosmos and Omnimodal World Foundation Models

Imagine you are responsible for validating an automatic emergency braking policy before it ships. Controlled track tests can use pedestrian surrogates, and a simulator can supply repeatable labels and rare-event variations. Neither, on its own, establishes how a perception stack will respond to every real camera condition. Road recordings help ground the evaluation but may contain too few examples of a pedestrian stepping out from behind a parked van at dusk in the rain.

The tradeoff is concrete: the simulator gives you control and labels but may have an appearance gap. Real video may undersample the scenarios you most need to test. A conditional generator could help vary scene appearance and propose candidate rare scenarios. It cannot certify their frequency, physical validity, or label fidelity merely by producing plausible pixels.

World foundation models offer a third research route, and they are the subject of this chapter. Different Cosmos releases use broad physical-world pretraining, then offer different interfaces: Transfer models can adapt a simulator render, while supported Predict variants condition on prompts or observed clips. The output can supply candidates for synthetic data or evaluation, provided downstream checks reject outputs that violate geometry, labels, or task constraints. NVIDIA's Cosmos platform, introduced in 2025, is our running example; it is a family of releases, not one model with one architecture or license.

Those jobs may benefit from related visual and temporal pretraining, but they need not share weights, representations, or an interchangeable conditioning interface. A model trained on driving video may learn how wet asphalt reflects headlights or how a van occludes a pedestrian. Whether it can photorealize a simulator render or generate a controlled rare scenario depends on the particular release, on its training, and on its supported inputs. The foundation-model proposition is reuse and adaptation across such jobs, not proof that one generator solves both by switching a prompt.

The word omnimodal should not be projected backward onto every Cosmos release. It fits Cosmos 3, whose documented model family jointly processes and generates language, images, video, audio, and action sequences. Older Predict and Transfer models expose narrower task-specific interfaces. Even with a unified interface, accepting a modality does not guarantee faithful use of it; contradictory text, depth, segmentation, or action inputs need explicit adherence tests.

This chapter follows video generators and interactive generated environments. The Cosmos family illustrates conditional simulation and data-generation workflows, while Cosmos 3 also reaches into action modeling. The next chapter develops that action interface directly in vision-language-action and world-action models. The boundary is therefore a teaching distinction, not an architectural separation that every current model obeys.

Three ideas recur throughout and are worth stating up front, because they are the lenses through which every Cosmos component should be read.

• Compression shapes the interface. A video tokenizer sets a nominal latent-grid size and affects which details survive. The generator may patchify those latents further and add condition tokens, so its actual sequence length, training cost, memory use, and latency also depend on architecture and serving choices. A published context length must be checked for the specific model and tokenization path before converting it into seconds of video.

• A generator models a conditional distribution of futures. A diffusion or autoregressive model can sample from an estimated $p(\text{future}\mid\text{context})$, while a point forecast summarizes it with one value. One rollout is one possible future under that learned distribution; its event frequencies cannot be read as real-world probabilities without validation.

• For labeled training data, control adherence matters alongside appearance. A photorealistic video that no longer matches simulator labels may harm perception training. Cosmos-Transfer1 offers spatially weighted conditioning for supported controls; that mechanism motivates adherence checks, not an assumption that every Cosmos release shares the same branch or preserves labels.

“Cosmos” names a platform and several related but nonidentical model families. The original Cosmos world-foundation-model platform report describes video tokenizers, video generation, transfer, curation, and guardrails. It is the source for the platform-level pipeline below. Cosmos-Predict2.5 and Transfer2.5, reported later, include a flow-based Text2World/Image2World/Video2World generator and a control-net-style transfer system; those claims belong to the 2.5 family, not retroactively to Predict1. NVIDIA reports improvements and benchmarks for those releases, but their reported scores are not independent evidence that generated clips are safe or label-preserving for a new deployment.

Cosmos 3 is the release that makes omnimodal literal: NVIDIA documents a unified Mixture-of-Transformers architecture spanning text, image, video, audio, and action sequences, with Reasoner and Generator runtime surfaces. The Cosmos 3 model card describes an autoregressive tower for discrete text tokens and a diffusion tower for continuous multimodal outputs. That is not the same as placing every modality in one undifferentiated token stream, and a supported input–output combination still needs task-specific evaluation. The model card lists the Cosmos 3 weights under OpenMDW-1.1; the older Cosmos documentation lists the platform model weights under the NVIDIA Open Model License. Check the particular artifact's license rather than treating “Cosmos” as one license.

The worked calculations in this chapter are pedagogical, not Cosmos 3 benchmarks or reproductions. When we discuss a tokenizer, flow objective, control branch, or guardrail, we will name the relevant release or mark a proposed system design. This distinction matters most for runtime safety monitors: NVIDIA documents content guardrails for generation, whereas the probability-threshold monitor implemented below is our teaching example, not a shipped, calibrated Cosmos safety component.