Cosmos and Omnimodal World Foundation Models › Conditioning, Guardrails, and Customization

Conditional control, screening, and adaptation make a generative model useful in specific workflows. They are separate capabilities: having one does not establish the others. The Cosmos releases offer examples of each, with different interfaces and evidence requirements.

Broadly, conditioning signals fall into four groups. An omnimodal family may support several input–output combinations across them, but supported combinations and optional inputs depend on the particular model and task. The groups also differ in how strongly they constrain output and how failures should be measured.

• Semantic and stylistic: text prompts can describe scene, weather, time of day, or camera behavior. Language leaves many pixels unspecified, so adherence to a particular object or trajectory needs separate checks.
• Structural and geometric: depth maps, semantic segmentation, edge maps, LiDAR returns, 3D bounding boxes, camera intrinsics and extrinsics, and trajectories from a planner. These can constrain geometry and improve label alignment, but generated pixels still need adherence checks before the original labels are reused.
• Appearance reference: a first frame or an image of a specific object can guide identity and texture. Whether the generated clip preserves either across frames must be tested.
• Control and action: steering, throttle, joint velocities, latent actions inferred from observed changes, or discrete keyboard-style actions can condition a future on a proposed choice when a compatible model was trained for them. A causal counterfactual additionally needs adequate action coverage and assumptions about confounding; supplying an action token alone is insufficient.

Two useful architectural patterns for injecting them are sequence concatenation and parallel control branches. The choice changes what can be constrained and how the model is adapted.

Sequence concatenation. Modality encoders can produce tokens that enter a shared transformer with modality identifiers and attention masks. Training on missing-modality patterns can let one model accept several subsets without changing its network shape; concatenation alone does not guarantee every subset works. Attention also gives a soft influence rather than a hard geometric or action constraint. This is a general pattern, not a claim that every Cosmos 3 modality uses one undifferentiated stream.

Parallel control branches. A side network injects spatial controls into a frozen main network through zero-initialized projections, following the ControlNet pattern. At initialization the side branch contributes zero, so the composite model matches the base model. Once trained, that branch can change outputs, and neither the pretrained prior nor control adherence is protected automatically. Branch weights can be scaled, including over space and time when the implementation supports it.

Cosmos-Transfer1 reports adaptive multimodal control: multiple spatial conditions, including depth and segmentation, can receive different weights at different locations. This can help the generator use stronger signals where they are informative, but 3D consistency and adherence are measured outcomes, not consequences guaranteed by the architecture. For a driving dataset, inspect alignment separately for near-field objects, the horizon, and small labeled regions.

For labeled training data, measure whether generated pixels still match simulator segmentation. A visually better clip with mismatched labels can damage a downstream learner. The options are to reject it, relabel it, or validate that the remaining mismatch is acceptable for the specific task; appearance quality is not a substitute for this check. Part VI discusses data quality in world-model learning.

Classifier-free guidance is one common way to trade sample diversity for condition adherence. In its usual form, a model trained with conditioning randomly dropped combines conditional and unconditional predictions:

$$\hat{v} = v_{\text{uncond}} + w \left( v_{\text{cond}} - v_{\text{uncond}} \right).$$

When $w = 1$ this formula returns the model's plain conditional velocity. Larger $w$ extrapolates away from its unconditional velocity. In many generators that improves condition adherence at the cost of diversity and can introduce artifacts or suppress a mode. The effect depends on the trained fields and condition; no fixed guidance weight has a universal interpretation as a calibrated probability distribution.

Guidance can change event frequencies in the sampled distribution, beyond visible appearance differences. Even if unguided samples were calibrated for a particular event and evaluation set, increasing $w$ would require a new calibration check. In the toy experiment below, we call a sample a crossing-branch outcome when its lateral position exceeds $0.5$ m. This branch-frequency diagnostic differs from the later monitor's stricter conflict-line event at $1.9$ m. At $c=0.35$, guidance drives the measured crossing-branch frequency below the analytic mixture weight. That direction and size of bias are properties of this toy model, not a general collision-risk correction factor.

Guidance settings should be recorded with generated samples. If sampled futures inform risk estimates, validate event-frequency calibration on independent data for the exact model, guidance setting, context range, and sampling procedure. Using $w=1$ avoids extrapolative guidance but does not itself make a learned model calibrated. Appearance-tuned and decision-analysis sample streams can be kept separate; neither should be presented as a safety probability without validation.

"Guardrail" can refer to different checks. NVIDIA's documented Cosmos guardrails cover prompt and generated-content safety. Physical-plausibility screening and runtime risk monitoring are separate system components proposed here; the latter is not a shipped Cosmos safety feature.

Layer one: input and data filtering. Corpus preparation may screen personal or licensed material. Separately, the documented Cosmos pre-guard checks text prompts with a blocklist and content-safety classifier. Those mechanisms enforce a content policy; they do not verify physical dynamics or label fidelity, and their coverage depends on the classifier and release.

Layer two: output screening. The documented Cosmos post-guard screens generated frames for disallowed visual content and blurs faces. A physical-AI data pipeline also needs separate checks for interpenetrating objects, inconsistent scale or shadows, implausible motion, and label mismatches. These are dataset-validity or physical-plausibility problems, not proof that the shipped content filter detects them.

Layer three: runtime risk monitoring, if a world model is placed in a decision loop. An application might estimate event frequency from rollouts and compare it with warning or fallback thresholds. Such a monitor must be calibrated, stress-tested for shift, and backed by independent safeguards before deployment. It is not supplied by the content guardrails above, and a thresholded sample estimate alone is not a certified controller.

The following toy monitor only shows how distribution shift from guidance can change a thresholded decision. Its thresholds are illustrative, not operating limits for a vehicle.

Two further mechanisms matter for a deployed design, although this notebook does not implement them. Independently trained ensembles can reveal some epistemic disagreement; changing a random seed in one fixed model chiefly measures sampling variability, not epistemic uncertainty. Support and shift detection asks whether the current context resembles those used to train and calibrate the model. Outside that domain, calibration evidence may not transfer. Both checks can motivate a fallback, but neither guarantees that a rollout is safe or physically correct.

Pretrained generators can be specialized in several ways. The following options differ in their training data, changed parameters, and evaluation burden; their cost ordering depends on implementation and scale.

Instruction and control tuning. Fine-tune on paired (condition, video) data to add or improve a camera, weather, time-of-day, or layout control. Whether this is cheaper than an adapter depends on the parameters updated and data required; its adherence must be tested on held-out conditions.

Parameter-efficient adaptation. Low-rank adaptation adds $\frac{\alpha}{r}BA$ to a frozen weight $W$, where $A \in \mathbb{R}^{r \times k}$ and $B \in \mathbb{R}^{d \times r}$. For a large $d \times k$ weight, $r$ is typically much smaller than $\min(d,k)$; the tiny first layer in the toy below is an exception. With $B$ initialized to zero, the combined network equals the base model at step zero. Training changes the adapter, so original-domain behavior must be rechecked. Small adapters can be stored separately for site or camera variants when the serving implementation supports swapping them.

Subject and scene personalization. Reference images and a subject identifier can adapt a generator toward a particular car, robot, or site, in the style of DreamBooth. Identity persistence across frames is a result to evaluate, not a property guaranteed by the method.

Preference and reward-based post-training. Optimize against human preferences or a learned reward for domain-specific outputs. This can encourage reward hacking or physical-distribution drift, so retain an independent fidelity check rather than treating reward improvement as proof of physical validity.

Let's implement LoRA on our toy flow model and adapt it to a shifted one-dimensional geometry: the stopping-position mean moves laterally from $-1.0$ to $-0.6$ m and the crossing-position mean from $+2.0$ to $+2.6$ m. These are lateral coordinates, not physical forward/backward distances. This is an adapter mechanics exercise, not an adaptation experiment on a world foundation model or a real site.