Cosmos and Omnimodal World Foundation Models › Foundation-Model Infrastructure for Simulation

The model also needs a data and serving pipeline. Tokenizer rate, sampling schedule, hardware, and filtering throughput affect cost and usefulness, alongside architecture and model quality. Published generator weights are tied to their trained tokenizer and input interface; two teams cannot freely substitute arbitrary tokenizers and expect the same model to work. The calculations below isolate a few resource terms rather than ranking their real-world importance.

We can write down two dominant matrix-multiplication terms for an illustrative full-attention transformer. This is an order-of-magnitude comparison of tokenization choices, not a latency or hardware-sizing estimate for a released Cosmos model. For $L$ tokens, model width $d$, $n_\ell$ layers, and $S$ sampling steps, the simplified forward cost per step is

$$\text{FLOPs per step} \;\approx\; \underbrace{4\, n_\ell\, L^2\, d}_{\text{attention}} \;+\; \underbrace{16\, n_\ell\, L\, d^2}_{\text{feed-forward}} .$$

The attention score/value term is quadratic in token count; the feed-forward term is linear in tokens and quadratic in width. The estimate omits Q/K/V/output projections, embeddings, normalization, tokenizer and decoder work, and communication. Tiled exact attention can avoid materializing the quadratic attention matrix, while its arithmetic remains quadratic. Hardware utilization and memory traffic also shape wall-clock time, so the numbers below are illustrative FLOP estimates, not measured runtime or strict latency lower bounds.

A deployment may need two quite different operating regimes; it need not use the same model or serve both.

Offline data generation emphasizes validated clips per unit cost. Batching, parallel generation, sampler distillation, reduced-precision inference, and caching can help if supported and measured for the chosen model. Fewer tokens can lower cost, but the resulting loss of small objects or labels must be checked against downstream utility.

Interactive simulation has a per-step latency budget imposed by its controller and sensors. Causal or autoregressive models, caching, local attention, and shorter diffusion schedules are possible responses, each requiring task-specific quality and timing checks. The full-attention curves above illustrate why one dense architecture may be expensive; they do not set a universal lookahead limit for every serving design.

Suppose a synchronous controller runs at 10 Hz and requires camera, depth, and semantic outputs at every step. The complete model-and-sensor pipeline then has at most 100 ms before the next decision. One 80-ms denoising step leaves too little time for a multi-step sampler and other work. This does not imply that Cosmos's long-video extension meets interactive latency: Predict2's documented autoregressive inference chains diffusion-generated clips and reruns denoising for each chunk. It differs from Predict1's discrete autoregressive model. Measure end-to-end rate before calling either an interactive simulator.

The place where all of this becomes concrete is the simulator. Four integration patterns recur.

Real2sim reconstruction. Take a real scene and estimate simulator assets, layout, trajectories, and camera poses to obtain a scene that can be varied. This is hard and largely outside the generator. Reconstruction errors can propagate into later variations, although later fitting and validation may detect or correct some of them.

Conditional photorealization. Render a reconstructed scene with simulator labels such as depth, segmentation, boxes, and optical flow; supply the render and supported controls to a generator. Adaptive multimodal control can improve alignment, but generated appearance does not automatically preserve the labels or reproduce the real-world distribution. Measure adherence, reject or relabel mismatches, and validate whether the retained clips help a downstream perception model.

Neural evaluation environments. Use the world model directly as a policy evaluation environment. This ambitious use needs especially strong checks: model errors can bias the evaluation metric, and an optimizer may exploit weaknesses in a learned simulator. Part VIII discusses this model-exploitation risk. Compare important results with independent evidence from real tests or a separately validated simulator; optimization against the learned environment alone cannot establish deployment performance.

Sensor simulation. Synthesize LiDAR or radar returns as well as camera images. Consistent cross-sensor outputs benefit from geometric structure or aligned geometric cues, but an explicitly geometric shared latent is not the only design: LiDAR diffusion can use a sensor-specific latent, and camera-conditioned radar generation uses a radar representation with geometric guidance. The evaluation question is whether generated modalities agree on scene geometry and sensor-specific effects.

It is worth being explicit about the boundary, because infrastructure discourse tends to overclaim.

More accelerators can raise generation throughput, but that change alone does not establish physical plausibility, long-horizon consistency, control adherence, or calibration. A faster generator can also increase the number of clips that require screening. A production pipeline therefore needs measured acceptance rates and downstream utility, not an assumed fraction of usable outputs. Curation continues after generation because synthetic clips can contain physical or label-consistency failures absent from the simulator source.