Cosmos and Omnimodal World Foundation Models › Physical AI Data and Tokenization

The training corpus and video tokenizer both shape what a world foundation model can learn from observations. They are not the only determinants: architecture, objective, conditioning data, and post-training matter too. A curation pass that discards fast-motion clips may conceal a temporal codec's weakness on fast events; aggressive temporal compression can discard task-relevant information even in a high-frame-rate corpus. Evaluate data slices and reconstructions together.

"Physical AI" is a term for systems whose intelligence is expressed through bodies in the world: vehicles, manipulators, legged robots, drones, and warehouse robots. Their data can differ from typical web video in the prevalence of ego-motion and contact, the availability of action annotations and synchronized sensors, and the continuity of trajectories. Those differences affect data collection and the tests a world model needs.

• An agentic viewpoint. Many cameras move with a vehicle or robot, so ego-motion can account for substantial pixel change; independently moving objects and lighting matter too. Video-only prediction can learn correlations along recorded trajectories without identifying how futures change under alternative actions. That distinction matters when a model is used for planning.
• Physically plausible dynamics. Objects often persist across frames, occlusions may persist or resolve, and camera trajectories should remain coherent with the scene; abrupt physical motion can still be real. For a continuous physical rollout, cuts, edits, and cross-fades are discontinuities rather than environmental dynamics. Because edited web video often contains them, a corpus assembled without care can encourage the model to reproduce discontinuities.
• Action relevance. Near-misses, contact, occlusion boundaries, and degraded visibility can be consequential yet underrepresented in collections dominated by routine operation. If they are scarce in the target corpus, targeted collection or candidate generation may help. Measure coverage and choose sampling weights for the evaluation task rather than treating every recorded second as equally informative.
• Multimodal alignment. Some physical-AI datasets include depth, LiDAR, radar, IMU, odometry, or joint readings alongside video. Available sensors vary by platform; paired signals need appropriate spatial calibration and temporal synchronization. A one-frame offset can corrupt a supervised relationship without necessarily causing visible hallucination in every output.
• Condition diversity. Time of day, weather, sensor degradation, lens dirt, and seasonality change observations; some also change physics. Wet pavement, for example, changes tire–road friction and braking behavior. Appearance conditioning alone cannot stand in for dynamics coverage, so test both.

Large raw video collections usually need curation before training. The original Cosmos report documents stages that affect the data distribution; the order and retention rules should be chosen for the intended task and measured rather than copied wholesale.

1. Session and shot splitting. The original Cosmos pipeline extracts clips of about 2 to 60 seconds and detects shot boundaries using frame changes. A hard edit between unrelated views is not an ordinary physical transition; training across it can confuse temporal prediction. Overly short clips may omit the event evolution needed for a task.
2. Quality filtering. Screen clips that are blurry, over- or under-exposed, mostly static, dominated by lens occlusion, or dominated by text overlays and picture-in-picture. Retention criteria depend on the target task: static or degraded clips can be useful for some evaluations. Burned-in captions and watermarks can be copied into generated outputs, so measure their prevalence rather than assuming a particular amplification factor.
3. Captioning. Use a vision-language model to annotate clips or clip segments, ideally with temporal grounding. The original Cosmos report describes a caption per 256 frames, not necessarily one exhaustive description of every full clip. Captions support text conditioning and retrieval; their coverage and errors affect which scenarios users can request reliably.
4. Deduplication. Check near-duplicates within the corpus and between training and evaluation sets. Leakage can inflate a reported generalization score; the score alone cannot reveal which examples leaked.
5. Diversity adjustment, shuffling, and sharding. Balance the mixture across domains, sensor types, and environments. Then write it into shards that can be streamed by a distributed loader without random seeks into object storage. Mixture balance is a modelling decision in disguise: the relative weight of highway, urban, and rural footage helps shape which environments the model handles well.

The original Cosmos report describes a 20-million-hour raw collection curated into about 100 million clips. Its documented stages include shot splitting, filtering, annotation, deduplication, and sharding. The counts belong to that release; a new corpus needs its own retention criteria, labeling plan, and evaluation split.

Data coverage and architecture both constrain physical-AI world models. Large collections of synchronized, action-labeled interaction video are difficult to curate, especially for rare events. Real recordings, structured simulators, and neural generators can complement one another, but each brings distinct limitations in coverage, fidelity, or label validity.

Here a video tokenizer acts as a learned compression codec. Its encoder maps a clip $x \in \mathbb{R}^{T \times H \times W \times C}$ into latents

$$z = E(x), \qquad z \in \mathbb{R}^{\lceil T/c_t\rceil \times \lceil H/c_h\rceil \times \lceil W/c_w\rceil \times d}$$

and a decoder maps them back, $\hat{x} = D(z)$. This shape assumes suitable padding and integer downsampling factors; learned codecs need not correspond to independent, nonoverlapping blocks. The factors are often written $c_t \times c_h \times c_w$. The nominal token count is

$$L \;=\; \left\lceil \frac{T}{c_t} \right\rceil \left\lceil \frac{H}{c_h} \right\rceil \left\lceil \frac{W}{c_w} \right\rceil .$$

This $L$ counts latent-grid cells under the stated shape assumption, not necessarily transformer tokens: a generator may patchify the latent grid and add condition tokens. More cells generally increase compute, but the relationship depends on attention pattern, implementation, precision, hardware, and serving. Dense attention has a quadratic arithmetic term in sequence length; memory-optimized exact attention need not materialize a quadratic attention matrix. Neither a fixed horizon nor a frame rate follows from the codec factor alone.

The choice of what to compress is not symmetric either. Aggressive temporal compression can lose fast events, but a learned temporal codec may integrate information across all frames in its receptive field; it is not necessarily simple frame decimation. Whether a lane crossing survives must be tested on reconstructed clips. Aggressive spatial compression can lose brake lights, fingers, distant pedestrians, or lane boundaries. As with state abstractions in Part III, the relevant question is downstream sufficiency, not average fidelity alone.

Continuous versus discrete tokensLink Copied

Continuous and discrete latents are two common design choices. They affect the generator objective and interface but do not uniquely determine the generator family.

Continuous tokenizers produce floating-point latents, sometimes with KL or perceptual regularization. They pair naturally with latent diffusion and flow models. A 16-dimensional latent stored in bf16 occupies 256 storage bits per token before any entropy coding; that is not a measured rate–distortion result. The PCA toy below uses float64 in memory unless explicitly cast. Continuous latents do not preclude likelihood modeling or reuse of all sequence-model tooling, but they do not provide a native categorical next-token objective.

Discrete tokenizers map latents to integer indices in a vocabulary of size $K$. A fixed-width representation needs $\lceil \log_2 K \rceil$ bits per index; entropy coding can yield a different average rate. A learned VQ-VAE codebook is one way to construct such indices, but the original Cosmos tokenizer uses finite scalar quantization (FSQ): six scalar coordinates with $(8,8,8,5,5,5)$ levels respectively define a 64,000-entry implicit vocabulary, not a learned VQ codebook. Discrete indices support categorical autoregression, subject to the model's training and interface. Temperature, top-$k$/nucleus sampling, and KV caching may be used when the decoder supports them; text–video interleaving and speculative decoding require compatible designs. Quantization error and autoregressive drift still matter. Learned-codebook collapse and EMA updates are VQ-specific concerns, not requirements of Cosmos's FSQ tokenizer.

The original Cosmos platform reported continuous and discrete video tokenizers and causal variants. A causal tokenizer avoids dependence on future frames and is suited to streaming. A non-causal tokenizer can still be used online with a buffering window and corresponding delay; it is not categorically unusable. Rate–distortion, latency, and model compatibility all matter to the choice.

What to measureLink Copied

A tokenizer is often evaluated on reconstruction, but reconstruction alone is not a sufficient selection target for a policy-facing world model. Average pixel fidelity says little about whether the compressed representation still supports the decisions you intend to make. The useful panel of measurements looks like this.

• Distortion: PSNR and SSIM for pixel/structural fidelity, LPIPS for learned perceptual similarity, and FVD for distribution-level video comparison rather than a per-frame reconstruction score. Each can miss a small safety-critical object, so slice by task-relevant events.
• Rate: measure actual compressed bits per second including codebook or entropy-coder overhead where relevant, then relate that to latent token count and model compute. A bf16 storage estimate is not a fair bitrate comparison to entropy-coded discrete tokens.
• Downstream utility: does a policy trained or evaluated on reconstructed observations behave the same as on originals? A tokenizer can score well on average reconstruction while losing a small decision-relevant detail. For a policy-facing use case, this check should carry substantial weight, despite the extra work of running the downstream task.
• Compression-shape sensitivity: measure distortion separately on fast-motion and fine-detail slices. Average PSNR can hide small-object failures. Slicing by motion magnitude and object size takes additional annotation or detection effort but makes the operating envelope clearer.

The arithmetic is simple enough to do on paper, and doing it once makes every infrastructure discussion in this chapter concrete.

Take a 10-second clip at 30 frames per second and 1280×720 resolution. That is $T = 300$ frames. With an 8x16x16 tokenizer, the temporal axis becomes $\lceil 300/8 \rceil = 38$ groups, the height becomes $45$, and the width becomes $80$. The nominal token count is $38 \times 45 \times 80 = 136{,}800$, before text or other modalities. With 8x8x8, it quadruples to $547{,}200$. Compare these counts with the actual context and attention scheme of the model being evaluated rather than an unspecified language-model context window.

These latent-grid counts illustrate a resource pressure, not the input length of a particular Cosmos transformer. Patchification, attention layout, and conditioning change the actual token sequence. Full attention has quadratic arithmetic in sequence length, although exact memory-efficient implementations can avoid a quadratic attention-matrix allocation. Diffusion or flow inference performs a model evaluation per sampler step, but step counts vary by release and distillation; Predict2.5, for example, reports a four-step distilled variant. A hypothetical 30 GPU-seconds for ten generated video seconds is a throughput example, not a measured wall-clock latency or Cosmos benchmark.

Let us build a miniature version of this pipeline. We will synthesize a tiny driving clip with a synchronized depth map and a semantic segmentation map, exactly the kind of multimodal bundle that physical AI corpora contain. We will use a fixed random seed so the example is deterministic and reproducible. Then we will run a deliberately naive tokenizer over it.