Hierarchical, Hybrid, and Omnimodal Models › Shared State Across Modalities

A robot may receive vision, depth, touch, proprioception, audio, and language. A shared-latent multimodal world model gives those signals a common predictive state, allowing one modality to help disambiguate another. Separate private streams can also exchange information through cross-attention, so one literal shared bottleneck is not the only solution. Tolerance to missing inputs likewise requires suitable architecture and training; the label omnimodal does not guarantee it. The analogy to classical filtering is useful but limited: a Kalman filter combines measurements under an explicit probabilistic model, whereas a neural fusion rule and its state are learned.

Three common fusion strategies are early fusion, late fusion, and intermediate exchange through mechanisms such as cross-modal attention. The first two describe placement near the input or output, while attention is an interaction operator that can be inserted at different depths and can coexist with either layout. Both placement and mechanism affect robustness and the kinds of correlations the model can capture.

Early fusion. A common design concatenates modality streams near the input and passes them through a shared encoder. Without explicit reliability weights, that encoder must infer each modality's influence during training. Masks, imputation, or fixed missing-value slots can make the design tolerate absent inputs; untrained omissions remain a distribution shift.

Late fusion. Each modality is encoded independently, and modality-specific predictions are combined at the output. Dropping an encoder is structurally simple, though the combiner still needs training for missing inputs. Because interaction occurs only after separate interpretations have formed, this design has less opportunity to model fine cross-modal correlations.

Intermediate exchange with cross-modal attention. In one common design, modalities receive separate preprocessing or encoders before attention exchanges information among their representations. Flamingo is a direct visual-to-language cross-attention example; Perceiver IO is a related multimodal architecture that routes arbitrary inputs through a latent bottleneck. This exchange can occur at one or several depths rather than defining a single placement category. Cross-modal attention adds interaction cost relative to independent encoders plus a simple combiner. Dense pairwise attention does not scale gracefully by itself, but sparse, asymmetric, or bottlenecked variants can. Attention masks make absent streams structurally representable, but reliable performance generally requires training on the relevant missingness patterns.

A shared latent $z_t$ with $M$ modalities is inferred from

$$q(z_t \mid o_t^{(1)}, \dots, o_t^{(M)}) = q\!\left(z_t \;\middle|\; \text{Fuse}\big(e_1(o_t^{(1)}), \dots, e_M(o_t^{(M)})\big)\right),$$

where:

• $q(z_t \mid \cdot)$: the approximate posterior over the shared latent given the available modality observations.
• $z_t$: shared latent state at time $t$.
• $o_t^{(m)}$: observation from modality $m$ at time $t$.
• $M$: number of modalities.
• $t$: time step index.
• $m$: modality index running from $1$ to $M$.
• $e_m$: per-modality encoder for modality $m$.
• $\text{Fuse}$: fusion function (concatenation, attention, or a learned pooling operator). The decoder mirrors this by producing modality-specific heads from the same $z_t$.

The equation says that each available modality is encoded and that a fusion operator maps the resulting embeddings into a shared latent. Modality-specific heads decode from $z_t$. The latent is a strict shared bottleneck only if it is the sole predictive path from inputs to outputs. Private or skip paths do not make the latent unshared, but they make the bottleneck nonexclusive.

What forces the shared latent to be shared? Left to their own devices, per-modality decoders can send most information through private paths and never develop a common state. A pure reconstruction objective does not prevent this because each decoder only has to rebuild its own input. Four objectives encourage fusion:

• Reconstruction. Each modality head must reconstruct its own input from $z_t$. Simple but weak on its own for encouraging cross-modal content, since it can be satisfied without any cross-modal interaction.
• Cross-modal prediction. Predict one modality from the others via $z_t$: for example, recover depth from RGB and proprioception, or audio from video. This rewards content that is predictable across modalities. It cannot recover target information that is statistically absent from the inputs.
• Contrastive alignment. Raise the similarity of matched cross-modal pairs relative to sampled negatives, thereby lowering the negatives' relative scores. A common formulation is the InfoNCE loss

$$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(z_t^{(a)}, z_t^{(b)})/\tau)}{\exp(\text{sim}(z_t^{(a)}, z_t^{(b)})/\tau) + \sum_{j=1}^{N} \exp(\text{sim}(z_t^{(a)}, z_j^{(b)})/\tau)}$$

where:

• $\mathcal{L}_{\text{InfoNCE}}$: the InfoNCE loss value, minimized when matching pairs have high similarity relative to negative samples.
• $z_t^{(a)}$: embedding of modality $a$ at time $t$.
• $z_t^{(b)}$: embedding of modality $b$ at time $t$.
• $z_j^{(b)}$: embedding of modality $b$ for the $j$-th negative sample.
• $j$: negative-sample index.
• $N$: number of negative samples.
• $t$: time step index.
• $\text{sim}(\cdot, \cdot)$: similarity function, usually cosine similarity.
• $\tau$: temperature parameter (smaller $\tau$ sharpens the distribution over negatives).

The numerator scores the positive pair, while the denominator sums the positive and negative scores, so minimizing the loss raises the positive pair's similarity relative to the negatives. The temperature $\tau$ controls how sharply the loss penalizes near-miss negatives: a small $\tau$ makes the model care about fine distinctions, while a large $\tau$ makes it more forgiving.

• Downstream compatibility. Train $z_t$ as the state for a control or planning task. When the latent is the only predictive path, the task loss directly rewards retaining information needed for that task. It does not guarantee that every decision-relevant fact survives finite data, capacity, and optimization.

Practical systems often combine a subset of these objectives with task-specific weights. Reconstruction can provide a useful auxiliary signal, cross-prediction rewards predictable shared content, contrastive alignment organizes joint embeddings, and downstream tasks reward decision relevance. None guarantees a shared, stable, useful latent by itself, and the best combination depends on the modalities and downstream use.

An important empirical test is tolerance to missing inputs, though it measures ablation tolerance rather than proving that a latent is shared. At test time, inputs may be occluded, corrupted, or unavailable on a platform. Modality dropout exposes a model to many missingness patterns and can improve performance on covered or similar subsets. Architectures can also use fixed latent queries that attend to available tokens, or fixed modality slots with masks. Neither mechanism guarantees graceful degradation on unseen missingness patterns.

Empirical tolerance varies substantially with the dataset, task, fusion architecture, training modality set, and evaluation missingness pattern. In some trained systems, a task-specific subset of informative modalities preserves much of full-input performance, but the retained fraction depends on the dataset, task, architecture, and missingness training.

A simple reliability-weighted fusion model shows the shape of graceful degradation. The next cell computes the posterior standard deviation of a shared latent as modalities are removed in different orders.

This calculation assumes conditionally independent, unbiased Gaussian measurements of one scalar latent with known precisions. It shows that posterior uncertainty is dominated by the assigned high-precision measurements; it does not establish redundancy among real sensor modalities. Deployment sensor choices therefore require task-specific ablations rather than a general rule that fewer sensors will suffice.

In this chapter, an omnimodal world model means an any-to-any generative model designed to accept arbitrary supported input subsets and produce arbitrary supported output subsets. That definition is stronger than merely handling many modalities or learning a shared embedding. Gato and Unified-IO were early unified token-based multimodal systems: modality-specific serialization or tokenization feeds shared transformer processing. ImageBind instead aligns separate modality encoders in one embedding space and is not an arbitrary-modality generator. These systems anticipated parts of later any-to-any designs, but do not all satisfy the strong definition used here. In a dynamics configuration, a unified token model can predict trained future modality tokens from supported histories and actions.