Hierarchical, Hybrid, and Omnimodal Models

A household robot wakes at 7:00 AM, unloads the dishwasher, hears you ask for coffee, and gets to work. It plans a sequence of steps that unfold over minutes: find the mug, walk to the coffee maker, fill the reservoir, grind beans, brew. Each step is executed by a fast motor-control loop that issues torque commands on a millisecond timescale. Along the way it fuses sight, sound, touch, and proprioception into a coherent picture of the kitchen. Different subsystems handle navigation, manipulation, and language, passing control as the task changes.

Consider how much that scenario asks of the complete agent. Its planner or task memory must preserve a goal for minutes while its controllers react to millisecond-scale events. It typically benefits from object-level or relational information for planning and continuous contact information for contact-rich manipulation, whether those signals are explicit or encoded across cooperating components. A termination rule or policy must detect when a subtask is finished and decide when to hand control between high-level skills and lower-level controllers. The system must also detect and respond safely when a camera is occluded or a microphone becomes uninformative. Trained missing-modality inference, filtering, and validated fallback behavior can improve tolerance to missing inputs, but do not by themselves guarantee safety. A monolithic fixed-cadence architecture can be inefficient and difficult to train for this combination, which motivates explicit abstraction and modularity.

Single-level world models, such as one recurrent state-space model from Part V, Ch 1 or one autoregressive predictor, often face difficult efficiency and error-propagation tradeoffs on this scenario when they lack explicit temporal abstraction. Temporal depth, representation type, modality fusion, and parameter sharing are distinct but interacting design axes: a temporally flat model can use discrete tokens, modality-specific encoders, or routed experts, while a hierarchy can still use continuous states and shared parameters. The architectural families in this chapter address those axes separately, though choices along one axis can constrain compute, interfaces, optimization, and useful choices along the others. A fixed-rate model may repeatedly recompute expensive slow context. An unstructured continuous latent can encode a fact such as "the mug is on the table," but does not expose relations and logical constraints as directly as a symbolic representation. Naive early fusion asks one encoder to accommodate signals with different scales, rates, and noise. Fully shared parameters can also invite interference across tasks, though flat models may instead use task heads, adapters, or experts.

Long-horizon manipulation, sim-to-real robotics, multimodal perception, and multi-task control expose overlapping limitations that can motivate temporal abstraction, hybrid modeling, modality-aware fusion, or modular parameters. None of these settings maps exclusively to one response. The useful point is that the responses compose: they are reusable ingredients rather than four mutually exclusive designs.

This chapter develops four architectural families that together lift those limitations:

• Hierarchical models organize states or computation across temporal, spatial, or abstraction levels; a given design need not use every kind of scale.
• Hybrid models combine learned components with symbolic reasoning or with hand-crafted simulators, using each where it is strongest.
• Omnimodal models in this chapter support broad or any-to-any combinations of supported input and output modalities; the term is not yet standardized across the literature. Shared latent states and graceful missing-modality behavior are useful designs, not consequences of the label alone.
• Modular models divide computation among components. Mixture-of-experts models are one modular family, using learned routers; other modular systems use fixed or symbolic composition.

These four families are not rivals. Modern systems can stack them: a hierarchical planner whose slow level uses a symbolic simulator and whose fast level uses sensor-specific experts around a multimodal latent. The rest of this chapter builds intuition for each, then compares a window-conditioned context model with a separate routed model.

As you read, it helps to keep three questions in mind for every architecture. What assumption of the flat model does it relax? What new failure mode does it introduce in exchange? And what quantity should we measure to tell whether it is helping? Most of the practical wisdom in this area lives in those three questions rather than in the architectures themselves.

A useful household-robot architecture must span multiple functional timescales and often benefits from predictions and decisions at several levels of abstraction, whether those levels are explicit, implicit, or distributed across cooperating models and controllers. "What will the room look like in thirty seconds if I keep pouring?" is a slow, coarse prediction. "What torque should the wrist apply this millisecond?" is a fast control decision. The relevant state changes at different rates. A fixed-rate model can waste computation if it repeatedly recomputes expensive slow context at the fast control rate, and recursive prediction exposes it to compounding rollout error.

Recall from Part I, Ch 6 that when we roll a learned model forward, each predicted state becomes the input to the next prediction. Consider true and predicted rollouts that start from the same state and use the same action sequence. Suppose the transition is uniformly $L$-Lipschitz in state over the rollout region and the one-step model discrepancy is uniformly bounded by $\varepsilon$ in the same norm. Their error then obeys $e_{h+1} \leq L e_h + \varepsilon$ with $e_0 = 0$. Under these conditions, the worst-case envelope can remain bounded, grow linearly, or grow geometrically across $H$ steps. If the transition amplifies differences, the bound compounds geometrically; at unit gain it grows linearly; if the map is contractive, it saturates. A fast fixed-step model chains more learned transitions over a fixed physical horizon, though cadence is not itself the cause of the error. With initial error $e_0 > 0$, the same bound also contains the propagated term $L^H e_0$.

The compounding effect is easiest to see directly. The next cell solves that recurrence for a few values of $L$, holding the uniform per-step discrepancy fixed and using $e_0 = 0$.

Temporal hierarchy reduces the frequency of high-level goal or option decisions while retaining fast low-level control. If a subgoal latent updates every $K$ fast steps, a five-minute plan at one hundred control updates per second can contain three hundred one-second macro-decisions instead of thirty thousand primitive decisions. This shortens the high-level search chain. It can reduce compounding prediction error when coarse-transition error remains small enough and fine-level error is controlled, for example by local rollouts or regular correction from observations; hierarchy does not guarantee an empirical advantage.

A canonical reinforcement-learning formalization of temporal abstraction is the options framework of Sutton, Precup, and Singh (1999). The idea is to let the agent choose not just individual actions but whole temporally extended behaviors, and to reason about those behaviors as if they were single actions. Let $\mathcal{S}$ and $\mathcal{A}$ denote the underlying MDP's state and action spaces. An option is a triple $(\mathcal{I}, \pi, \beta)$ where:

• $\mathcal{I} \subseteq \mathcal{S}$ is the set of states in which the option may be started.
• $\pi(a \mid s)$ is the option's internal action distribution: $\pi(a \mid s) \in [0,1]$ and $\sum_{a \in \mathcal{A}} \pi(a \mid s)=1$ for each $s$.
• $\beta: \mathcal{S} \to [0,1]$ is the termination probability.

Read the triple as a small policy with a guard rail. The initiation set $\mathcal{I}$ says where the option makes sense, the internal policy $\pi$ says what to do while it runs, and the termination probability $\beta$ says when to stop. When the option is chosen at time $t$, it executes its internal policy until $\beta$ fires. Assuming the option terminates almost surely, its duration is a positive integer-valued random variable $\tau \in \{1, 2, \dots\}$; depending on the option and environment, $\tau$ may be stochastic or degenerate at a fixed duration. This turns the Markov decision process (MDP) into a semi-Markov decision process (SMDP), where the value of taking option $o$ in state $s$ is

where:

• $Q(s,o)$: the expected return from taking option $o$ in state $s$.
• $s$: the current state.
• $s_t$: the state at time step $t$.
• $o$: the option.
• $o_t$: the option chosen at time step $t$.
• $\tau$: the random duration of the option.
• $t$: the current time step.
• $\gamma$: discount factor.
• $r_i$: reward at time step $i$.
• $i$: time index running from $t+1$ to $t+\tau$.
• $V$: continuation value under the policy followed after the option terminates.
• $s_{t+\tau}$: the state after option $o$ terminates.

Here the expectation is over the option's internal action choices, the environment's transitions and rewards, and the random termination time.

Compare this to an ordinary one-step action value, which contains one immediate reward plus one continuation value discounted by $\gamma$. The option-value function instead sums the rewards earned during the whole option and then discounts the continuation value by $\gamma^{\tau}$, the discount over the option's realized duration. Different options therefore carry different effective continuation discounts, and the high-level policy chooses temporally extended actions rather than atomic ones. For $0<\gamma<1$, continuation value after a longer realized duration receives a smaller weight.

The effective discount for an option is $\gamma^{\tau}$, so duration directly changes the weights the high-level policy compares.

Getting the discounting right is essential because an incorrect duration discount can distort option values whenever it changes the weight on a nonzero continuation or reward term. For example, replacing $\gamma^\tau$ with $\gamma$ overweights positive continuation value after long options; the direction of the resulting preference also depends on within-option rewards and the sign of the continuation value.

Several hierarchical world-model architectures implement temporal abstraction with multiscale latent states, latent skills, or latent goals rather than literal call-and-return option triples. One two-level generative design stacks latent variables that update at different rates:

• A slow latent $z^{s}_k$ that changes once every $K$ fast steps, indexed by macro-time $k$.
• A fast latent $z^{f}_t$ that updates every step and receives $z^{s}_{\lfloor t/K \rfloor}$ as conditioning context.

When training makes it encode persistent behavior-level context, the slow latent can play a role analogous to option identity or high-level intent. The fast latent updates each step and can represent variation that the persistent context does not explain. Conditioning the fast transition on the current slow latent lets its predictions specialize to that context.

The joint generative factorization is

where:

• $z^{f}_{1:T}$: sequence of fast latent states over fast time steps $t = 1, \dots, T$.
• $z^{s}_{1:L}$: sequence of slow latent states over slow time steps $k = 1, \dots, L$.
• $z^{f}_0$: initial fast latent state before the sequence.
• $z^{s}_0$: initial slow latent state.
• $L$: number of slow steps (macro-steps).
• $K$: number of fast steps per slow step, so $T = L \times K$.
• $a_t$: action at fast time step $t$; the joint distribution is conditioned on the action sequence $a_{0:T-1}$.
• $p$: probability distribution.
• $z^{s}_k$: the slow latent state at macro-time step $k$.
• $z^{f}_t$: the fast latent state at fast time step $t$.
• $\lfloor \cdot \rfloor$: the floor function, giving the macro-time index for fast time $t$.

The outer product over $k$ runs over macro-steps, and the inner product over $t$ runs over the $K$ fast steps inside each macro-step. Reading the factorization from left to right, the slow equation defines the temporal coarsening; the fast equation defines the fine-grained dynamics conditioned on the current slow context. Notice how the slow transition $p(z^{s}_{k+1} \mid z^{s}_k)$ does not depend on the actions at all in this minimal form: it is a coarse drift model of how the sub-goal evolves, while the fast level absorbs the effect of actions. In richer variants the slow transition is also conditioned on an abstract action or goal, but the essential split is that the slow level summarizes and the fast level reacts.

Encoders can mirror this structure. A training-time smoothing encoder $q(z^{s}_k \mid o_{kK : (k+1)K - 1})$ aggregates an entire observation window, while a fast encoder $q(z^{f}_t \mid o_t, z^{s}_{\lfloor t/K \rfloor})$ captures per-step detail. Here $o_t$ is the observation at fast time step $t$, and $o_{kK:(k+1)K-1}$ is the corresponding window for slow step $k$. Because this smoothing form uses the completed window, it is not a causal online encoder for early steps in that same block. An online system can instead infer $z^s_k$ from the preceding completed window $o_{(k-1)K:kK-1}$ before block $k$, or delay use of the summary until its window is complete. In either case, the asymmetry is the point: the slow encoder compresses a batch of observations, while the fast encoder sees a single observation together with the summary currently in force. A useful analogy is human scene perception: an initial glance can provide scene gist that helps guide subsequent fixations toward task-relevant detail.

FeUdal Networks (Vezhnevets et al., 2017) use a lower-temporal-resolution Manager whose dilated recurrent state and multi-step objective produce persistent goal conditioning, while a Worker chooses primitive actions each step. The Manager and Worker are learned concurrently with separate objectives, without propagating gradients between the two modules. Director adds hierarchical policies over the latent state space of a learned pixel-based world model: a Manager selects latent goals every eight steps and a Worker chooses primitive actions to reach them. These are policy hierarchies rather than evidence that Director contains slow and fast world-model state transitions. More generally, persistent context can spare a fast predictor from rediscovering behavior-level information at every step, but smaller residuals are an empirical outcome rather than a guarantee.

Temporal scales have a spatial counterpart. A building-scale robot may use room-level topology for navigation together with millimeter-scale contact geometry and high-rate force or torque signals for manipulation. With a fixed-budget dense representation, one resolution is inefficient across these scales: a fine global grid allocates detail where it is irrelevant, while a coarse grid misses local contact structure. A dense volumetric representation of an entire building at one-millimeter resolution is computationally impractical, so sparse or hierarchical representations allocate fine resolution selectively. Two common strategies are:

• Coarse-to-fine latent pyramids. The state has a low-resolution "global" component, for example a volumetric or object-level map, plus fine local patches that are only queried where the agent is currently interacting. The global component ignores details that do not matter; the local component ignores the rest of the world. The two are linked by the agent's current focus, so the fine patches are always embedded in a coarse frame of reference.
• Multi-resolution observation models. The decoder reconstructs a coarse observation everywhere and a fine observation only inside an attention window around the agent. Biological foveation offers an analogy: cone density and visual acuity peak in a small central region, while gaze shifts sample the rest of the scene. Most retinal photoreceptors, predominantly rods, are outside the fovea.

Structure, not resolution alone, should drive the transition model. Regular-grid maps often suit convolution because of spatial locality and translation equivariance; topological maps often suit graph networks because they encode adjacency and relations. Continuous-time models can help with irregularly timed observations or queries, while contact dynamics may additionally require event-aware modeling and accurate numerical integration. Graph networks can themselves be effective for contact dynamics. The practical rule is to match an inductive bias to the representation's actual topology, smoothness, events, and sampling process.

Hierarchical models add complexity, and that complexity has costs. A slow policy can propose subgoals that a fast policy cannot realize. Termination can be poorly calibrated, causing premature or excessively long execution. When observations support several contexts, slow-state inference can become uncertain or lock onto the wrong one unless history, actions, or later observations correct it. A window encoder can also exploit spurious phase or boundary cues. Window length itself becomes a shortcut when variable lengths or padding patterns correlate with the target. Such correlations fail when task periods or window construction change.

Several of these failure modes can manifest as disagreement between the levels, with no training signal forcing them to reconcile. Silent inconsistency is especially difficult because it may not raise an obvious error. The slow level believes the robot is near the mug, the fast level believes it is near the sink, and the model produces plausible but wrong predictions for both. This drift can remain hidden from aggregate one-step validation loss, so cross-level agreement should be evaluated explicitly or included as an equivalent targeted validation term.

Cross-level consistency losses address the disagreement subset, not every failure above. If the slow level predicts a projected coarse summary of the next window and the fast level predicts its fine state sequence, applying the same projection to the fast prediction gives a shared quantity on which the two levels can be compared. Training on that agreement supplies a direct gradient to the slow level. Subgoal feasibility instead calls for reachability or capability checks; termination needs dedicated closed-loop stopping metrics; and window shortcuts need distribution-shift tests and ablations. Joint end-to-end training and a bounded window matched to the task's timescale may support these safeguards, but do not replace them.

Pure learned models offer flexible function approximation, but their data requirements depend on architecture, priors, pretraining, and task complexity. With sufficient capacity, suitable data, and successful optimization, they can approximate a broad range of patterns, though they do not automatically enforce logical consistency. Symbolic systems can enforce their encoded rules exactly, while hand-crafted simulators can be accurate within their assumptions and numerical tolerances. Both become brittle when those assumptions fail. Hybrid models combine these strengths: learned components can handle perception, uncertainty, and difficult-to-model residuals, while symbolic or simulation components can implement explicit planning rules, logical constraints, and well-understood physics. The central design choice is where to draw the boundary and how information should cross it.

There is no single way to combine neural and symbolic components. In practice the integrations fall into three broad patterns, distinguished by how much information flows between the learned and non-learned parts and in which direction.

Pipeline hybrids. One common pipeline hybrid uses learned perception to map raw observations into a symbolic state of typed objects, predicates, and relations. In the neural-symbolic pipeline described here, a symbolic reasoner plans over that state and a learned controller executes the plan; other pipeline hybrids may use classical or hand-designed controllers. Shakey was an early precursor to this perception-model-planning-execution pattern, and some modern task-and-motion planning systems use learned perception or predicates as a front end. Interfaces between modules are discrete and non-differentiable: a planning cycle may run from perception to planner to controller, while execution feedback triggers new perception and replanning. The symbolic layer makes assumptions inspectable. The cost is that downstream plan failure supplies no end-to-end gradient to correct a perception mistake.

Differentiable hybrids. The symbolic component is made differentiable (or a soft relaxation is used) so gradients can flow through it. Differentiable planners, differentiable logic layers, and differentiable physics simulators all live here. The advantage is end-to-end learning: because the gradient reaches the symbolic part, the learned perception can be shaped by the downstream planning objective, closing the loop that the pipeline pattern leaves open. The disadvantage is that soft relaxations can be approximate and the optimizer may drift into regimes where the relaxation is unfaithful. A differentiable planner that relaxes discrete decisions into continuous weights might, during training, settle into a continuous solution that has no valid discrete counterpart, and the resulting policy can be brittle at deployment.

Residual hybrids. A hand-crafted model supplies a nominal prediction, and a neural network predicts an additive correction; when the nominal model is sufficiently accurate, it may account for most of the final prediction. During training, the residual is fit to the model's mismatch. This pattern sidesteps the optimization hazards of the differentiable hybrid by treating the non-learned part as fixed: the simulator is evaluated in the forward direction and the network learns its prediction error. Multi-step use is open loop only when predicted states are recursively fed back without observation correction. Let $f_{\text{sim}}$ be the simulator and $g_\theta$ the neural correction. The hybrid dynamics are

and the training loss is

where:

• $\hat{s}_{t+1}$: the predicted next state.
• $f_{\text{sim}}(s_t, a_t)$: the simulator's predicted next state.
• $g_\theta(s_t, a_t)$: the neural residual correction.
• $\theta$: parameters of the residual correction network.
• $t$: time step index.
• $s_t$: the current state.
• $a_t$: the action at time $t$.
• $s_{t+1}$: the true next state.
• $\mathcal{L}(\theta)$: the mean squared residual loss used to train the correction network.
• $\mathbb{E}_{(s_t, a_t, s_{t+1})}$: expectation over the training distribution of transitions.
• $\|\cdot\|_2^2$: squared Euclidean norm.

Read the loss as asking the network to explain only the part of the transition that the simulator misses. When the simulator's expected error is small relative to the scale of the chosen full-model target under the same normalization and distribution, the residual learner receives a smaller-magnitude target. Residual learning is generally more favorable when the remaining mismatch is predictable, observable, and representable by the learner.

The practical benefit is that the residual network can see a smaller target. The next cell isolates that scale effect by simulating residual targets for an accurate simulator and a biased simulator while holding the true transition distribution fixed. It does not test whether either residual is predictable from the model inputs. The sample is drawn with seed 20240517.

Residual hybrids are used in robotics when a calibrated simulator captures much of the target dynamics and a learner can model the remaining friction, contact, or deformation effects. Under those conditions, $g_\theta$ receives a smaller target than a model trained to predict the full transition from scratch. The gain depends on the base simulator and on whether its mismatch is observable and representable.

A differentiable simulator exposes gradients of simulated states with respect to control inputs and physical parameters. Instead of training a separate learned model, the simulator can serve as the world model for well-understood parts of the scene. Examples include DiffTaichi, Dojo, legacy Brax physics pipelines, and MuJoCo's current MJX-JAX implementation; MJX-Warp does not currently support automatic differentiation. These gradients support parameter fitting and, when reliable, end-to-end policy or planner optimization through simulation.

The cost is that simulator gradients can be biased or numerically delicate near contacts and other discontinuities. An infinitesimal input change may alter the contact sequence, producing a gradient that is uninformative for the intended update. When contact gradients are unreliable, a residual or other forward-only hybrid that treats the simulator as fixed avoids backpropagating through those gradients and may be preferable; its reliability still has to be established for the particular simulator and task.

The relational planning representation used here describes entities through predicates such as on(mug, table), inside(bean, grinder), and reachable(gripper, handle); a domain may also assign entity types. Neuro-symbolic systems can learn such predicates from data. A perception encoder embeds detected objects, and a relation head produces soft truth values that a planner can query. A differentiable training path can use those soft values, while a discrete planner may threshold them at inference time.

The key modeling benefit is systematic generalization. Because lifted rules are stated over variables, a symbolic planner can apply them to unseen object combinations when the symbolic state is accurate, all predicates, types, and constraints hold, its operators cover the case, and search succeeds. Many standard neural architectures perform poorly on controlled compositional splits unless their architecture, objective, or training distribution supports systematic recombination. As covered in Part IV, Ch 6, explicit relational structure is one way to encode that support.

A practical hybrid pattern is simulator-in-the-loop training. When real interaction is expensive, simulation can supply much of the training data while limited real data supports calibration, residual learning, or validation. A learned residual can reduce systematic simulator error; domain randomization instead trains the policy to tolerate a family of simulator variations. The physics engine supplies encoded structure, and the learned component addresses selected gaps. A corrected model may reduce exploitable bias, but policy optimization can still seek its remaining errors. Uncertainty penalties, robust optimization, constrained rollouts, and real-system validation are separate safeguards against that failure.

A robot may receive vision, depth, touch, proprioception, audio, and language. A shared-latent multimodal world model gives those signals a common predictive state, allowing one modality to help disambiguate another. Separate private streams can also exchange information through cross-attention, so one literal shared bottleneck is not the only solution. Tolerance to missing inputs likewise requires suitable architecture and training; the label omnimodal does not guarantee it. The analogy to classical filtering is useful but limited: a Kalman filter combines measurements under an explicit probabilistic model, whereas a neural fusion rule and its state are learned.

Three common fusion strategies are early fusion, late fusion, and intermediate exchange through mechanisms such as cross-modal attention. The first two describe placement near the input or output, while attention is an interaction operator that can be inserted at different depths and can coexist with either layout. Both placement and mechanism affect robustness and the kinds of correlations the model can capture.

Early fusion. A common design concatenates modality streams near the input and passes them through a shared encoder. Without explicit reliability weights, that encoder must infer each modality's influence during training. Masks, imputation, or fixed missing-value slots can make the design tolerate absent inputs; untrained omissions remain a distribution shift.

Late fusion. Each modality is encoded independently, and modality-specific predictions are combined at the output. Dropping an encoder is structurally simple, though the combiner still needs training for missing inputs. Because interaction occurs only after separate interpretations have formed, this design has less opportunity to model fine cross-modal correlations.

Intermediate exchange with cross-modal attention. In one common design, modalities receive separate preprocessing or encoders before attention exchanges information among their representations. Flamingo is a direct visual-to-language cross-attention example; Perceiver IO is a related multimodal architecture that routes arbitrary inputs through a latent bottleneck. This exchange can occur at one or several depths rather than defining a single placement category. Cross-modal attention adds interaction cost relative to independent encoders plus a simple combiner. Dense pairwise attention does not scale gracefully by itself, but sparse, asymmetric, or bottlenecked variants can. Attention masks make absent streams structurally representable, but reliable performance generally requires training on the relevant missingness patterns.

A shared latent $z_t$ with $M$ modalities is inferred from

where:

• $q(z_t \mid \cdot)$: the approximate posterior over the shared latent given the available modality observations.
• $z_t$: shared latent state at time $t$.
• $o_t^{(m)}$: observation from modality $m$ at time $t$.
• $M$: number of modalities.
• $t$: time step index.
• $m$: modality index running from $1$ to $M$.
• $e_m$: per-modality encoder for modality $m$.
• $\text{Fuse}$: fusion function (concatenation, attention, or a learned pooling operator). The decoder mirrors this by producing modality-specific heads from the same $z_t$.

The equation says that each available modality is encoded and that a fusion operator maps the resulting embeddings into a shared latent. Modality-specific heads decode from $z_t$. The latent is a strict shared bottleneck only if it is the sole predictive path from inputs to outputs. Private or skip paths do not make the latent unshared, but they make the bottleneck nonexclusive.

What forces the shared latent to be shared? Left to their own devices, per-modality decoders can send most information through private paths and never develop a common state. A pure reconstruction objective does not prevent this because each decoder only has to rebuild its own input. Four objectives encourage fusion:

• Reconstruction. Each modality head must reconstruct its own input from $z_t$. Simple but weak on its own for encouraging cross-modal content, since it can be satisfied without any cross-modal interaction.
• Cross-modal prediction. Predict one modality from the others via $z_t$: for example, recover depth from RGB and proprioception, or audio from video. This rewards content that is predictable across modalities. It cannot recover target information that is statistically absent from the inputs.
• Contrastive alignment. Raise the similarity of matched cross-modal pairs relative to sampled negatives, thereby lowering the negatives' relative scores. A common formulation is the InfoNCE loss

where:

• $\mathcal{L}_{\text{InfoNCE}}$: the InfoNCE loss value, minimized when matching pairs have high similarity relative to negative samples.
• $z_t^{(a)}$: embedding of modality $a$ at time $t$.
• $z_t^{(b)}$: embedding of modality $b$ at time $t$.
• $z_j^{(b)}$: embedding of modality $b$ for the $j$-th negative sample.
• $j$: negative-sample index.
• $N$: number of negative samples.
• $t$: time step index.
• $\text{sim}(\cdot, \cdot)$: similarity function, usually cosine similarity.
• $\tau$: temperature parameter (smaller $\tau$ sharpens the distribution over negatives).

The numerator scores the positive pair, while the denominator sums the positive and negative scores, so minimizing the loss raises the positive pair's similarity relative to the negatives. The temperature $\tau$ controls how sharply the loss penalizes near-miss negatives: a small $\tau$ makes the model care about fine distinctions, while a large $\tau$ makes it more forgiving.

• Downstream compatibility. Train $z_t$ as the state for a control or planning task. When the latent is the only predictive path, the task loss directly rewards retaining information needed for that task. It does not guarantee that every decision-relevant fact survives finite data, capacity, and optimization.

Practical systems often combine a subset of these objectives with task-specific weights. Reconstruction can provide a useful auxiliary signal, cross-prediction rewards predictable shared content, contrastive alignment organizes joint embeddings, and downstream tasks reward decision relevance. None guarantees a shared, stable, useful latent by itself, and the best combination depends on the modalities and downstream use.

An important empirical test is tolerance to missing inputs, though it measures ablation tolerance rather than proving that a latent is shared. At test time, inputs may be occluded, corrupted, or unavailable on a platform. Modality dropout exposes a model to many missingness patterns and can improve performance on covered or similar subsets. Architectures can also use fixed latent queries that attend to available tokens, or fixed modality slots with masks. Neither mechanism guarantees graceful degradation on unseen missingness patterns.

Empirical tolerance varies substantially with the dataset, task, fusion architecture, training modality set, and evaluation missingness pattern. In some trained systems, a task-specific subset of informative modalities preserves much of full-input performance, but the retained fraction depends on the dataset, task, architecture, and missingness training.

A simple reliability-weighted fusion model shows the shape of graceful degradation. The next cell computes the posterior standard deviation of a shared latent as modalities are removed in different orders.

This calculation assumes conditionally independent, unbiased Gaussian measurements of one scalar latent with known precisions. It shows that posterior uncertainty is dominated by the assigned high-precision measurements; it does not establish redundancy among real sensor modalities. Deployment sensor choices therefore require task-specific ablations rather than a general rule that fewer sensors will suffice.

In this chapter, an omnimodal world model means an any-to-any generative model designed to accept arbitrary supported input subsets and produce arbitrary supported output subsets. That definition is stronger than merely handling many modalities or learning a shared embedding. Gato and Unified-IO were early unified token-based multimodal systems: modality-specific serialization or tokenization feeds shared transformer processing. ImageBind instead aligns separate modality encoders in one embedding space and is not an arbitrary-modality generator. These systems anticipated parts of later any-to-any designs, but do not all satisfy the strong definition used here. In a dynamics configuration, a unified token model can predict trained future modality tokens from supported histories and actions.

Increasing appropriately trained model capacity often improves performance, but larger dense models increase memory and inference cost and may be impractical to deploy. A sparse mixture-of-experts (MoE) layer can decouple total parameter count from per-example expert FLOPs by keeping many parameter sets but activating only a few per input. Parameter storage, routing, dispatch, communication, and memory traffic can still grow with the number and placement of experts. In world models the pattern is doubly useful: different regions of state space, different modalities, or different task contexts call for different dynamics, and a router can learn to select the right one. The first benefit is computational: a model that would be too slow to run in full can reduce expert computation by evaluating only a fraction of its experts per input. The second is an architectural inductive bias: specialization may reduce interference by assigning different regimes to different experts, although it does not automatically enlarge the representable function class relative to an unrestricted sufficiently large monolithic network.

A mixture of $K$ experts with a learned router $g$ computes

where:

• $p(y \mid x)$: the output distribution over $y$ given input $x$.
• $x$: input to the mixture.
• $y$: output variable.
• $K$: number of experts.
• $k$: expert index running from $1$ to $K$.
• $g_k(x)$: router weight for expert $k$ given input $x$.
• $f_k(y \mid x)$: the predictive density produced by expert $k$ for output $y$ given input $x$.

The output density is a mixture of expert densities with input-dependent weights. Unlike an ordinary finite mixture with input-independent component proportions, an MoE uses a learned gating network whose weights depend on $x$. Influential sparse Transformer MoEs commonly use a lightweight linear router, though more structured routers are possible:

where:

• $x$: input feature vector passed to the router.
• $g(x)$: router probability vector over experts.
• $W_g$: weight matrix of the router.
• $b_g$: bias vector of the router.
• $\text{softmax}$: softmax function.

The softmax turns raw scores into non-negative weights that sum to one, giving a valid mixture density. Keeping the router lightweight limits overhead. Specialization depends more directly on its inputs, objective, load balancing, capacity, and optimization than on router size alone.

Dense MoE computes all experts and combines them, so its expert compute grows with $K$ and it does not provide the conditional-compute scaling advantage of sparse MoE. A common sparse token-choice MoE selects the top $k$ experts (often $k = 1$ or $2$) and may renormalize weights over the selected set; other sparse routing schemes include top-1 Switch routing, expert-choice routing, threshold routing, and fixed or learned alternatives. Sparse MoE grows capacity (number of parameters) without growing per-example expert compute (which scales with $k$, not $K$). This decoupling is the central practical advantage: you can add experts to raise the model's total capacity while keeping the expert cost per example essentially flat, because only the selected experts run.

The router can be trained in several ways, and the choice trades off stability, compute, and gradient quality.

• Learned linear gate. Simplest and most common. Backpropagates through $g(x)$, so the router is trained jointly with the experts. Every expert receives a task gradient weighted by its router probability, giving continuous routing gradients, although saturated probabilities or optimization dynamics can still produce instability or negligible gradients.
• Top-1 or top-k hard routing. The router still produces soft logits, but only the top-$k$ experts are evaluated and only those receive gradients. Sparse but potentially unstable, because hard assignments can starve experts of data: an expert that is never selected never receives a gradient and can never improve enough to be selected.
• Noisy top-k. Add Gaussian noise to the router logits before top-$k$ selection so that different samples can explore different experts (Shazeer et al., 2017). With a nonzero noise scale large enough relative to the logit margins, this can reduce the risk that a small early advantage starves an expert of future updates. It may improve load distribution, but does not guarantee balanced traffic; explicit balancing objectives and capacity mechanisms are commonly used alongside it.
• Soft relaxations. Dense soft routing computes all experts with continuous weights and trades expert compute for smoother routing gradients. Differentiable soft-top-k relaxations can instead be truncated to preserve sparse execution while approximating discrete selection.

Without effective mitigation, learned sparse token-choice routers can develop severe load imbalance or collapse, with a few experts absorbing most traffic. The collapse can be self-reinforcing, because experts that start slightly ahead receive more data and improve faster, which makes them even more likely to be chosen next. A common mitigation is an auxiliary load-balancing loss. For the common top-1 Switch-style form, let $n_k$ be the fraction of tokens routed to expert $k$ in a batch and let $p_k$ be the average router probability for expert $k$. Because each token receives one hard assignment, both vectors sum to one, and the auxiliary loss is

where:

• $\mathcal{L}_{\text{balance}}$: auxiliary load-balancing loss used to discourage concentrated routing.
• $K$: number of experts.
• $k$: expert index running from $1$ to $K$.
• $n_k$: fraction of tokens assigned to expert $k$ by top-1 routing in the batch.
• $p_k$: average router probability for expert $k$ over the batch.

Uniform top-1 routing is a useful reference point. If $n_k = p_k = 1/K$ for every expert, then

The loss is added to the task objective with a small coefficient. A concentrated router instead makes the hard token fractions and average soft probabilities large for the same experts, increasing their dot product.

The auxiliary loss has a simple shape as routing concentrates. The next cell computes the top-1 load-balancing loss for a four-expert router as the highest-weight expert receives an increasing fraction of the traffic.

This auxiliary objective is a heuristic rather than a guarantee of uniform use because $n_k$ and $p_k$ are different statistics; implementations also rely on routing-capacity constraints or alternative objectives. If its coefficient is too small, concentrated routing may persist. If it is too large, the router can be pushed toward uniformity even when the data favors specialization.

Routing is not only about scaling. A decomposed world model can capture qualitatively different dynamics. Different regions of state space, different modes of a hybrid system, different tasks, or different modalities may benefit from distinct dynamics. A router that learns to send samples to the right expert effectively performs an implicit soft discretization of state space. Even when conditional-compute savings are not the main concern, explicit routing can provide a useful modular inductive bias and may improve optimization or sample efficiency by assigning different regimes to different experts. It does not automatically enlarge the representable function class relative to an unrestricted sufficiently large monolithic network.

This is related to the switched systems of classical control, where $\dot{s} = f_{\sigma(t)}(s, a)$ and $\sigma(t)$ identifies an active discrete mode. Here $\dot{s}$ is the state derivative, $a$ is the control, and $f_{\sigma(t)}$ is mode-specific dynamics. A hard-routed MoE can be interpreted as learning a state/action-dependent mode selector; a soft MoE instead learns a weighted blend. Under a hard, mode-specialized interpretation, router chattering near a boundary is one relevant control concern.

Expert capacity should suit the complexity of the regime it handles. Near mode boundaries, routing and expert outputs should also avoid spurious jumps or rapid chattering while preserving real discontinuities in the system. Noise encourages routing exploration; entropy regularization discourages overly sharp probabilities; soft kernel-based routers can interpolate continuously. Output compatibility, hysteresis, or explicit continuity regularization may still be needed. The right treatment depends on whether the physical transition is continuous at that boundary.

• Multi-embodiment robots. Different bodies can share perception while using embodiment-specific dynamics. If embodiment information is omitted or aliased in the represented input, a point-loss shared model can average incompatible body-specific transitions. Experts routed by embodiment are one remedy; conditioning a sufficiently expressive dense model on morphology is another.
• Multi-task control. Navigation, manipulation, and speech respond to inputs differently; separate experts reduce interference. Task interference is the reason a single network trained on many tasks can lose accuracy on each of them, and routing sidesteps it.
• Heterogeneous modalities. Different sensors have different statistics and noise; separate experts per modality concentrate compute where it is needed.
• Regime-switching physical systems. Contact, free-fall, and sliding dynamics have qualitatively different structure. A learned soft boundary can track gradual transitions when the dynamics are continuous and the router is trained on the boundary region; discontinuous transitions may require hard, hysteretic, or event-aware treatment.

We now compare a flat baseline with two separate window-conditioned alternatives on a controlled 2D system with a hidden context. One compresses a history window into a continuous context latent; the other uses the window to route across experts. The first borrows a representation idea from hierarchy but recomputes its context every step, so it is not a true multirate temporal hierarchy. The system has two contexts that change how actions affect velocity:

• Context A is a damped point mass: each velocity component decays toward zero, and actions add acceleration.
• Context B adds a rotational coupling term in velocity space, as if the agent is inside a rotating frame.

Context is fixed per episode and its label is not supplied to the models. A flat model sees only (state, action). A history-conditioned model and an MoE router each receive a short state window that can provide more statistical evidence about the context. Finite state history without past actions does not guarantee exact identification, and a single post-divergence state can itself carry probabilistic context information.

Both contexts agree at the initial rest state. After velocity becomes nonzero, subsequent transitions reveal their different update laws, and a longer window may add evidence. Two experts provide the cardinality needed for a possible one-to-one context alignment, but training must still discover and validate that alignment. Multi-step rollout error is therefore the main outcome, while routing statistics diagnose how the MoE uses its capacity.

We build a NumPy rollout that produces an episode given the hidden context. The system has four state dimensions (position and velocity in 2D) and two action dimensions (accelerations).

Under the matched actions shown here, context B's rotational coupling curves the velocity response and reduces displacement on average across these three pairs. A state snapshot does not contain the context label, but after the dynamics have diverged its value may contain probabilistic evidence. A recent window supplies additional evidence about how velocity has evolved.

For both window-conditioned models we build supervised examples of the form (window of recent states, current state, action, next state). The window offers evidence about which dynamics are in force, without supplying the context label or guaranteeing exact recovery.

The output confirms the expected 32-feature dimension for an eight-state, four-dimensional window. The preceding slice and flattening operation supply the full contiguous recent-state window used for context inference.

Three modules: a flat baseline that sees only (state, action), a history-conditioned model whose encoder compresses the window into a context latent, and a separate MoE with a learned router. The latter two receive more input and have more parameters than the flat baseline, so this is a diagnostic comparison, not a controlled estimate of an architectural advantage.

We share one training function across models. A lightweight forward adapter lets the flat, history-conditioned, and routed models use the same loop.

We compare training losses and multi-step rollout error on held-out test episodes.

In this run the two models perform almost identically. The comparison neither isolates a benefit from history conditioning nor tests temporal abstraction: a controlled study would add a capacity-matched flat window baseline, match compute, and train across multiple seeds.

Now let us compare the continuous context latent with a differentiable router and two experts. The experts see the same (state, action) inputs as the flat model, while the router uses the history window. The dense soft router blends both experts; it does not make a discrete mode decision, and specialization must be measured rather than assumed.

Two observations close the example. First, neither the history-conditioned model nor the MoE establishes an advantage over the flat baseline in this single run. Second, the MoE's context-conditioned mean gates do not align with the known contexts. Mode alignment is useful when interpretable recovery is an explicit goal, but it should be paired with per-example routing statistics, probes, expert ablations, and task-level rollout performance. Low alignment alone does not prove that predictions average destructively across regimes.

The key parameters for the worked history-conditioned and routed models are:

• WINDOW: Number of past states supplied to the history encoder and router. More history can add context evidence when it is relevant, but also increases input dimension and shortcut risk.
• LATENT_DIM: Dimensionality of the continuous context latent. The example does not claim that two dimensions are necessary or sufficient; that requires an ablation and a context probe.
• N_EXPERTS: Number of experts. Two experts permit a one-to-one alignment with two contexts, but the alignment still has to be learned and evaluated.
• HIDDEN: Width of the MLP hidden layers in the flat model, history encoder, context-conditioned predictor, router, and experts.
• LEARNING_RATE / BATCH_SIZE / TRAIN_EPOCHS: Optimizer step size, minibatch size, and training passes in the shared loop.
• DT: Forward-Euler simulation step. A smaller value generally reduces discretization error only when physical duration and the continuous action signal are held fixed while the step count changes accordingly.

Hierarchical, hybrid, and modular architectures add failure modes beyond those of a single-level model. Hierarchy changes credit assignment: it introduces inter-level attribution and backup-design problems, while a well-chosen temporal abstraction can shorten long-horizon credit paths. Sparse slow updates create fewer slow computation nodes, but each node may aggregate gradients from many fast-step losses; signal strength depends on the objective and gradient paths. A fast predictor can also compensate for a drifting slow state unless training or evaluation checks the intended representation. Learned termination and update boundaries may fail only in closed-loop rollouts. These are risks of temporally sparse hierarchies, not mechanisms exercised by the worked example's per-step history encoder. Evaluate them with decision-centric and cross-level metrics in addition to one-step error.

Hybrid models inherit assumptions from their non-learned parts. If a simulator is structurally wrong, an additive residual may still correct it when the mismatch is observable and representable, but the task can become harder than learning a different model or boundary. Spurious predicates and missing axioms can produce invalid plans; execution monitoring, uncertainty, and abstraction repair help limit that risk. Differentiable hybrids add another hazard because a soft relaxation's gradient need not match the hard objective. A useful structural prior can improve efficiency when the learned component can represent the remaining task-relevant mismatch. Residual learning is therefore a conditional design choice for simulator hybrids, not a rule for every hybrid architecture.

Modular and omnimodal architectures face collapse, dominance, and shortcut risks. Aggressive sparse routing can concentrate traffic without an effective balancing design. Training and inference can diverge when they use different noise, temperature, truncation, capacity rules, or input distributions. In multimodal training, the objective may over-rely on whichever stream is easiest, whether vision, audio, touch, or another modality, and a shared latent may encode cross-modal shortcuts. A reconstruction score can therefore coexist with poor control utility. Use decision-aware objectives when task relevance is the problem, and combine them with balancing, masking, missing-input tolerance, uncertainty, or anti-shortcut interventions appropriate to the observed failure.

These architectures have delivered gains under specific conditions. Temporal abstraction can extend effective planning horizons; simulator hybrids can improve real-robot data efficiency when the simulator supplies a useful prior; broad multimodal architectures support training across mixed datasets and adaptation to deployment inputs; sparse MoE layers can add parameter capacity while evaluating few experts per example. None is automatic. Each exchanges a simpler architecture for additional assumptions, diagnostics, and failure modes.

This chapter studied four architectural families that lift limitations of flat world models.

• Hierarchical models organize states or decisions across temporal, spatial, or abstraction levels. A slower update rate shortens the high-level decision chain; this can reduce compounding prediction error when coarse-transition error remains sufficiently small and fine-level error is controlled through accurate local dynamics or observation correction.
• Hybrid models combine learned components with symbolic reasoning or hand-crafted simulators. Pipeline, differentiable, and residual patterns have distinct assumptions; a residual target is attractive when a useful simulator leaves an observable, representable mismatch.
• Omnimodal models support broad input-output modality combinations. Cross-modal attention enables interaction. A task-dependent combination of reconstruction, cross-modal prediction, contrastive alignment, and decision-aware losses can encourage useful shared and complementary content, but none guarantees it alone. Masking and modality dropout can improve tolerance to missing inputs, which still depends on the missing streams and training coverage.
• Modular models include mixture-of-experts architectures with learned routers. Sparse top-k routing can grow parameter capacity while evaluating few experts, but often needs an effective routing, capacity, or balancing mechanism to reduce harmful expert collapse; expert loads should be checked empirically.

The worked example compared a per-step history-conditioned context model and a separate routed model with a flat baseline. The window-conditioned models had more input and capacity, yet neither clearly improved multi-step position error in this single run. The MoE's context-conditioned mean gates showed little alignment with the two labels; richer routing diagnostics would be needed to characterize specialization.

The architectures in this chapter sit at the end of Part V. Part VI is organized around how world models are learned: predictive and generative objectives, interaction and passive-observation data, curation and synthesis, continual learning, and scaling. Part VI opens with predictive and generative objectives and examines how objective choice shapes learned representations and downstream use.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about hierarchical, hybrid, and omnimodal world models.