Hierarchical, Hybrid, and Omnimodal Models › Hierarchical, Hybrid, and Omnimodal Models

A household robot wakes at 7:00 AM, unloads the dishwasher, hears you ask for coffee, and gets to work. It plans a sequence of steps that unfold over minutes: find the mug, walk to the coffee maker, fill the reservoir, grind beans, brew. Each step is executed by a fast motor-control loop that issues torque commands on a millisecond timescale. Along the way it fuses sight, sound, touch, and proprioception into a coherent picture of the kitchen. Different subsystems handle navigation, manipulation, and language, passing control as the task changes.

Consider how much that scenario asks of the complete agent. Its planner or task memory must preserve a goal for minutes while its controllers react to millisecond-scale events. It typically benefits from object-level or relational information for planning and continuous contact information for contact-rich manipulation, whether those signals are explicit or encoded across cooperating components. A termination rule or policy must detect when a subtask is finished and decide when to hand control between high-level skills and lower-level controllers. The system must also detect and respond safely when a camera is occluded or a microphone becomes uninformative. Trained missing-modality inference, filtering, and validated fallback behavior can improve tolerance to missing inputs, but do not by themselves guarantee safety. A monolithic fixed-cadence architecture can be inefficient and difficult to train for this combination, which motivates explicit abstraction and modularity.

Single-level world models, such as one recurrent state-space model from Part V, Ch 1 or one autoregressive predictor, often face difficult efficiency and error-propagation tradeoffs on this scenario when they lack explicit temporal abstraction. Temporal depth, representation type, modality fusion, and parameter sharing are distinct but interacting design axes: a temporally flat model can use discrete tokens, modality-specific encoders, or routed experts, while a hierarchy can still use continuous states and shared parameters. The architectural families in this chapter address those axes separately, though choices along one axis can constrain compute, interfaces, optimization, and useful choices along the others. A fixed-rate model may repeatedly recompute expensive slow context. An unstructured continuous latent can encode a fact such as "the mug is on the table," but does not expose relations and logical constraints as directly as a symbolic representation. Naive early fusion asks one encoder to accommodate signals with different scales, rates, and noise. Fully shared parameters can also invite interference across tasks, though flat models may instead use task heads, adapters, or experts.

Long-horizon manipulation, sim-to-real robotics, multimodal perception, and multi-task control expose overlapping limitations that can motivate temporal abstraction, hybrid modeling, modality-aware fusion, or modular parameters. None of these settings maps exclusively to one response. The useful point is that the responses compose: they are reusable ingredients rather than four mutually exclusive designs.

This chapter develops four architectural families that together lift those limitations:

• Hierarchical models organize states or computation across temporal, spatial, or abstraction levels; a given design need not use every kind of scale.
• Hybrid models combine learned components with symbolic reasoning or with hand-crafted simulators, using each where it is strongest.
• Omnimodal models in this chapter support broad or any-to-any combinations of supported input and output modalities; the term is not yet standardized across the literature. Shared latent states and graceful missing-modality behavior are useful designs, not consequences of the label alone.
• Modular models divide computation among components. Mixture-of-experts models are one modular family, using learned routers; other modular systems use fixed or symbolic composition.

These four families are not rivals. Modern systems can stack them: a hierarchical planner whose slow level uses a symbolic simulator and whose fast level uses sensor-specific experts around a multimodal latent. The rest of this chapter builds intuition for each, then compares a window-conditioned context model with a separate routed model.

As you read, it helps to keep three questions in mind for every architecture. What assumption of the flat model does it relax? What new failure mode does it introduce in exchange? And what quantity should we measure to tell whether it is helping? Most of the practical wisdom in this area lives in those three questions rather than in the architectures themselves.