Hierarchical, Hybrid, and Omnimodal Models › Summary

This chapter studied four architectural families that lift limitations of flat world models.

• Hierarchical models organize states or decisions across temporal, spatial, or abstraction levels. A slower update rate shortens the high-level decision chain; this can reduce compounding prediction error when coarse-transition error remains sufficiently small and fine-level error is controlled through accurate local dynamics or observation correction.
• Hybrid models combine learned components with symbolic reasoning or hand-crafted simulators. Pipeline, differentiable, and residual patterns have distinct assumptions; a residual target is attractive when a useful simulator leaves an observable, representable mismatch.
• Omnimodal models support broad input-output modality combinations. Cross-modal attention enables interaction. A task-dependent combination of reconstruction, cross-modal prediction, contrastive alignment, and decision-aware losses can encourage useful shared and complementary content, but none guarantees it alone. Masking and modality dropout can improve tolerance to missing inputs, which still depends on the missing streams and training coverage.
• Modular models include mixture-of-experts architectures with learned routers. Sparse top-k routing can grow parameter capacity while evaluating few experts, but often needs an effective routing, capacity, or balancing mechanism to reduce harmful expert collapse; expert loads should be checked empirically.

The worked example compared a per-step history-conditioned context model and a separate routed model with a flat baseline. The window-conditioned models had more input and capacity, yet neither clearly improved multi-step position error in this single run. The MoE's context-conditioned mean gates showed little alignment with the two labels; richer routing diagnostics would be needed to characterize specialization.

The architectures in this chapter sit at the end of Part V. Part VI is organized around how world models are learned: predictive and generative objectives, interaction and passive-observation data, curation and synthesis, continual learning, and scaling. Part VI opens with predictive and generative objectives and examines how objective choice shapes learned representations and downstream use.