Hierarchical, Hybrid, and Omnimodal Models › Multiple Temporal and Spatial Scales

Temporal hierarchy reduces the frequency of high-level goal or option decisions while retaining fast low-level control. If a subgoal latent updates every $K$ fast steps, a five-minute plan at one hundred control updates per second can contain three hundred one-second macro-decisions instead of thirty thousand primitive decisions. This shortens the high-level search chain. It can reduce compounding prediction error when coarse-transition error remains small enough and fine-level error is controlled, for example by local rollouts or regular correction from observations; hierarchy does not guarantee an empirical advantage.

A canonical reinforcement-learning formalization of temporal abstraction is the options framework of Sutton, Precup, and Singh (1999). The idea is to let the agent choose not just individual actions but whole temporally extended behaviors, and to reason about those behaviors as if they were single actions. Let $\mathcal{S}$ and $\mathcal{A}$ denote the underlying MDP's state and action spaces. An option is a triple $(\mathcal{I}, \pi, \beta)$ where:

• $\mathcal{I} \subseteq \mathcal{S}$ is the set of states in which the option may be started.
• $\pi(a \mid s)$ is the option's internal action distribution: $\pi(a \mid s) \in [0,1]$ and $\sum_{a \in \mathcal{A}} \pi(a \mid s)=1$ for each $s$.
• $\beta: \mathcal{S} \to [0,1]$ is the termination probability.

Read the triple as a small policy with a guard rail. The initiation set $\mathcal{I}$ says where the option makes sense, the internal policy $\pi$ says what to do while it runs, and the termination probability $\beta$ says when to stop. When the option is chosen at time $t$, it executes its internal policy until $\beta$ fires. Assuming the option terminates almost surely, its duration is a positive integer-valued random variable $\tau \in \{1, 2, \dots\}$; depending on the option and environment, $\tau$ may be stochastic or degenerate at a fixed duration. This turns the Markov decision process (MDP) into a semi-Markov decision process (SMDP), where the value of taking option $o$ in state $s$ is

where:

• $Q(s,o)$: the expected return from taking option $o$ in state $s$.
• $s$: the current state.
• $s_t$: the state at time step $t$.
• $o$: the option.
• $o_t$: the option chosen at time step $t$.
• $\tau$: the random duration of the option.
• $t$: the current time step.
• $\gamma$: discount factor.
• $r_i$: reward at time step $i$.
• $i$: time index running from $t+1$ to $t+\tau$.
• $V$: continuation value under the policy followed after the option terminates.
• $s_{t+\tau}$: the state after option $o$ terminates.

Here the expectation is over the option's internal action choices, the environment's transitions and rewards, and the random termination time.

Compare this to an ordinary one-step action value, which contains one immediate reward plus one continuation value discounted by $\gamma$. The option-value function instead sums the rewards earned during the whole option and then discounts the continuation value by $\gamma^{\tau}$, the discount over the option's realized duration. Different options therefore carry different effective continuation discounts, and the high-level policy chooses temporally extended actions rather than atomic ones. For $0<\gamma<1$, continuation value after a longer realized duration receives a smaller weight.

The effective discount for an option is $\gamma^{\tau}$, so duration directly changes the weights the high-level policy compares.

Getting the discounting right is essential because an incorrect duration discount can distort option values whenever it changes the weight on a nonzero continuation or reward term. For example, replacing $\gamma^\tau$ with $\gamma$ overweights positive continuation value after long options; the direction of the resulting preference also depends on within-option rewards and the sign of the continuation value.

Several hierarchical world-model architectures implement temporal abstraction with multiscale latent states, latent skills, or latent goals rather than literal call-and-return option triples. One two-level generative design stacks latent variables that update at different rates:

• A slow latent $z^{s}_k$ that changes once every $K$ fast steps, indexed by macro-time $k$.
• A fast latent $z^{f}_t$ that updates every step and receives $z^{s}_{\lfloor t/K \rfloor}$ as conditioning context.

When training makes it encode persistent behavior-level context, the slow latent can play a role analogous to option identity or high-level intent. The fast latent updates each step and can represent variation that the persistent context does not explain. Conditioning the fast transition on the current slow latent lets its predictions specialize to that context.

The joint generative factorization is

where:

• $z^{f}_{1:T}$: sequence of fast latent states over fast time steps $t = 1, \dots, T$.
• $z^{s}_{1:L}$: sequence of slow latent states over slow time steps $k = 1, \dots, L$.
• $z^{f}_0$: initial fast latent state before the sequence.
• $z^{s}_0$: initial slow latent state.
• $L$: number of slow steps (macro-steps).
• $K$: number of fast steps per slow step, so $T = L \times K$.
• $a_t$: action at fast time step $t$; the joint distribution is conditioned on the action sequence $a_{0:T-1}$.
• $p$: probability distribution.
• $z^{s}_k$: the slow latent state at macro-time step $k$.
• $z^{f}_t$: the fast latent state at fast time step $t$.
• $\lfloor \cdot \rfloor$: the floor function, giving the macro-time index for fast time $t$.

The outer product over $k$ runs over macro-steps, and the inner product over $t$ runs over the $K$ fast steps inside each macro-step. Reading the factorization from left to right, the slow equation defines the temporal coarsening; the fast equation defines the fine-grained dynamics conditioned on the current slow context. Notice how the slow transition $p(z^{s}_{k+1} \mid z^{s}_k)$ does not depend on the actions at all in this minimal form: it is a coarse drift model of how the sub-goal evolves, while the fast level absorbs the effect of actions. In richer variants the slow transition is also conditioned on an abstract action or goal, but the essential split is that the slow level summarizes and the fast level reacts.

Encoders can mirror this structure. A training-time smoothing encoder $q(z^{s}_k \mid o_{kK : (k+1)K - 1})$ aggregates an entire observation window, while a fast encoder $q(z^{f}_t \mid o_t, z^{s}_{\lfloor t/K \rfloor})$ captures per-step detail. Here $o_t$ is the observation at fast time step $t$, and $o_{kK:(k+1)K-1}$ is the corresponding window for slow step $k$. Because this smoothing form uses the completed window, it is not a causal online encoder for early steps in that same block. An online system can instead infer $z^s_k$ from the preceding completed window $o_{(k-1)K:kK-1}$ before block $k$, or delay use of the summary until its window is complete. In either case, the asymmetry is the point: the slow encoder compresses a batch of observations, while the fast encoder sees a single observation together with the summary currently in force. A useful analogy is human scene perception: an initial glance can provide scene gist that helps guide subsequent fixations toward task-relevant detail.

FeUdal Networks (Vezhnevets et al., 2017) use a lower-temporal-resolution Manager whose dilated recurrent state and multi-step objective produce persistent goal conditioning, while a Worker chooses primitive actions each step. The Manager and Worker are learned concurrently with separate objectives, without propagating gradients between the two modules. Director adds hierarchical policies over the latent state space of a learned pixel-based world model: a Manager selects latent goals every eight steps and a Worker chooses primitive actions to reach them. These are policy hierarchies rather than evidence that Director contains slow and fast world-model state transitions. More generally, persistent context can spare a fast predictor from rediscovering behavior-level information at every step, but smaller residuals are an empirical outcome rather than a guarantee.

Temporal scales have a spatial counterpart. A building-scale robot may use room-level topology for navigation together with millimeter-scale contact geometry and high-rate force or torque signals for manipulation. With a fixed-budget dense representation, one resolution is inefficient across these scales: a fine global grid allocates detail where it is irrelevant, while a coarse grid misses local contact structure. A dense volumetric representation of an entire building at one-millimeter resolution is computationally impractical, so sparse or hierarchical representations allocate fine resolution selectively. Two common strategies are:

• Coarse-to-fine latent pyramids. The state has a low-resolution "global" component, for example a volumetric or object-level map, plus fine local patches that are only queried where the agent is currently interacting. The global component ignores details that do not matter; the local component ignores the rest of the world. The two are linked by the agent's current focus, so the fine patches are always embedded in a coarse frame of reference.
• Multi-resolution observation models. The decoder reconstructs a coarse observation everywhere and a fine observation only inside an attention window around the agent. Biological foveation offers an analogy: cone density and visual acuity peak in a small central region, while gaze shifts sample the rest of the scene. Most retinal photoreceptors, predominantly rods, are outside the fovea.

Structure, not resolution alone, should drive the transition model. Regular-grid maps often suit convolution because of spatial locality and translation equivariance; topological maps often suit graph networks because they encode adjacency and relations. Continuous-time models can help with irregularly timed observations or queries, while contact dynamics may additionally require event-aware modeling and accurate numerical integration. Graph networks can themselves be effective for contact dynamics. The practical rule is to match an inductive bias to the representation's actual topology, smoothness, events, and sampling process.

Hierarchical models add complexity, and that complexity has costs. A slow policy can propose subgoals that a fast policy cannot realize. Termination can be poorly calibrated, causing premature or excessively long execution. When observations support several contexts, slow-state inference can become uncertain or lock onto the wrong one unless history, actions, or later observations correct it. A window encoder can also exploit spurious phase or boundary cues. Window length itself becomes a shortcut when variable lengths or padding patterns correlate with the target. Such correlations fail when task periods or window construction change.

Several of these failure modes can manifest as disagreement between the levels, with no training signal forcing them to reconcile. Silent inconsistency is especially difficult because it may not raise an obvious error. The slow level believes the robot is near the mug, the fast level believes it is near the sink, and the model produces plausible but wrong predictions for both. This drift can remain hidden from aggregate one-step validation loss, so cross-level agreement should be evaluated explicitly or included as an equivalent targeted validation term.

Cross-level consistency losses address the disagreement subset, not every failure above. If the slow level predicts a projected coarse summary of the next window and the fast level predicts its fine state sequence, applying the same projection to the fast prediction gives a shared quantity on which the two levels can be compared. Training on that agreement supplies a direct gradient to the slow level. Subgoal feasibility instead calls for reachability or capability checks; termination needs dedicated closed-loop stopping metrics; and window shortcuts need distribution-shift tests and ablations. Joint end-to-end training and a bounded window matched to the task's timescale may support these safeguards, but do not replace them.