Hierarchical, Hybrid, and Omnimodal Models › Neural-Symbolic and Simulator Hybrids

Pure learned models offer flexible function approximation, but their data requirements depend on architecture, priors, pretraining, and task complexity. With sufficient capacity, suitable data, and successful optimization, they can approximate a broad range of patterns, though they do not automatically enforce logical consistency. Symbolic systems can enforce their encoded rules exactly, while hand-crafted simulators can be accurate within their assumptions and numerical tolerances. Both become brittle when those assumptions fail. Hybrid models combine these strengths: learned components can handle perception, uncertainty, and difficult-to-model residuals, while symbolic or simulation components can implement explicit planning rules, logical constraints, and well-understood physics. The central design choice is where to draw the boundary and how information should cross it.

There is no single way to combine neural and symbolic components. In practice the integrations fall into three broad patterns, distinguished by how much information flows between the learned and non-learned parts and in which direction.

Pipeline hybrids. One common pipeline hybrid uses learned perception to map raw observations into a symbolic state of typed objects, predicates, and relations. In the neural-symbolic pipeline described here, a symbolic reasoner plans over that state and a learned controller executes the plan; other pipeline hybrids may use classical or hand-designed controllers. Shakey was an early precursor to this perception-model-planning-execution pattern, and some modern task-and-motion planning systems use learned perception or predicates as a front end. Interfaces between modules are discrete and non-differentiable: a planning cycle may run from perception to planner to controller, while execution feedback triggers new perception and replanning. The symbolic layer makes assumptions inspectable. The cost is that downstream plan failure supplies no end-to-end gradient to correct a perception mistake.

Differentiable hybrids. The symbolic component is made differentiable (or a soft relaxation is used) so gradients can flow through it. Differentiable planners, differentiable logic layers, and differentiable physics simulators all live here. The advantage is end-to-end learning: because the gradient reaches the symbolic part, the learned perception can be shaped by the downstream planning objective, closing the loop that the pipeline pattern leaves open. The disadvantage is that soft relaxations can be approximate and the optimizer may drift into regimes where the relaxation is unfaithful. A differentiable planner that relaxes discrete decisions into continuous weights might, during training, settle into a continuous solution that has no valid discrete counterpart, and the resulting policy can be brittle at deployment.

Residual hybrids. A hand-crafted model supplies a nominal prediction, and a neural network predicts an additive correction; when the nominal model is sufficiently accurate, it may account for most of the final prediction. During training, the residual is fit to the model's mismatch. This pattern sidesteps the optimization hazards of the differentiable hybrid by treating the non-learned part as fixed: the simulator is evaluated in the forward direction and the network learns its prediction error. Multi-step use is open loop only when predicted states are recursively fed back without observation correction. Let $f_{\text{sim}}$ be the simulator and $g_\theta$ the neural correction. The hybrid dynamics are

$$\hat{s}_{t+1} = f_{\text{sim}}(s_t, a_t) + g_\theta(s_t, a_t),$$

and the training loss is

$$\mathcal{L}(\theta) = \mathbb{E}_{(s_t, a_t, s_{t+1})} \big\| s_{t+1} - f_{\text{sim}}(s_t, a_t) - g_\theta(s_t, a_t) \big\|_2^2.$$

where:

• $\hat{s}_{t+1}$: the predicted next state.
• $f_{\text{sim}}(s_t, a_t)$: the simulator's predicted next state.
• $g_\theta(s_t, a_t)$: the neural residual correction.
• $\theta$: parameters of the residual correction network.
• $t$: time step index.
• $s_t$: the current state.
• $a_t$: the action at time $t$.
• $s_{t+1}$: the true next state.
• $\mathcal{L}(\theta)$: the mean squared residual loss used to train the correction network.
• $\mathbb{E}_{(s_t, a_t, s_{t+1})}$: expectation over the training distribution of transitions.
• $\|\cdot\|_2^2$: squared Euclidean norm.

Read the loss as asking the network to explain only the part of the transition that the simulator misses. When the simulator's expected error is small relative to the scale of the chosen full-model target under the same normalization and distribution, the residual learner receives a smaller-magnitude target. Residual learning is generally more favorable when the remaining mismatch is predictable, observable, and representable by the learner.

The practical benefit is that the residual network can see a smaller target. The next cell isolates that scale effect by simulating residual targets for an accurate simulator and a biased simulator while holding the true transition distribution fixed. It does not test whether either residual is predictable from the model inputs. The sample is drawn with seed 20240517.

Residual hybrids are used in robotics when a calibrated simulator captures much of the target dynamics and a learner can model the remaining friction, contact, or deformation effects. Under those conditions, $g_\theta$ receives a smaller target than a model trained to predict the full transition from scratch. The gain depends on the base simulator and on whether its mismatch is observable and representable.

A differentiable simulator exposes gradients of simulated states with respect to control inputs and physical parameters. Instead of training a separate learned model, the simulator can serve as the world model for well-understood parts of the scene. Examples include DiffTaichi, Dojo, legacy Brax physics pipelines, and MuJoCo's current MJX-JAX implementation; MJX-Warp does not currently support automatic differentiation. These gradients support parameter fitting and, when reliable, end-to-end policy or planner optimization through simulation.

The cost is that simulator gradients can be biased or numerically delicate near contacts and other discontinuities. An infinitesimal input change may alter the contact sequence, producing a gradient that is uninformative for the intended update. When contact gradients are unreliable, a residual or other forward-only hybrid that treats the simulator as fixed avoids backpropagating through those gradients and may be preferable; its reliability still has to be established for the particular simulator and task.

The relational planning representation used here describes entities through predicates such as on(mug, table), inside(bean, grinder), and reachable(gripper, handle); a domain may also assign entity types. Neuro-symbolic systems can learn such predicates from data. A perception encoder embeds detected objects, and a relation head produces soft truth values that a planner can query. A differentiable training path can use those soft values, while a discrete planner may threshold them at inference time.

The key modeling benefit is systematic generalization. Because lifted rules are stated over variables, a symbolic planner can apply them to unseen object combinations when the symbolic state is accurate, all predicates, types, and constraints hold, its operators cover the case, and search succeeds. Many standard neural architectures perform poorly on controlled compositional splits unless their architecture, objective, or training distribution supports systematic recombination. As covered in Part IV, Ch 6, explicit relational structure is one way to encode that support.

A practical hybrid pattern is simulator-in-the-loop training. When real interaction is expensive, simulation can supply much of the training data while limited real data supports calibration, residual learning, or validation. A learned residual can reduce systematic simulator error; domain randomization instead trains the policy to tolerate a family of simulator variations. The physics engine supplies encoded structure, and the learned component addresses selected gaps. A corrected model may reduce exploitable bias, but policy optimization can still seek its remaining errors. Uncertainty penalties, robust optimization, constrained rollouts, and real-system validation are separate safeguards against that failure.