Hierarchical, Symbolic, Language, and Multi-Agent Planning › Evaluating a structured planning stack

An end-to-end success rate cannot tell you which abstraction helped or which interface failed. A useful evaluation therefore treats every boundary in the stack as a contract. The contract states what enters a component, what it returns, which uncertainty accompanies the return, and which downstream decision uses it.

The distinction from earlier chapters remains essential. A model can predict an option's terminal-state distribution accurately while a planner chooses the wrong option because its reward or constraint is misspecified. Conversely, a biased predictor can still support the correct action on a narrow decision boundary. Report both kinds of measurement:

• Predictive measurements test option duration, cumulative reward, predicate effects, language-grounding probabilities, and opponent-action probabilities against held-out outcomes.
• Decision measurements test task return, constraint violations, replanning frequency, coordination success, and regret against a stated baseline.
• Interface measurements test whether predicted symbolic effects agree with the predicates observed after option execution and whether language-derived goals match annotated intent.

The measurements should share evaluation episodes, but they should not be collapsed into one score. A high task success rate can hide a poor grounder when the planner reaches the goal through a dataset shortcut. A low success rate can blame the grounder unfairly when the required option does not exist.

The toy program uses deterministic states and effects so that the binding logic is visible. A learned system should expose distributions or confidence sets instead:

• An option model can return $P(R_\omega,s',\tau\mid s,\omega)$ rather than one terminal state.
• A predicate detector can return calibrated probabilities or abstain near its decision boundary rather than assert a brittle Boolean fact.
• A language grounder can return several goal hypotheses with posterior weights rather than silently choose one interpretation.
• An opponent model can maintain a distribution over types or policies and update it after observed actions and messages.

Passing those uncertainties forward does not require every layer to share one representation. It requires the consumer to know what the producer's number means. A calibrated predicate probability is not the same object as an option-success probability, and neither is automatically a probability that the whole plan will succeed. The events are dependent: a grounding error changes which options are attempted, while an option failure changes which predicates are observed next.

For a fixed candidate plan $o_{1:m}$, a planner may estimate a joint success probability by propagating its belief through each option model,

$$b_k(s_k) = \sum_{s_{k-1}} P(s_k,\text{success}_k \mid s_{k-1},o_k)b_{k-1}(s_{k-1}),$$

with the mass of $b_k$ equal to the probability that the prefix has succeeded under the modeled dynamics. Here $b_{k-1}$ is a subprobability mass function over state and successful-prefix execution; its total mass is the modeled probability that the prefix has succeeded. Failure probability remains visible as missing mass rather than being hidden by renormalization. Multiplying independent scalar success rates would be valid only under a conditional-independence model that is rarely justified. Belief propagation preserves how earlier outcomes alter later preconditions.

A focused test suite should deliberately perturb one interface at a time:

The point is not to make all perturbations equally realistic. It is to give each component a falsifiable contract. In the gridworld, execute_symbolic_plan performs two such checks: it rejects a violated precondition before dispatch and compares the operator's predicted symbolic effects with the abstraction of the physical result afterward. A stochastic implementation would replace equality with a calibrated likelihood or acceptance region.

Hierarchical planning should not be compared only with exhaustive primitive search. Useful baselines include a receding-horizon primitive planner, a reactive or learned policy, and the same high-level planner with different option libraries. Symbolic planning should be compared both with a continuous planner and with an ablation that removes online feasibility checks. Language grounding should be compared with an oracle goal specification so that grounding error is separated from planning error. Multi-agent methods should face fixed policies, held-out populations, and adapting co-players rather than a single training opponent.

Control the quantity that supports each conclusion. If a plot claims that temporal abstraction reduces node expansions, hold the search algorithm, heuristic, task, and solution-quality requirement fixed while changing the action representation. If it claims faster wall-clock planning, include the cost of building option models and checking continuous feasibility. If it claims transfer, re-ground the same operator schema in changed environments and report both successful reuse and rejected infeasible instances.

This evaluation discipline also prevents a common category error. The symbolic plan in the worked example is geometry-independent as a data structure, but the system is not geometry-independent: the bound navigation options and effect checks still depend on the map. The abstraction is valuable because it isolates that dependence behind a testable interface, not because it makes the dependence disappear.