Hierarchical, Symbolic, Language, and Multi-Agent Planning › Language-conditioned goals and plans

If symbols ground plans, language specifies them. "Put the red mug on the top shelf" is a natural-language instruction that a symbolic planner can translate directly into a goal state: on(red_mug, top_shelf). The instruction does not say how to do it (that is the planner's job), but it does say what "done" means. Language-conditioned planning turns natural-language instructions into executable plans by grounding the instruction into predicates, rewards, or constraints that the planner can optimize.

Language is a useful planning interface because it can name objects, relations, preferences, and orderings. The mapping is not syntactic substitution: reference resolution, context, missing concepts, and ambiguity all matter. A phrase such as “the red mug” must first identify an entity represented by the system before a goal such as on(red_mug, top_shelf) is meaningful.

There are three regimes worth distinguishing.

Instruction as goal. The simplest regime: the instruction names a target state, and the planner finds a path there. "Go to the kitchen" translates to agent_at(kitchen). The instruction acts as a goal specification channel: the target predicate becomes the goal, and the planner uses the same world model and search machinery it would use for any other goal-conditioned problem. The plan it produces is only as good as the grounding, so a wrong predicate yields a wrong plan even when the planner is correct. Because the world model is the same one you would use for any other goal, the machinery is nearly identical to what we built in Search and Belief-Space Planning.

Instruction as procedure. The instruction gives intermediate steps, ordering, or constraints. “First heat the pan, then add oil, then add the eggs” is a totally ordered procedure; a partially ordered example could allow whisking the eggs while the pan heats, then require both to finish before cooking. The planner fills in missing details and checks feasibility. A supplied order can reduce search, but it can also exclude a necessary alternative.

Instruction as reward. A third regime: the instruction defines a preference, not a hard constraint. "Tidy the room" does not specify a particular arrangement; it describes a family of acceptable outcomes. In this regime the instruction can be compiled into a reward or cost function, and planning proceeds as ordinary reward maximization. This is the regime most relevant to language-conditioned RL and to the reward-from-language literature. A common instantiation treats the instruction as defining a reward $r(s, a; u)$ that depends on the instruction $u$, so that the induced objective is the expected return

$$J(\pi; u) = \mathbb{E}_{\pi}\left[ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t; u) \right]$$

where:

• $\pi$: the policy being optimized, mapping states to distributions over actions
• $u$: the natural-language instruction that conditions the reward
• $r(s_t, a_t; u)$: the instruction-conditioned reward at timestep $t$
• $\gamma \in [0, 1)$: the discount factor
• $J(\pi; u)$: the expected discounted return of policy $\pi$ under instruction $u$

Maximizing this objective yields an instruction-conditioned policy. Rolling that policy out from a particular initial state produces a trajectory; computing a contingent or open-loop plan requires the corresponding planning procedure.

These interfaces trade specificity and flexibility in different ways; none is uniformly more expressive without fixing the goal, procedure, and reward languages. A procedure can constrain action order more than a goal, while a reward can encode graded preferences yet leave behavior under-specified. The right interface depends on which parts of the task the instruction states reliably and which parts the planner should infer.

For the goal-predicate regime, let $\mathcal{P}$ be the vocabulary of atomic grounded predicates. A grounding function maps an instruction $u$ and state or observation context $s$ to a desired goal set $G \subseteq \mathcal{P}$:

$$g_{\mathrm{goal}}:\mathcal{U}\times\mathcal{S}\rightarrow 2^{\mathcal{P}}, \qquad g_{\mathrm{goal}}(u,s)=G.$$

where:

• $\mathcal{U}$: the space of instructions (natural-language strings or token sequences)
• $\mathcal{S}$: the state space of the world model (continuous or latent)
• $\mathcal{P}$: the atomic grounded-predicate vocabulary
• $u$: the specific instruction being grounded
• $s$: the specific state the instruction is being grounded against

A common supervised approach fits a probabilistic grounder to paired instructions, contexts, and goal sets. With data distribution $p_{\text{data}}(u,s,G)$, maximum conditional likelihood gives

$$\hat{\theta}=\arg\max_{\theta} \mathbb{E}_{(u,s,G)\sim p_{\text{data}}} \left[\log p_\theta(G\mid u,s)\right].$$

Here $G$ is the whole target set, so $-\log p_\theta(G\mid u,s)$ is a per-example negative log likelihood. A multilabel Bernoulli model is one implementation; contrastive and pretrained-model approaches are also possible. Reward, constraint, and procedure grounding use different output types rather than pretending they are predicate sets. Even with a useful world-state representation, grounding may require relational reasoning, reference resolution, and uncertainty estimation.

A useful first stage is referent selection: given an instruction and an observation, identify candidate objects and relations. Object-centric representations can supply candidate entities, but slots may not align with names, attributes, events, affordances, or sets. Grounding remains a modeling problem rather than a lookup.

If a fixed world representation and action interface do not encode a referred concept, language-only scaling cannot reliably ground or execute it. Joint representation learning, new perception, tools, or external memory can change that interface, so the limitation belongs to the fixed system rather than to language in general.

One family of systems uses a language model to propose high-level skills, then scores proposals with affordance or value models, simulators, or environmental feedback. SayCan combines language scores with learned skill affordances; Inner Monologue conditions replanning on feedback. These scoring components are fallible models, not correctness certificates. The analogy to neural MCTS is limited: a policy prior guides exploration and a value model estimates leaves; neither automatically verifies feasibility.

Proposal and checking separate candidate coverage from candidate evaluation. Reliability depends on both: the proposer must include a workable candidate, and the scorer must be calibrated on that candidate's state-action distribution. Model error, missing constraints, and distribution shift can still admit an infeasible plan.

In the fixed-library architecture developed here, language specifies objectives and helps compose existing skills. It cannot create a missing actuator or low-level controller. It can still improve task decomposition, parameterization, tool selection, and plan quality within those physical limits.

If a robot has no controller or tool for opening a drawer, a clearer instruction does not supply the missing physical capability. Better language reasoning can nevertheless choose and compose available skills more effectively, so grounding quality and skill coverage should be evaluated separately.