Robotic Manipulation › Task Planning with Learned Affordances

The preceding sections mainly examined predictions of action effects, although grasp selection can also depend on later task outcomes. A manipulation task may also need to choose which object to interact with, where to reach, and in what order. Affordance estimates can help propose actions; when a world model is available, it can predict their consequences.

Learned affordance models can condition on a scene, action, and sometimes a goal. In a rightward-push task, a contact point on one side of an object might be useful, but the outcome depends on geometry, friction, and the action direction. The toy map below instead shows a predictor's one-step displacement for a fixed rightward command; it is not a goal-conditioned success map.

One possible affordance architecture has three components:

• Feature extractor. Encodes an image or structured scene state.
• Optional goal conditioning. Specifies a target pose, instruction, or image when the output is task-specific.
• Prediction head. Estimates an explicitly defined quantity, such as success probability, action value, or displacement.

Interaction logs or simulated outcomes can supervise such a head. For a probabilistic goal-conditioned formulation, $m(x\mid o,g,a)$ might estimate whether action $a$ at location $x$ helps achieve goal $g$ from observation $o$. Other heads predict an effect vector instead. Language-conditioned variants connect to Part IX, but their calibration and transfer require evaluation.

One possible architecture is a two-level loop:

1. High-level selector. Uses estimated action effects or feasibility to propose a sub-task or contact region.
2. Low-level controller. Executes a candidate action and uses feedback; it may use model-predictive control, but need not.

This decomposition can help diagnose errors, but a failed push may involve perception, effect prediction, planning, or control together. Component tests and interventions, rather than the architecture alone, help assign causes (Part XI).

Task-and-motion planning can also search over symbolic predicates and call geometric or learned feasibility checks (Part VII). A PDDL-style planner is one option; the affordance predictor above does not require one.

An affordance predictor can be learned directly from outcomes without an explicit world model. Where both are present, comparing predicted effects can expose disagreement: a proposed rightward push is suspect if the dynamics model instead predicts a tip. Joint training may encourage agreement but does not guarantee physical or task-level consistency. Testing the proposed action in the real system remains decisive.