Self-Models, Body Schemas, and Affordances › Capabilities, Constraints, and Affordances

Knowing your body configuration is not yet knowing what you can do. The next layer of the self-model summarizes possibility: which actions are feasible, which outcomes are reachable, and which interactions the environment currently supports. These summaries are where self-models first pay off in the decision loop, because they let a planner prune huge portions of the action space before any expensive rollout.

The need for this layer comes from a scaling problem that raw self-models cannot solve. A body schema plus a world model is, in principle, complete: for any candidate action sequence, the agent could roll the pair forward and find out what happens. But that completeness is combinatorially expensive, and most of the expense is wasted on actions that were never viable. A planner that rolls out a grasp attempt in a region the arm cannot physically reach has not learned anything useful; it has paid full model-evaluation cost to rediscover a geometric fact it could have known in advance. Capability summaries are the cache that prevents this waste. They answer the coarse question "is this even possible?" cheaply, so that the expensive question "what exactly happens if I try?" is reserved for candidates that survive the first filter.

Let $\mathcal{C} \subseteq \mathbb{R}^n$ denote the set of body configurations the agent can occupy or command, where $n$ is the number of independent configuration variables (joint angles, base pose, gripper aperture). For a robot arm, $\mathcal{C}$ is carved out by several families of constraints:

• Joint limits: each joint angle lies in an interval.
• Actuation limits: torques and velocities stay within motor capabilities, which couple across joints.
• Self-collision: the body must not intersect itself, a nonconvex geometric constraint.
• Stability: legged and wheeled systems must keep their center of mass and contact forces within support regions.
• Task constraints: keep the cup upright, keep the end effector out of the sterile field.

These constraint families have different mathematical forms. Joint limits are box constraints on individual variables. Self-collision is a nonlinear, nonconvex geometric predicate involving many variables at once. Stability constraints include dynamics and change with the contact state. Task constraints are imposed from outside the agent and may change with each job. A feasible set assembled from such pieces is often nonconvex and can be disconnected when obstacles, contacts, or task constraints separate regions of configuration space. Exact characterization can be expensive, so capability maps commonly approximate the forward-kinematic image of the feasible set by analysis, optimization, or sampling. Their reliability depends on the guarantee provided by the chosen method.

The part of the world that the agent's body can realize is then $f(\mathcal{C})$, the forward-kinematic image of the feasible set. For a two-link arm this image is an annulus: the end effector can reach anything between radius $|L_1 - L_2|$ and radius $L_1 + L_2$ from the shoulder, where $L_1$ and $L_2$ are the two link lengths, and nothing inside that inner hole. This geometric fact, derived entirely from the self-model with no reference to any object, is the prototype of a capability. The inner hole deserves a moment of reflection, because it is the most counterintuitive part of the result and the most decision-relevant. A planner without the capability map, shown a target near the shoulder, would try to plan a reach there, fail, retry with different parameters, and fail again; each failure costs a full planning episode. The map converts all of those episodes into a single lookup. And note that the knowledge is purely about the agent: the annulus exists whether or not there is a mug anywhere in the scene.

The "mediated by state" clause is doing real work in that definition. Capabilities sound like fixed properties of an agent, the way a spec sheet lists a robot's payload and reach, but the useful notion is state-dependent: a folded arm cannot reach what an extended one can, a grasping gripper cannot pinch, a robot with a depleted battery has a smaller feasible torque set. Capabilities are therefore best understood as functions of the self-state, not constants, and the capability summaries that matter in practice are indexed by the same $z^{\text{self}}_t$ that the body schema models.

The ecological psychologist James J. Gibson introduced and developed the affordance concept in his 1966 work, then gave it its canonical mature treatment in The Ecological Approach to Visual Perception (1979). Standard physics vocabulary can describe a flat rigid surface at knee height, but it does not directly say whether a particular body can sit on it. Affordances express that relation between environment and agent. A bench's height in centimeters is an object property; "sit-on-able" is an action-relevant summary, and it depends on the sitter.

::: {.callout-note title="Affordance"} An affordance is a relation between an agent's capabilities and an environmental state that constrains which interactions are possible and what outcomes they may produce. In this chapter, we operationalize it as a skill-, outcome-, and evaluation-horizon-conditioned prediction. For an observation $o_t$, a skill $k$, an outcome class $e$, and a horizon $H$,

$$\Phi_{k,H}(e \mid o_t, z^{\text{self}}_t) = \Pr(\text{outcome } e \text{ succeeds within } H \mid o_t,\ z^{\text{self}}_t,\ \text{execute skill } k),$$

where:

• $\Phi_{k,H}$: the affordance function for skill $k$ and evaluation horizon $H$, returning the probability that executing $k$ from this situation yields outcome class $e$ within that horizon
• $e$: the outcome class whose success probability is being predicted
• $k$: the skill being executed
• $H$: the evaluation horizon within which success must occur
• $o_t$: the observation (or state) at time $t$
• $z^{\text{self}}_t$: the agent's self-state at time $t$ (its body configuration together with its actuator states)

An affordance holds when this probability exceeds a decision threshold.

The threshold clause is the hinge between graded prediction and discrete commitment. For a fixed skill and scene, the success probability varies with the self-state (here, reach extension); the planner's go/no-go boundary is where the curve crosses the threshold.

:::

Several features of this definition deserve attention.

• Affordances are conditioned on the self. The same visual scene affords different actions to a robot with a parallel-jaw gripper versus a suction gripper. This is exactly why affordances live in this chapter rather than in the environment-modeling chapters: the function $\Phi_{k,H}$ takes the self-state as an argument. Strip the $z^{\text{self}}_t$ argument out and the function silently changes meaning, from "what can this agent do here?" to "what can some agent do here?", and the resulting quantity is much less useful for the agent's own planning.
• Affordances are skill-relative. "Graspable" is not a property of the mug; it is a property of the mug under a particular grasp policy. If the policy improves, the affordance map changes. Sahin and colleagues formalized this agent-relativity in their 2007 treatment of affordances in robotics, and it remains the standard framing. This skill-relativity has a practical implication that is easy to miss: affordance estimates are only as good as the policies they were learned under, and reusing an affordance map after upgrading the skill is a form of stale-model error, exactly analogous to using a kinematic schema after the robot has been repaired.
• Affordances are decision-useful summaries. Here we connect to a theme this book returns to repeatedly: the distinction between open-loop prediction quality and closed-loop usefulness. A full world model can answer "what happens if I execute the grasp skill here?" by rolling out predictions. An affordance map answers it in one lookup, at the cost of discarding information irrelevant to that action-outcome pair. Affordances are, in the language of Sufficient State and State Abstraction, an abstraction of the world model tuned to a decision problem. They trade generality for cheaper evaluation; any gain in robustness has to be demonstrated under the relevant shift.

That last point deserves a fuller statement, because it is the chapter's clearest example of the four-way split among notions of model quality introduced at the outset. An affordance map answers fewer questions than the world model it summarizes, and it answers them coarsely through the lossy bottleneck of a success probability. What it can buy in exchange is speed and task alignment: the map is cheaper to evaluate and can be learned directly from success labels. A task-aligned summary may transfer better when it discards nuisance variation while retaining the causal factors that determine success, but low dimensionality does not make stability under distribution shift automatic. Systems can therefore maintain both representations: a fuller model for novel situations and detailed planning, and an affordance layer for fast filtering when its calibration has been validated.

The reachability map of a robot arm is the cleanest example. Rather than asking, for every candidate goal, "can some sequence of joint motions bring the end effector here?", the agent precomputes or learns a map $\mathcal{R}: \mathcal{W} \to \{0,1\}$ over the workspace $\mathcal{W}$. A planner may safely prune a goal only when the map supplies a conservative infeasibility certificate; an unsampled cell is not such a certificate. The same pattern scales up. Modern robotics includes visual affordance models for action categories such as grasp, lift, open, support, push, and pull. Some predict semantic action regions; others estimate robot-specific success from interaction data. In each case, the representation compresses a class of counterfactual rollouts into a per-location or per-action score whose meaning depends on the skill, embodiment, and evaluation horizon.

There are two common routes, and they are often combined:

• Self-supervised exploration. The agent executes its skills across states and records successes and failures, then fits $\Phi_{k,H}$ from its own interaction log. This inherits the strengths of self-model learning: control over the sampled situations and continual recalibration. Its weakness is coverage: affordances for rare configurations are poorly estimated, and exploration in the real world is expensive.
• Transfer from demonstration or priors. Affordance structure observed in human data, other robots, or simulation seeds $\Phi$ before the agent has explored. This is fast but imports someone else's embodiment, so the imported affordances must be re-grounded in the agent's actual capabilities, which is precisely what the self-model provides.

At foundation scale, observation-based pretraining can supply useful structure before a robot acts, but the imported predictions still need robot-specific grounding. Human video reflects human hands, reach, scale, and force. Applied directly to a different embodiment, its action predictions can be miscalibrated in systematic ways. Robot interaction data and the self-model supply the correction through the robot's reach envelope, gripper geometry, controller competence, and force limits. This is a useful division of labor, not a guaranteed mathematical factorization: prior data propose environment-conditioned action structure, while robot-specific trials determine whether that structure predicts success for this skill and body.

We will meet the exploration side again in the planning chapters of Part VII: Planning and Agency, where information gathering about affordances competes with task progress. Every probe is a step not spent directly on the task, while a task attempt becomes an affordance sample only if the agent records the outcome. Too much probing delays commitment; too little produces overconfident extrapolation from sparse evidence. The uncertainty-aware treatment of affordance estimates in Part IV: Space, Physics, and Causality begins to formalize that tradeoff.