Self-Models, Body Schemas, and Affordances › Body Schemas and Kinematic State

Why can a cat pounce onto a narrow ledge without measuring anything, and why can you touch your nose with your eyes closed? Both rely on an internal model of the body's geometry and its current configuration. The cat does not compute the ledge height in world coordinates and compare it against a table of jump distances; it consults an integrated sense of where its body is and what it can do from here. You do not locate your nose visually; you consult a body-centered map that has been maintained continuously since long before you could read. We use body schema for the functional, and typically learned, model of the agent's own body and its relation to actions.

It is worth being precise about what "model" means here, because the body schema is not one function but a cluster of them, tied together by a shared subject. There is the command-to-configuration map (what does this motor command do to my joints?), the configuration-to-sensation map (what will I feel if my joints move like this?), and the configuration-to-geometry map (where is my hand relative to my shoulder?). Classical robotics often specifies these with separate analytic and calibrated models; a learned body schema may fit them jointly. What makes them a schema rather than a grab-bag of regressions is that they concern the same self-state and can be checked for cross-model consistency.

Recall from our discussion of sufficient state in Part II: Inference, Decisions, and Control that a Markov state must contain everything relevant to future prediction. For an embodied agent, the body configuration $q_t$ (joint angles, base pose, gripper aperture) is unambiguously part of that state. Omitting it has immediate consequences: two observationally identical camera views can correspond to completely different futures depending on arm posture, because one posture can reach the target and the other cannot. If your latent state does not encode posture, your model cannot predict which actions will work, and no amount of visual fidelity fixes it.

This can be a concrete instance of the partial-observability diagnosis from Markov and Partially Observable Decision Processes. If posture is hidden and the available camera history does not disambiguate action-relevant futures, a camera-only agent is partially observable rather than merely noisy. A vision-only latent model may still score well on passive frame prediction when the scene dominates the image and hidden posture has little visual effect. The deficiency appears under control, where posture changes which actions are viable. This is another reason passive prediction scores do not by themselves establish closed-loop competence.

This is why proprioception is not a nice-to-have sensor. Joint encoders and other proprioceptive sensors often provide high-rate, low-latency measurements of the agent's own state, although their reliability and calibration still depend on the hardware and estimator. In Part III: Representing Agents and Worlds's opening chapter on observations and sensors, we classified modalities by what they measure. Proprioception primarily measures the self; exteroception primarily measures the world. A world model for an embodied agent typically needs both, fused into a single latent state.

The minimal kinematic self-model has two parts:

• Forward kinematics: the map $f$ from joint configuration to body geometry, for example the end-effector position $x = f(q)$.
• The Jacobian: the derivative $J(q) = \partial f / \partial q$, which linearizes the map and tells you how end-effector motion responds to small joint motions. Resolved-rate controllers and many local inverse-kinematics methods consume $J(q)$ directly; model-predictive controllers may instead use other derivatives or a full dynamics model.

Why does the Jacobian deserve attention alongside the map itself? Local inverse kinematics asks which small joint adjustment moves the hand toward a target. The Jacobian answers that differential question. For redundant or constrained arms, one practical iterative method repeatedly solves a linearized problem $J(q)\,\Delta q = \Delta x$, takes a controlled step, and re-linearizes. Resolved-rate controllers and many local inverse-kinematics methods use some version of this loop, which is why the learned-schema experiment checks a derivative instead of reporting only point predictions.

The Jacobian's role in control can be seen directly by running the resolved-rate loop $J(q)\,\Delta q = \Delta x$ from a starting posture toward a target. The first few steps need not reduce the residual monotonically because each solve uses a local linearization. Once the arm enters the target's neighborhood, the damped updates produce near-geometric decay until floating-point precision becomes the limiting scale.

Dynamics (masses, inertias, gravity, friction, and contact forces) sit on top of kinematics. A kinematic schema answers "where will my hand be if my joints move like this"; a dynamic schema answers "what torques achieve that motion, and how fast." Kinematics can be sufficient for geometric path generation in slow, position-controlled manipulation when a lower-level controller handles the dynamic effects. It is not a claim that those effects disappear. Heavy payloads, fast motions, and contact-rich tasks make dynamics important at the planning layer as well. The classical robotics literature on computed-torque control and operational-space formulations shows how the two layers compose; a learned agent faces the same layering choice, deciding which dynamic structure must be explicit for the task.

Classical robotics writes $f(q)$ in closed form from CAD models. This works when the robot matches its blueprint but requires recalibration when damage, deformation, wear, or an unmodeled payload changes the plant. Bongard, Zykov, and Lipson demonstrated an adaptive alternative in their 2006 Science paper on resilient machines: a four-legged robot used actuation-sensation data to select a self-model, detected disagreement after damage, and synthesized a compensating gait. The lesson generalizes beyond that platform: a self-model estimated from data can be revised when observations reveal mismatch.

The important distinction is not analytic versus learned, but monitored versus unmonitored and fixed versus updateable. Either an analytic or learned model can be checked through prediction residuals. A parameterized or learned model may be easier to revise when those residuals reveal damage or wear. The resilient robot follows that loop: disagreement grows, the agent detects it, and it searches for an updated self-model. This is closely related to the filtering logic from Part II: Inference, Decisions, and Control, where an unexpectedly large innovation can trigger diagnosis or re-estimation.

Learning a body schema gives the agent unusual influence over its calibration data: it can choose safe motor commands and observe their consequences. That does not mean full control of the distribution. Joint limits, contacts, safety constraints, actuator health, and sensor noise restrict which probes are possible and what they reveal. Even so, self-model data can often be gathered more actively than data about an uncontrolled environment. A well-designed embodied agent can exploit that advantage through periodic calibration actions and residual monitoring, while respecting the same coverage and safety constraints that govern any system-identification experiment.

The same machinery that models the arm can model a held tool. Kinematically, a gripper holding a rigid rod is a chain with an added geometry and attachment transform. The agent can learn an extended forward map $f_{\text{tool}}(q, \theta^{\text{kin}}_{\text{tool}})$, where $\theta^{\text{kin}}_{\text{tool}}$ contains quantities such as length, shape, and grasp transform. A dynamic extension would separately include mass, inertia, compliance, and friction. Grasping a tool can therefore involve estimating parameters and selecting the appropriate extension. Within a fixed parameterized tool family, a new tool may be handled as new parameter values rather than an entirely new model. We will see the same compositional idea again when we treat object-centric structure in Object-Centric and Relational Worlds and physical constraints in Part IV: Space, Physics, and Causality.

The parameter-estimation view also suggests a computational account of the brief calibration period when a person picks up a new tool. The agent can actively probe length, orientation, and response, then use the estimate $\hat{\theta}_{\text{tool}}$ in an extended schema. This account predicts characteristic errors, such as reaching short with an underestimated rake length, that should shrink as calibration samples accumulate. Human and primate studies do show task-dependent changes in body and peripersonal-space representations after tool use. Those observations are qualitatively consistent with calibration, but they do not establish that the nervous system performs the specific parameter-estimation procedure written here.