Self-Models, Body Schemas, and Affordances › Where the Agent Ends and the World Begins

In the controlled dynamical systems we studied in Part II: Inference, Decisions, and Control, we wrote the world as a transition model $z_{t+1} = F(z_t, a_t) + w_t$, where $w_t$ is process noise, and treated $a_t$ as an input channel distinct from everything else. But which physical variables belong inside $z_t$, and which are left outside as "environment"? For a robot arm gripping a mug, the joint angles belong to the agent. The mug's pose belongs to the environment. What about the mug's pose relative to the gripper once contact is made? What about a tool the arm is holding? The placement of those variables is a modeling choice with downstream consequences.

It matters because the two sides of the boundary are usually treated differently downstream. Variables on the self side often have stronger action coupling and more direct proprioceptive readout, so the agent may have lower-latency, higher-confidence estimates of them. Variables on the environment side rely more heavily on prediction and inference, with greater uncertainty and delay. These are tendencies rather than definitions: internal actuator states can be hidden, while an external object can be observed precisely. If a variable that behaves like part of the body is filed under "environment," the agent may underuse reliable action and sensor structure. If a variable that behaves like the world is filed under "self," the agent may become overconfident about something it cannot command. The Kalman filters and controllers of Part II: Inference, Decisions, and Control assumed this partition was given to them; learning agents must discover or design it, and the choice shapes everything downstream, from which latent dimensions receive action inputs to which parts of the model transfer to a new task.

Three properties guide where to draw the line:

To make the criteria concrete, score a set of representative variables on the first two properties: how directly a command moves them, and how directly a sensor reads them. Variables near the top right are natural members of $z^{\text{self}}$; variables near the bottom left belong to the environment; and the off-diagonal quadrants are exactly the ambiguous cases the text describes.

• Direct controllability: variables in $z^{\text{self}}$ respond to actions through short, reliable causal paths. Commands change joint torques within milliseconds; they change the mug's location only through contact dynamics.
• Direct observability: dedicated sensors such as joint encoders, IMUs, muscle spindles, and battery monitors provide measurements closely tied to self-state, often with less scene interpretation than cameras or lidar. They can still require filtering and state estimation; an IMU, for example, does not directly report pose.
• Transportability across tasks: the self-model is the part of the world model that persists when you move the robot to a new kitchen. The mug model does not. Factoring the self out of the environment model is therefore a powerful form of transfer: you can relearn the kitchen without relearning your arm.

Each of these criteria can fail independently. A variable can be controllable but hard to observe: joint torques respond to commands, but some robots lack direct joint-torque sensing and must estimate them. A variable can be observable but not controllable: a fixed camera sees the room but cannot move it. A variable can be both controllable and observable yet fail to transfer: a gripper's learned friction model depends on the objects it has gripped. Transportability is valuable because the self-model is meant to be reused, but whether a variable belongs there still depends on the downstream task.

This connects directly to the state-abstraction machinery from Sufficient State and State Abstraction in this part. There, we asked which observation features a policy needs. Here we ask a complementary question: which latents are action-controllable, and which are merely observable? In a learned latent dynamics model, some latent dimensions will move sharply and predictably under actions while others respond only weakly or not at all. Identifying a controllable subspace is one useful step toward discovering the body. In classical robotics the partition is handed to you: the manufacturer tells you which encoders are on the arm. In a learned world model trained purely on observation-action sequences, the model instead has to infer which dimensions its actions steer. Short response latency, action-conditioned mutual information, and persistence across scenes are useful heuristics for proposing controllable latents, but none uniquely identifies self-state. Recovering a defensible self/nonself split therefore requires intervention and transfer tests, not a single statistical signature.

The boundary is also not fixed. When a person uses a rake, their effective reaching range changes; neuropsychologists observe that peripersonal space (the region the brain treats as "within reach") expands to include the tool. A good computational analogue: after a few minutes of driving, a driver's self-model extends to the car's corners, and experienced drivers know "where the hood is" without seeing it. We will return to tool incorporation when we discuss learned body schemas. The plasticity of the boundary is not a biological curiosity; it is a design resource. A robot that can dynamically reclassify a grasped object as part of itself inherits a compact, already-calibrated kinematic model for it, rather than treating every object-hand interaction as a fresh contact-dynamics problem. The cost is symmetric: the boundary must also contract when the object is dropped, and a stale extension is worse than no extension, because the agent will confidently predict motion (its own, extended) that does not occur.

Finally, note a subtlety about actions themselves. In the POMDP formalism from Markov and Partially Observable Decision Processes, the action is an input, not part of the state. But real actuators have state: motor temperature, battery charge, worn gears, a gripper that is partially closed. A complete self-model includes these actuator states inside $z^{\text{self}}$, because they shape how commands translate into forces. Ignoring them is a classic source of model error that worsens over a long episode. The failure pattern is characteristic and worth recognizing: the model is accurate early in the episode, when actuator states are near their nominal values, and drifts later as batteries deplete and motors heat. An agent that evaluates its model only on short rollouts will never see the problem; an agent that tracks its own actuator states can predict the onset of the discrepancy, which is exactly the kind of self-knowledge that separates a system that fails gracefully from one that fails inexplicably.