Robotic Manipulation › Robotic Manipulation

Imagine moving a mug across a table with a robot arm. A push might slide it, tip it, or miss it; a grasp introduces different contact and force questions. From a camera image alone, the controller may not know the mug's exact pose, mass distribution, or friction. This is a useful thought experiment, not a measured twelve-second robot trial. The executable example later is narrower: a two-dimensional pusher and puck with a deliberately simple contact rule.

Perception, contact, and control interact: pose error changes the selected action, and that action changes the next observation. A world model can help a controller compare possible consequences, but agreement among imperfect components is not enough for correctness. Predicted motion must be checked against observed outcomes under the intended task and sensing conditions.

Analytic mechanics remains useful for known geometry and calibrated contacts. Deformable objects, uncertain surface properties, and partial observation make those models harder to apply directly; learned predictors can complement them by estimating effects from interaction data. Neither a rigid-body model nor a learned model by itself resolves arbitrary cloth, cables, or granular materials.

Learning from data can help identify task-relevant effects, but it does not prove that color, material, texture, velocity, or history are irrelevant. Which variables may be discarded depends on the object and task. We will distinguish explicit geometric state, learned latent state, and hybrids that retain known structure while estimating uncertain effects.

The chapter asks what a manipulation model needs to predict and how a controller uses that prediction. We compare state choices and the distinct demands of grasping, pushing, and dexterity; then examine visual foresight and affordances as related approaches. The worked experiment tests only planar pushing: two small regressors, one-step and recursive errors, an open-loop random-shooting plan, and a single-action effect field. The broader concepts connect to Part III, Part IV, Part V, and Part VIII.

Contact is one recurring challenge: active constraints can change when bodies meet, slip, or separate. State choice, sensing, uncertainty, and planning horizon matter too. The toy below isolates one contact switch without claiming to model full manipulation.