Robotic Manipulation › Grasping, Pushing, and Dexterous Control

Grasping, pushing, and dexterous manipulation emphasize different contacts and outcomes, but they overlap. A push can maintain contact over an interval, and a grasp can involve slipping or repositioning. The useful state, prediction horizon, and sensors depend on the actual task rather than the label alone.

Grasp planning may estimate form or force closure from geometry, rank candidates with a learned grasp-quality predictor, or simulate what follows contact. These are different tools. A predictor of grasp success from depth or RGB-D observations need not be a component of a world model.

Dex-Net 2.0 is one influential grasp-quality example: it trained on 6.7 million synthetic point-cloud, grasp, and analytic-quality examples, rather than tens of thousands of real labeled attempts. Its grasp-quality score answers a narrower question than a rollout model. A task that requires pre-grasp rearrangement or evaluation of an object after pickup may need additional prediction or planning, but not necessarily one monolithic world model.

Depending on the task, a grasping model may need to predict some of the following:

• The contact configuration between gripper and object.
• The force distribution across the contact patches.
• The object's response: does it stay gripped, slip, rotate, or tip?
• The downstream consequences: once the gripper has the object, what task-relevant properties (pose, orientation, stability) does it now have?

An immediate success score does not by itself evaluate later use of the object. A task-aware model could compare candidate grasps by predicted downstream outcomes, provided its dynamics and reward model cover those outcomes. A single model serving different tasks by changing only the objective is a design possibility, not a general guarantee.

Pushing moves an object through contact without closing a gripper. Contact can be brief or sustained; pose, contact point, action direction, and friction may all matter. Quasi-static planar pushing is one useful approximation, not the definition of pushing.

Under appropriate slow-motion and frictional conditions, a quasi-static approximation can avoid explicit inertial state. It still needs relevant contact geometry and frictional assumptions. Fast motion and post-contact sliding can violate it. Our discrete toy rule below is not a quasi-static force model.

A pushing model can ask: given the pusher's displacement $u_t$ and the current puck pose $p_t$, what is the new puck pose $p_{t+1}$? Under suitable quasi-static conditions, the answer can depend on contact geometry and friction as well as the commanded motion. Write the pusher pose as $q_t$, the pusher–object contact point as $c(p_t,q_t)$, and object geometry as $G$. Holding other friction and contact parameters implicit in $f$, a schematic transition is:

$$p_{t+1} = f(p_t, u_t;\, c(p_t, q_t), G)$$

where:

• $p_{t+1}$: the new puck pose after the push, the quantity the model predicts
• $p_t$: the current puck pose
• $q_t$: the current pusher pose
• $u_t$: the commanded pusher displacement
• $c(p_t, q_t)$: the contact point induced by the puck and pusher geometry
• $G$: the object geometry, which determines where contact forces act
• $f$: a schematic push transition; its form and regularity depend on the contact and friction model

Local contact features can focus a predictor on the part of the scene relevant to a push; this is one way to add physical structure (Part IV). Whether that helps data efficiency depends on the state representation, task distribution, and model. Our example compares two feature sets under one synthetic distribution.

Contact may be sensed directly with tactile or force sensors, inferred from vision, or left latent. Accurate pose prediction does not prove that a model internally represents contact geometry in a particular way. In our toy, the contact-proximity feature is supplied to one predictor, not discovered through supervision on puck motion alone.

Dexterous in-hand manipulation can involve several independently moving fingers and changing contacts. An object with six pose degrees of freedom is not necessarily directly actuated in all of them: its motion is mediated by contact constraints and external forces such as gravity. In-hand rotation of a cube and rolling a ball between fingers are examples. Unlike the simple push and grasp-success examples above, contact can change repeatedly while the object remains supported. Multi-object pushing and complex grasping can also be high-dimensional.

Depending on the task and sensors, useful dexterous-model predictions may include:

• The object's pose in the hand across a motion that may involve occasional break of contact.
• The contact state on each fingertip: sticking, slipping, or rolling.
• The forces required to maintain the target pose without crushing the object.
• Task outcomes, like the final orientation of the cube or the position of a screw.

Three fingers with two contact coordinates each plus a six-degree-of-freedom object pose give twelve continuous coordinates in one explicit representation; discrete contact modes add combinations. This is an illustration, not a claim that every pushing or grasping task has fewer state variables. A latent model can compress this representation, but compression does not guarantee preservation of distinctions needed for control. The state choice depends on sensing, task, and evaluation.

Some tasks combine pushing, grasping, carrying, and in-hand reorientation. A system may use separate predictors at different resolutions, but model handoffs then introduce their own state-estimation and timing errors. Other systems use a shared policy or predictor; the appropriate decomposition remains an empirical design choice.