Robotic Manipulation

Imagine moving a mug across a table with a robot arm. A push might slide it, tip it, or miss it; a grasp introduces different contact and force questions. From a camera image alone, the controller may not know the mug's exact pose, mass distribution, or friction. This is a useful thought experiment, not a measured twelve-second robot trial. The executable example later is narrower: a two-dimensional pusher and puck with a deliberately simple contact rule.

Perception, contact, and control interact: pose error changes the selected action, and that action changes the next observation. A world model can help a controller compare possible consequences, but agreement among imperfect components is not enough for correctness. Predicted motion must be checked against observed outcomes under the intended task and sensing conditions.

Analytic mechanics remains useful for known geometry and calibrated contacts. Deformable objects, uncertain surface properties, and partial observation make those models harder to apply directly; learned predictors can complement them by estimating effects from interaction data. Neither a rigid-body model nor a learned model by itself resolves arbitrary cloth, cables, or granular materials.

Learning from data can help identify task-relevant effects, but it does not prove that color, material, texture, velocity, or history are irrelevant. Which variables may be discarded depends on the object and task. We will distinguish explicit geometric state, learned latent state, and hybrids that retain known structure while estimating uncertain effects.

The chapter asks what a manipulation model needs to predict and how a controller uses that prediction. We compare state choices and the distinct demands of grasping, pushing, and dexterity; then examine visual foresight and affordances as related approaches. The worked experiment tests only planar pushing: two small regressors, one-step and recursive errors, an open-loop random-shooting plan, and a single-action effect field. The broader concepts connect to Part III, Part IV, Part V, and Part VIII.

Contact is one recurring challenge: active constraints can change when bodies meet, slip, or separate. State choice, sensing, uncertainty, and planning horizon matter too. The toy below isolates one contact switch without claiming to model full manipulation.

Before predicting the next state, we must decide which quantities to represent. They can be assembled into a vector, graph, image, or latent history, but that choice determines what the transition model can update and what the planner can evaluate. Missing task-relevant variables cannot be recovered from a perfect transition fit to an insufficient state alone.

One explicit geometric representation assigns poses to relevant rigid bodies: the end-effector, gripper fingers, manipulated objects, and obstacles. A rigid-body pose lives in $SE(3)$, the special Euclidean group of rigid transformations, and can be written as:

where:

• $T \in SE(3)$: the $4 \times 4$ homogeneous transformation matrix combining rotation and translation
• $R \in SO(3)$: the $3 \times 3$ rotation matrix that captures orientation
• $t \in \mathbb{R}^3$: the $3 \times 1$ translation vector that captures position

The rotation part has three degrees of freedom and the translation part has three, so a pose has six degrees of freedom in total. The homogeneous form is convenient because it makes composition of transforms a matrix multiplication: applying one transform after another is just multiplying their matrices, and finding the relative transform between two bodies is a single matrix inverse followed by a product. That algebraic convenience is why pose-based state representations are so common in robotics: the geometry of a scene becomes linear algebra over $4 \times 4$ matrices.

Pose alone is not a complete dynamical state. Velocities, robot joint states, shape and material properties, and contact geometry or history can matter even for rigid objects. A deformable object may require a mesh, particles, or another compact representation. Contact modes summarize useful information, but they are not the only missing variables. Two arrangements with different contact geometry generally also differ in their poses relative to object shapes; a contact graph is one representation of that distinction, not the sole source of it.

The table lists common design choices without ranking their accuracy. Any of them can succeed or fail depending on observation quality, training coverage, dynamics, and the downstream decision.

Manipulation and locomotion both include contact-mode changes. In a simplified rigid-contact model, dynamics may be smooth within a fixed mode but switch at touch, slip, or separation. A physical impact, a compliant-contact model, and a discrete-time simulator need not have the same mathematical regularity. The hard switch below is a property of our toy simulator, not a theorem that every manipulation transition is discontinuous.

A one-dimensional version of the code's pushing rule makes its hard switch concrete. First move the pusher by the commanded displacement $u_t$, clipping it to the workspace if necessary; then check its distance to the puck at its new position $\tilde q_{t+1}$. The puck update is

where:

• $p_t$ and $q_t$: puck and pusher positions before the step; $\tilde q_{t+1}$ is the pusher position after its commanded move and workspace clipping.
• $u_t$: the commanded displacement, not necessarily the displacement actually achieved after clipping.
• $B$: workspace coordinate bound; $r_c$: the distance threshold in this toy contact test.
• $\alpha$: an ad hoc coupling gain, not a Coulomb-friction coefficient.
• $\mathbb{1}[\cdot]$: one when the post-move pusher–puck distance is below $r_c$, zero otherwise.

When the post-move test is on, the puck receives $\alpha u_t$ in this toy rule; otherwise it stays put. The code applies the full command to the puck even if the pusher is clipped at the workspace edge. That is a simplifying artifact, not a physical contact law. For fixed nonzero $u_t$, the next puck position jumps as the contact indicator changes. Away from the switch its derivative with respect to $p_t$ is one; at the switch the hard map is not differentiable. It is misleading to describe this as a large finite curvature.

A local linearization within the no-contact region misses the effect of an action that crosses the threshold in the same step. Smoothing the indicator can make a fitted predictor easier to optimize, while an explicit contact mode keeps the switch visible. Neither choice alone guarantees good plans near contact.

This is why contact-rich dynamics receive separate treatment in Part IV. Depending on the system, they can introduce several difficulties:

• Mode changes. A hard simulator may switch between no contact, sticking, and sliding; compliant models can smooth some transitions.
• Outcome uncertainty. Uncertain state or parameters can yield several plausible outcomes near a mode boundary.
• Long-horizon dependencies. An early grasp can affect a later pour, so an error may propagate into the task result.
• Partially observed parameters. Material and contact properties may need estimation from interaction or additional sensors.

Possible responses include contact features or explicit modes, uncertainty-aware predictions, history-conditioned state estimation, and online parameter updates. Their value is empirical: a one-step error metric alone will not reveal every planning failure, so evaluate both prediction and closed-loop behavior as discussed in Part I.

The four options in the table trade off observability, structure, estimation error, and computational cost. No three-question rule automatically selects a representation.

Explicit object state plus contact mode. Poses, velocities, material parameters, and contact labels can support a structured physical model when they are accurately known or estimated. Pose and mode alone are not generally a sufficient state, and explicit models are not inherently more predictive than learned alternatives.

Latent state learned from observations. An encoder can summarize RGB, depth, and proprioceptive history in $z_t$, then predict a successor conditioned on action. RSSMs and Dreamer illustrate this pattern (Part V, Part VIII). It avoids a prescribed object list but may omit rare contacts; recurrence can help retain useful history without ensuring that every hidden property is recovered.

Object-centric state. Slots or entity vectors can make object identity and interaction structure explicit (Part III, Part IV). They may help maintain identity across steps, but segmentation, tracking, and the learned interactions can fail; no general long-horizon advantage over monolithic latents follows from the representation alone.

Hybrid state. A system may combine calibrated robot kinematics with learned object dynamics or residual errors. Joint encoders do not make end-effector pose exact, and known kinematics do not by themselves confer safety. The model interface still needs validation against sensing, compliance, and contact effects.

Our code below makes a smaller comparison: two linear predictors over explicit positions, one with an additional contact-proximity feature. It does not implement a latent, object-centric, or hybrid robot model.

The transition target depends on the task. A point predictor can be useful, as in our toy, but a distribution is valuable when the sensed state or dynamics leave multiple plausible contact outcomes. Near a threshold, uncertainty in relative position can put probability mass on both no-contact and contact branches even if each fully specified branch is deterministic. Candidate outputs include contact-mode probabilities or mixtures over pose deltas. This distinction between uncertainty about state and uncertainty in dynamics also appears in Part IV.

Uncertain friction or contact geometry may reflect missing observations or unknown parameters, not irreducible physical randomness. Better sensing or system identification may reduce some of that uncertainty. Other effects can remain stochastic at the model's chosen resolution. A planner should represent whichever uncertainties materially change its action choice rather than assume that every contact is intrinsically multimodal.

Grasping, pushing, and dexterous manipulation emphasize different contacts and outcomes, but they overlap. A push can maintain contact over an interval, and a grasp can involve slipping or repositioning. The useful state, prediction horizon, and sensors depend on the actual task rather than the label alone.

Grasp planning may estimate form or force closure from geometry, rank candidates with a learned grasp-quality predictor, or simulate what follows contact. These are different tools. A predictor of grasp success from depth or RGB-D observations need not be a component of a world model.

Dex-Net 2.0 is one influential grasp-quality example: it trained on 6.7 million synthetic point-cloud, grasp, and analytic-quality examples, rather than tens of thousands of real labeled attempts. Its grasp-quality score answers a narrower question than a rollout model. A task that requires pre-grasp rearrangement or evaluation of an object after pickup may need additional prediction or planning, but not necessarily one monolithic world model.

Depending on the task, a grasping model may need to predict some of the following:

• The contact configuration between gripper and object.
• The force distribution across the contact patches.
• The object's response: does it stay gripped, slip, rotate, or tip?
• The downstream consequences: once the gripper has the object, what task-relevant properties (pose, orientation, stability) does it now have?

An immediate success score does not by itself evaluate later use of the object. A task-aware model could compare candidate grasps by predicted downstream outcomes, provided its dynamics and reward model cover those outcomes. A single model serving different tasks by changing only the objective is a design possibility, not a general guarantee.

Pushing moves an object through contact without closing a gripper. Contact can be brief or sustained; pose, contact point, action direction, and friction may all matter. Quasi-static planar pushing is one useful approximation, not the definition of pushing.

Under appropriate slow-motion and frictional conditions, a quasi-static approximation can avoid explicit inertial state. It still needs relevant contact geometry and frictional assumptions. Fast motion and post-contact sliding can violate it. Our discrete toy rule below is not a quasi-static force model.

A pushing model can ask: given the pusher's displacement $u_t$ and the current puck pose $p_t$, what is the new puck pose $p_{t+1}$? Under suitable quasi-static conditions, the answer can depend on contact geometry and friction as well as the commanded motion. Write the pusher pose as $q_t$, the pusher–object contact point as $c(p_t,q_t)$, and object geometry as $G$. Holding other friction and contact parameters implicit in $f$, a schematic transition is:

where:

• $p_{t+1}$: the new puck pose after the push, the quantity the model predicts
• $p_t$: the current puck pose
• $q_t$: the current pusher pose
• $u_t$: the commanded pusher displacement
• $c(p_t, q_t)$: the contact point induced by the puck and pusher geometry
• $G$: the object geometry, which determines where contact forces act
• $f$: a schematic push transition; its form and regularity depend on the contact and friction model

Local contact features can focus a predictor on the part of the scene relevant to a push; this is one way to add physical structure (Part IV). Whether that helps data efficiency depends on the state representation, task distribution, and model. Our example compares two feature sets under one synthetic distribution.

Contact may be sensed directly with tactile or force sensors, inferred from vision, or left latent. Accurate pose prediction does not prove that a model internally represents contact geometry in a particular way. In our toy, the contact-proximity feature is supplied to one predictor, not discovered through supervision on puck motion alone.

Dexterous in-hand manipulation can involve several independently moving fingers and changing contacts. An object with six pose degrees of freedom is not necessarily directly actuated in all of them: its motion is mediated by contact constraints and external forces such as gravity. In-hand rotation of a cube and rolling a ball between fingers are examples. Unlike the simple push and grasp-success examples above, contact can change repeatedly while the object remains supported. Multi-object pushing and complex grasping can also be high-dimensional.

Depending on the task and sensors, useful dexterous-model predictions may include:

• The object's pose in the hand across a motion that may involve occasional break of contact.
• The contact state on each fingertip: sticking, slipping, or rolling.
• The forces required to maintain the target pose without crushing the object.
• Task outcomes, like the final orientation of the cube or the position of a screw.

Three fingers with two contact coordinates each plus a six-degree-of-freedom object pose give twelve continuous coordinates in one explicit representation; discrete contact modes add combinations. This is an illustration, not a claim that every pushing or grasping task has fewer state variables. A latent model can compress this representation, but compression does not guarantee preservation of distinctions needed for control. The state choice depends on sensing, task, and evaluation.

Some tasks combine pushing, grasping, carrying, and in-hand reorientation. A system may use separate predictors at different resolutions, but model handoffs then introduce their own state-estimation and timing errors. Other systems use a shared policy or predictor; the appropriate decomposition remains an empirical design choice.

Visual foresight predicts future observations conditioned on candidate actions rather than requiring an explicit object-pose output. Images reveal some task-relevant changes but omit hidden forces, occluded geometry, and material properties. A video model therefore does not automatically predict the full physical state or generalize to every visible scene.

Given observation history $o_{\le t}$ and candidate actions $a_{t:t+H-1}$, a video model predicts future observations $o_{t+1:t+H}$. This action-conditioned view connects to Part IX. Design choices include:

• How actions are represented: end-effector deltas, joint commands, or other controls.
• Whether image prediction preserves object motion and contacts relevant to the task.
• How to evaluate both prediction and downstream action quality.

Some manipulation systems use compact controls, though the action-space dimension and prediction difficulty vary. A visually imperfect rollout can still support useful actions if it preserves the task-relevant effects. Conversely, plausible frames need not yield effective plans, so downstream evaluation matters alongside visual prediction metrics.

Training data can come from autonomous interaction, teleoperation, scripted actions, or simulation. Finn and Levine's action-conditioned video prediction and Ebert and colleagues' visual foresight work illustrate planning from robot interaction data without explicit object-pose labels in their demonstrated settings. Avoiding pose labels does not eliminate tracking, occlusion, or out-of-distribution problems; cloth, ropes, and fluids can be particularly hard to model reliably.

Rollouts can accumulate errors, especially near uncertain contacts, and video prediction can be computationally costly. Recurrence, latent frames, or generative sampling (Part V) offer different trade-offs; none guarantees a particular useful horizon. That horizon must be measured for the data, model, and control task at hand.

Given a video model, planning can follow the template introduced in Part VII:

1. Sample a set of candidate action sequences from a simple proposal distribution.
2. Roll out the video model for each candidate on the conditioned current observation.
3. Score each predicted clip by a task cost: distance of the object to the target in the final frame, presence of contact, stability of the predicted motion.
4. Execute a short prefix of a selected sequence, observe again, and re-plan; this is a closed-loop option, not what the later toy planner implements.

An image-based goal scorer may compare pixels, track designated points, segment an object, or estimate pose; each choice can be noisy. Receding-horizon replanning or a lower-level feedback controller may correct observed errors, subject to sensing and actuation limits.

Early action-conditioned video-planning papers demonstrated useful pushing and rearrangement behavior without explicit object-pose supervision in those experiments (Finn and Levine; Ebert et al.). Latent and pixel-space approaches make different computational and diagnostic trade-offs, but neither is uniformly easier to train or better at longer horizons. Some methods decode images for scoring or inspection; others plan entirely in latent space.

The preceding sections mainly examined predictions of action effects, although grasp selection can also depend on later task outcomes. A manipulation task may also need to choose which object to interact with, where to reach, and in what order. Affordance estimates can help propose actions; when a world model is available, it can predict their consequences.

Learned affordance models can condition on a scene, action, and sometimes a goal. In a rightward-push task, a contact point on one side of an object might be useful, but the outcome depends on geometry, friction, and the action direction. The toy map below instead shows a predictor's one-step displacement for a fixed rightward command; it is not a goal-conditioned success map.

One possible affordance architecture has three components:

• Feature extractor. Encodes an image or structured scene state.
• Optional goal conditioning. Specifies a target pose, instruction, or image when the output is task-specific.
• Prediction head. Estimates an explicitly defined quantity, such as success probability, action value, or displacement.

Interaction logs or simulated outcomes can supervise such a head. For a probabilistic goal-conditioned formulation, $m(x\mid o,g,a)$ might estimate whether action $a$ at location $x$ helps achieve goal $g$ from observation $o$. Other heads predict an effect vector instead. Language-conditioned variants connect to Part IX, but their calibration and transfer require evaluation.

One possible architecture is a two-level loop:

1. High-level selector. Uses estimated action effects or feasibility to propose a sub-task or contact region.
2. Low-level controller. Executes a candidate action and uses feedback; it may use model-predictive control, but need not.

This decomposition can help diagnose errors, but a failed push may involve perception, effect prediction, planning, or control together. Component tests and interventions, rather than the architecture alone, help assign causes (Part XI).

Task-and-motion planning can also search over symbolic predicates and call geometric or learned feasibility checks (Part VII). A PDDL-style planner is one option; the affordance predictor above does not require one.

An affordance predictor can be learned directly from outcomes without an explicit world model. Where both are present, comparing predicted effects can expose disagreement: a proposed rightward push is suspect if the dynamics model instead predicts a tip. Joint training may encourage agreement but does not guarantee physical or task-level consistency. Testing the proposed action in the real system remains decisive.

We will build a miniature 2D pushing predictor. The state contains puck and pusher positions $s_t=(p_x,p_y,q_x,q_y)\in\mathbb{R}^4$, and the action is a commanded pusher displacement $a_t=(u_x,u_y)\in\mathbb{R}^2$. A post-move distance test gates an ad hoc puck coupling gain. This is a deliberately synthetic rule, not a physical friction model or a reproduction of a planar-pushing benchmark. It lets us inspect feature choice, prediction error, open-loop rollout, and a one-step effect map; it does not demonstrate grasping, dexterity, or visual foresight.

We will do six things:

1. Generate random-play transition data from the contact-switch simulator and inspect how rarely the pre-move proximity feature is appreciably nonzero.
2. Fit a naive linear world model that ignores contact.
3. Fit a contact-aware world model that has access to a contact feature.
4. Compare one-step and multi-step rollout error.
5. Plan a push to a target using random shooting against the learned model.
6. Query the model for a one-step effect field, an affordance proxy.

The core comparison gives one regression additional contact-proximity features. We report both one-step and recursive rollout error; neither alone measures task success.

Before building the models, we fix a random seed so every experiment is reproducible, and we keep each model's input features explicit so the comparison is fair.

Everything in this chapter runs in a single notebook, and the notebook follows a strict computation and rendering boundary: data generation, model fitting, rollout evaluation, planning, and affordance-map computation all happen in ordinary (non-theme-rerun) cells that run once. The figures are produced by separate presentation-only cells, each of which declares #| theme-rerun: true, imports the shared plotting style locally, calls use_book_style(), and renders only values that were prepared in earlier cells. No cell that draws a figure trains a model, samples from a simulator, or mutates shared data.

This cell seeds the examples and locates the repository root by walking upward from the working directory. Launch it from the repository or a descendant directory; it will not find the style module from an unrelated working directory.

The dynamics fit on one screen: the pusher moves by the commanded action subject to workspace clipping; the puck moves by a fixed fraction of the command when the pusher's post-move position passes a distance test. The gain is not a friction coefficient.

The simulator and feature function are separate. The latter measures pre-move proximity and therefore does not encode the simulator's post-move contact decision exactly.

We will give this feature to one of the two world models and withhold it from the other, so we can see the difference. The contact-aware model uses it as an extra input feature, while the naive model sees only the raw state and action.

We generate synthetic random-action episodes from the toy simulator. This is not a faithful model of robot data collection; in this sampling scheme, pre-move proximity is rare, which matters for an unweighted regression loss.

Each transition is stored as an independent (state, action, next_state) tuple, which keeps the regression dataset simple: random play is the data source, not a trained controller.

We compare a naive linear model that regresses the next state from the current state and action using only a fixed affine map, with no contact term of any kind:

where:

• $s_t \in \mathbb{R}^4$: the state at time $t$, the concatenation of the puck's position $(p_x, p_y)$ and the pusher's position $(q_x, q_y)$
• $a_t \in \mathbb{R}^2$: the pusher's commanded displacement $(u_x, u_y)$ during step $t$
• $A$: the learned $4 \times 4$ state-transition matrix, capturing how the current state propagates in the absence of action
• $B$: the learned $4 \times 2$ control matrix, mapping the commanded displacement into the next state
• $b$: the learned intercept vector (Ridge fits an intercept by default)
• $\hat{s}_{t+1} \in \mathbb{R}^4$: the model's predicted next state, whose error is measured against the simulator's true $s_{t+1}$
• The affine prediction approximates the true next state; its errors are measured below.

against a contact-aware model that augments the input with the contact feature and its product with the action. The fitted coefficients remain fixed. The extra nonlinear features let this linear regression represent an action-dependent contact response without an explicit latent contact mode:

Both models are small regressions on hand-crafted features; only one receives the smoothed contact feature and its action products. In this seeded run, the contact-aware model's one-step RMSE is only marginally lower. Contact is rare in the sampled random-play data, and the feature uses pre-move separation while the simulator checks post-move separation. The added feature therefore does not isolate the simulator's hard switch. This comparison tests what the feature contributes under this data distribution; it does not establish that contact-aware modeling is generally better or worse.

One-step and recursive errors answer different questions. A planner may feed predictions back into the model; recursive rollout exposes the resulting distribution shift. The following 15-step test does not measure closed-loop control or establish an asymptotic growth law.

The first 9,600 transitions come from 320 complete episodes and the remaining 2,400 from 80 complete episodes, so the one-step train/test slices do not split episodes. The recursive evaluation below draws fresh start states and actions (50 trials), rather than reusing those test episodes. Both models see identical actions within each trial.

We use the contact-feature model for a small random-shooting demonstration (Part VII). It samples 400 candidate 12-step action sequences, scores each by the model-predicted final puck distance to a target, and selects one. The chosen sequence is then replayed open-loop in the simulator. This is not receding-horizon control, and the score does not establish task success.

The planner searches only over actions; the contact-aware model supplies the prediction that scores each candidate sequence, which keeps the planning loop separate from the learned dynamics.

The fitted model can be queried for a one-step effect field: from each selected pusher start, apply the same rightward command and plot the predicted puck displacement. This is one deterministic model query per cell, not an average, probability, confidence estimate, or goal-conditioned success score. Arrow magnitudes may vary for reasons that include sparse contact data and model error; their strongest values need not occur to the left of the puck.

The map displays only start cells whose pre-move contact feature exceeds 0.05. This is a visualization filter, not a guarantee that the simulator's post-move contact test would accept or reject the same cell.

The arrow field queries the learned model at selected start positions without running the simulator again. Its pre-move filter keeps the display near the puck but can omit starts that would make contact after the commanded move. Each arrow encodes the model's predicted puck displacement for the same rightward action. A task planner could rank candidate starts with these predictions, then validate selected actions in closed loop. This map alone does not establish that the strongest-looking arrow corresponds to a successful real push.

The arrow magnitude is the model's predicted effect, not confidence. A large vector can be confidently right, confidently wrong, or uncertain; this plot cannot distinguish those cases. Comparing queries with held-out simulator transitions or real actions would test whether these effects are useful for choosing a push.

The key parameters used in the miniature pushing experiment are:

• R_CONTACT: Pusher-puck center-distance threshold for the toy simulator's contact switch.
• N_EPISODES: Number of random-play episodes collected for training and evaluation.
• ALPHA: Fraction of the commanded displacement inherited by the puck when the simulator's contact test succeeds.
• BOUND: Workspace half-width used to clip pusher positions.
• grid_vals: Sampled pusher start positions for the illustrative affordance field. Increase this to visualize the affordance field at finer spatial resolution.
• push_action: Fixed displacement used to query that field.
• max_mag: Maximum predicted displacement magnitude, used to normalize displayed arrow lengths.
• T_EPISODE: 30 transitions per episode; with 400 episodes, the dataset contains 12,000 transitions.
• Train/test split and Ridge alpha: An episode-aligned 80/20 split (320/80 episodes); each linear model uses Ridge(alpha=1e-4).
• HORIZON and evaluation counts: Recursive rollouts use 15 actions, with 50 fresh trials for the aggregate RMSE and 30 fresh trials for the per-step plot.
• Planner search settings: Random shooting samples 400 action sequences of length 12 with Gaussian action scale 0.15.

The miniature model isolates a feature-choice and rollout-evaluation problem. Translating those observations to a robot requires additional perception, actuation, contact, and safety tests.

The following are design questions, not measured performance claims about deployed manipulation systems.

The first limitation is observation realism. Our model sees exact positions. A physical system may have camera, depth, tactile, and proprioceptive measurements with occlusion, calibration error, and latency. Prediction should be tested under the actual sensing regime and material/lighting variation, not extrapolated from this state-observed toy. Reliability and evaluation methods are discussed in Part XII and Part XI. We have not measured a simulation-to-real error ratio here.

The second limitation is horizon. We evaluate only 15 steps from synthetic starts. Error rises over this window but the experiment does not establish exponential growth or any useful time limit on other models. Re-observing and replanning can reduce open-loop drift, while recurrent latent models (Part VIII) offer another design option. Their relative effectiveness must be measured on the intended task.

The third limitation is contact uncertainty. Our toy has one distance-gated interaction and no slip, rolling, or force sensing. A richer task may benefit from explicit modes, ensembles, or probabilistic predictions. None is universally preferable, and this chapter supplies no industrial deployment comparison.

The fourth limitation is task composition. A multi-stage system may hand object state between pushing, grasping, and placement components. Each interface needs evaluation because the receiving component may see inputs outside its training distribution. An affordance estimate alone does not solve this mismatch.

The central evaluation lesson is to keep prediction, uncertainty calibration, and decision quality separate. Our contact-feature regression slightly improves one-step RMSE but slightly worsens mean 15-step RMSE in this seeded run; the open-loop plan misses its target. Those results illustrate why a favorable local metric is insufficient evidence of a useful controller. Physical-robot impact would require separate task-success and safety evidence.

• Contact can change the relevant dynamics or make outcomes uncertain, but not every contact model has a hard discontinuity or requires distributional output.
• Explicit, latent, object-centric, and hybrid states trade off observability, structure, and estimation error; none dominates by definition.
• Grasping, pushing, and dexterous manipulation overlap and may call for different predictors or a shared model, depending on the task.
• Visual foresight predicts action-conditioned observations, not hidden physical state; its planning value must be tested downstream.
• Affordance estimates can be learned without a world model. If paired with one, compare their predicted effects rather than assuming consistency.
• The code builds a miniature planar pushing model and a naive baseline. Its contact feature gives only a marginal one-step gain and slightly worse mean rollout error in this seeded run; the imperfect model is also used for a coarse plan and an illustrative affordance map.
• Separate one-step prediction, recursive prediction, and closed-loop task success, as in Part XI.

The planned next chapter, Locomotion, Humanoids, and Navigation, examines whole-body contact and spatial control within Part X.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about robotic manipulation world models, contact-rich dynamics, and planning.