Robotic Manipulation › Object State and Contact-Rich Dynamics

Before predicting the next state, we must decide which quantities to represent. They can be assembled into a vector, graph, image, or latent history, but that choice determines what the transition model can update and what the planner can evaluate. Missing task-relevant variables cannot be recovered from a perfect transition fit to an insufficient state alone.

One explicit geometric representation assigns poses to relevant rigid bodies: the end-effector, gripper fingers, manipulated objects, and obstacles. A rigid-body pose lives in $SE(3)$, the special Euclidean group of rigid transformations, and can be written as:

$$T = \begin{bmatrix} R & t \\ 0 & 1 \end{bmatrix}$$

where:

• $T \in SE(3)$: the $4 \times 4$ homogeneous transformation matrix combining rotation and translation
• $R \in SO(3)$: the $3 \times 3$ rotation matrix that captures orientation
• $t \in \mathbb{R}^3$: the $3 \times 1$ translation vector that captures position

The rotation part has three degrees of freedom and the translation part has three, so a pose has six degrees of freedom in total. The homogeneous form is convenient because it makes composition of transforms a matrix multiplication: applying one transform after another is just multiplying their matrices, and finding the relative transform between two bodies is a single matrix inverse followed by a product. That algebraic convenience is why pose-based state representations are so common in robotics: the geometry of a scene becomes linear algebra over $4 \times 4$ matrices.

Pose alone is not a complete dynamical state. Velocities, robot joint states, shape and material properties, and contact geometry or history can matter even for rigid objects. A deformable object may require a mesh, particles, or another compact representation. Contact modes summarize useful information, but they are not the only missing variables. Two arrangements with different contact geometry generally also differ in their poses relative to object shapes; a contact graph is one representation of that distinction, not the sole source of it.

The table lists common design choices without ranking their accuracy. Any of them can succeed or fail depending on observation quality, training coverage, dynamics, and the downstream decision.

Manipulation and locomotion both include contact-mode changes. In a simplified rigid-contact model, dynamics may be smooth within a fixed mode but switch at touch, slip, or separation. A physical impact, a compliant-contact model, and a discrete-time simulator need not have the same mathematical regularity. The hard switch below is a property of our toy simulator, not a theorem that every manipulation transition is discontinuous.

A one-dimensional version of the code's pushing rule makes its hard switch concrete. First move the pusher by the commanded displacement $u_t$, clipping it to the workspace if necessary; then check its distance to the puck at its new position $\tilde q_{t+1}$. The puck update is

$$p_{t+1}=p_t+\alpha u_t\,\mathbb{1}\!\left[|\tilde q_{t+1}-p_t|<r_c\right],\qquad \tilde q_{t+1}=\operatorname{clip}(q_t+u_t,-B,B).$$

where:

• $p_t$ and $q_t$: puck and pusher positions before the step; $\tilde q_{t+1}$ is the pusher position after its commanded move and workspace clipping.
• $u_t$: the commanded displacement, not necessarily the displacement actually achieved after clipping.
• $B$: workspace coordinate bound; $r_c$: the distance threshold in this toy contact test.
• $\alpha$: an ad hoc coupling gain, not a Coulomb-friction coefficient.
• $\mathbb{1}[\cdot]$: one when the post-move pusher–puck distance is below $r_c$, zero otherwise.

When the post-move test is on, the puck receives $\alpha u_t$ in this toy rule; otherwise it stays put. The code applies the full command to the puck even if the pusher is clipped at the workspace edge. That is a simplifying artifact, not a physical contact law. For fixed nonzero $u_t$, the next puck position jumps as the contact indicator changes. Away from the switch its derivative with respect to $p_t$ is one; at the switch the hard map is not differentiable. It is misleading to describe this as a large finite curvature.

A local linearization within the no-contact region misses the effect of an action that crosses the threshold in the same step. Smoothing the indicator can make a fitted predictor easier to optimize, while an explicit contact mode keeps the switch visible. Neither choice alone guarantees good plans near contact.

This is why contact-rich dynamics receive separate treatment in Part IV. Depending on the system, they can introduce several difficulties:

• Mode changes. A hard simulator may switch between no contact, sticking, and sliding; compliant models can smooth some transitions.
• Outcome uncertainty. Uncertain state or parameters can yield several plausible outcomes near a mode boundary.
• Long-horizon dependencies. An early grasp can affect a later pour, so an error may propagate into the task result.
• Partially observed parameters. Material and contact properties may need estimation from interaction or additional sensors.

Possible responses include contact features or explicit modes, uncertainty-aware predictions, history-conditioned state estimation, and online parameter updates. Their value is empirical: a one-step error metric alone will not reveal every planning failure, so evaluate both prediction and closed-loop behavior as discussed in Part I.

The four options in the table trade off observability, structure, estimation error, and computational cost. No three-question rule automatically selects a representation.

Explicit object state plus contact mode. Poses, velocities, material parameters, and contact labels can support a structured physical model when they are accurately known or estimated. Pose and mode alone are not generally a sufficient state, and explicit models are not inherently more predictive than learned alternatives.

Latent state learned from observations. An encoder can summarize RGB, depth, and proprioceptive history in $z_t$, then predict a successor conditioned on action. RSSMs and Dreamer illustrate this pattern (Part V, Part VIII). It avoids a prescribed object list but may omit rare contacts; recurrence can help retain useful history without ensuring that every hidden property is recovered.

Object-centric state. Slots or entity vectors can make object identity and interaction structure explicit (Part III, Part IV). They may help maintain identity across steps, but segmentation, tracking, and the learned interactions can fail; no general long-horizon advantage over monolithic latents follows from the representation alone.

Hybrid state. A system may combine calibrated robot kinematics with learned object dynamics or residual errors. Joint encoders do not make end-effector pose exact, and known kinematics do not by themselves confer safety. The model interface still needs validation against sensing, compliance, and contact effects.

Our code below makes a smaller comparison: two linear predictors over explicit positions, one with an additional contact-proximity feature. It does not implement a latent, object-centric, or hybrid robot model.

The transition target depends on the task. A point predictor can be useful, as in our toy, but a distribution is valuable when the sensed state or dynamics leave multiple plausible contact outcomes. Near a threshold, uncertainty in relative position can put probability mass on both no-contact and contact branches even if each fully specified branch is deterministic. Candidate outputs include contact-mode probabilities or mixtures over pose deltas. This distinction between uncertainty about state and uncertainty in dynamics also appears in Part IV.

Uncertain friction or contact geometry may reflect missing observations or unknown parameters, not irreducible physical randomness. Better sensing or system identification may reduce some of that uncertainty. Other effects can remain stochastic at the model's chosen resolution. A planner should represent whichever uncertainties materially change its action choice rather than assume that every contact is intrinsically multimodal.