Robotic Manipulation › Visual Foresight and Action-Conditioned Video

Visual foresight predicts future observations conditioned on candidate actions rather than requiring an explicit object-pose output. Images reveal some task-relevant changes but omit hidden forces, occluded geometry, and material properties. A video model therefore does not automatically predict the full physical state or generalize to every visible scene.

Given observation history $o_{\le t}$ and candidate actions $a_{t:t+H-1}$, a video model predicts future observations $o_{t+1:t+H}$. This action-conditioned view connects to Part IX. Design choices include:

• How actions are represented: end-effector deltas, joint commands, or other controls.
• Whether image prediction preserves object motion and contacts relevant to the task.
• How to evaluate both prediction and downstream action quality.

Some manipulation systems use compact controls, though the action-space dimension and prediction difficulty vary. A visually imperfect rollout can still support useful actions if it preserves the task-relevant effects. Conversely, plausible frames need not yield effective plans, so downstream evaluation matters alongside visual prediction metrics.

Training data can come from autonomous interaction, teleoperation, scripted actions, or simulation. Finn and Levine's action-conditioned video prediction and Ebert and colleagues' visual foresight work illustrate planning from robot interaction data without explicit object-pose labels in their demonstrated settings. Avoiding pose labels does not eliminate tracking, occlusion, or out-of-distribution problems; cloth, ropes, and fluids can be particularly hard to model reliably.

Rollouts can accumulate errors, especially near uncertain contacts, and video prediction can be computationally costly. Recurrence, latent frames, or generative sampling (Part V) offer different trade-offs; none guarantees a particular useful horizon. That horizon must be measured for the data, model, and control task at hand.

Given a video model, planning can follow the template introduced in Part VII:

1. Sample a set of candidate action sequences from a simple proposal distribution.
2. Roll out the video model for each candidate on the conditioned current observation.
3. Score each predicted clip by a task cost: distance of the object to the target in the final frame, presence of contact, stability of the predicted motion.
4. Execute a short prefix of a selected sequence, observe again, and re-plan; this is a closed-loop option, not what the later toy planner implements.

An image-based goal scorer may compare pixels, track designated points, segment an object, or estimate pose; each choice can be noisy. Receding-horizon replanning or a lower-level feedback controller may correct observed errors, subject to sensing and actuation limits.

Early action-conditioned video-planning papers demonstrated useful pushing and rearrangement behavior without explicit object-pose supervision in those experiments (Finn and Levine; Ebert et al.). Latent and pixel-space approaches make different computational and diagnostic trade-offs, but neither is uniformly easier to train or better at longer horizons. Some methods decode images for scoring or inspection; others plan entirely in latent space.