Video Generators as Candidate World Models › Controllability: Prompts, Cameras, Actions
World Models HandbookAll chapters Video Generators as Candidate World Models › Controllability: Prompts, Cameras, ActionsOne section · Sections in this chapter6 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
For this chapter, controllability means that a specified input changes the generated future in a predictable, task-relevant way. Three conditioning channels need different tests.
Text prompts. A phrase such as "a red ball rolls right" can specify semantic content and sometimes temporal order. Unless the system validates a mapping from prompt to action timing and units, prompt sensitivity is not evidence that it can execute a controller's timed command.
Camera and viewpoint controls. Specified camera pose, intrinsics, or motion can change the rendered observation while leaving world state fixed. In an embodied system, moving a camera can also move hardware or alter future information; state explicitly which mechanism is under test. Do not score camera-induced image motion as an object-action response.
Time-indexed action sequences. For a planner issuing physical controls, each has a defined time interval and meaning. Evaluate whether output changes with that action at the appropriate time and magnitude. A free-form text description does not establish an equivalent interface without calibration and validation.
All three can change pixels, but an output change alone does not identify its cause. Use controls that hold the scene and camera fixed while varying a feasible world action, and a separate camera intervention with the scene fixed. If the generator responds to these in indistinguishable ways, its outputs are unsuitable for a planner that needs their effects separated.
From a resettable scene with the same initial state and observation history , execute two feasible actions in the reference environment. Condition the generator on each action while holding other inputs fixed. Where possible, pair random seeds to reduce sampling variance, then repeat across seeds. Compare the distribution and timing of model branch differences with the reference branches; a zero difference is not an error when the two actions truly have the same task-relevant effect.
Matched branches expose weak action sensitivity when the reference actions have distinguishable effects. The toy corridor shows an action-blind predictor returning identical branches for left and right pushes, while the reference branches separate. This controlled comparison is stronger than comparing two attractive clips from unrelated initial states, but it still tests only the chosen actions, histories, and outcome measure.
Training-policy coverage is a separate axis from causal validity. A rarely chosen action has little direct evidence in that context, and a smooth generated response may still be extrapolation. A correct response on supported actions shows local predictive competence for the tested distribution; it does not by itself prove the dataset is adequate or that interventions outside support will generalize. Report action frequencies and evaluate supported and shifted action regimes separately.