Actions, Inverse Dynamics, and Latent Actions › Limitations and Impact

The techniques in this chapter are what let world models move beyond environments where every action is pre-labeled. They are also where most of the practical difficulty lives, and it is worth being honest about the limitations before we move on.

Inverse dynamics is limited by identifiability. A state transition does not always reveal the underlying action. When the forward map is not injective, or when external perturbations mimic agent effects, a deterministic mean-regression model can average incompatible hypotheses; a probabilistic model can preserve several hypotheses but cannot identify the true one without more information. Pseudo-labels from IDM-based methods can carry these errors into downstream training.

Latent actions are not unique. Different seeds, architectures, or datasets can produce different latent spaces with the same reconstruction loss. Reconstruction-only formulations retain a reparametrization symmetry unless the model class or assumptions break it. Paired supervision is one way to choose real-control coordinates; invariance, canonicalization, and shared structure add alternative assumptions, but none supplies semantics for free.

Latent-action transfer across domains is not guaranteed. It becomes plausible when environments share action semantics, paired anchors, or jointly learned structure. Arbitrary cross-embodiment transfer remains an open problem because visually similar transitions can require different controls and identical controls can have different effects.

Controllability in learned representations is fragile. Even with clean action labels, a latent state can conflate action-relevant and action-irrelevant features. Inverse prediction and Jacobian-rank analysis catch selected one-step failures, while held-out planning rollouts test the combined model more directly. Linear systems have a clean controllability-rank condition; general learned nonlinear representations do not have an equally simple global test.

Passive data biases the model toward observed behavior. Video shows what the recorded agents happened to do, not necessarily the full range of feasible interventions. The learned latent action space is therefore shaped by the behavior policy and coverage of the dataset. The next chapter, Interaction Data and Passive Observation, takes this up as a first-class concern.

These ideas have nevertheless produced concrete systems. VPT used an IDM to pseudo-label online Minecraft video; Genie learned a latent action interface for interactive video generation; and LAPA used latent-action pretraining before adaptation to robot controls. Each result depends on its domain, data, and alignment procedure rather than establishing generic action recovery from arbitrary video.

The clearest way to read the field at this point is to see the action slot as a design decision rather than a given. When you build a world model, you are deciding what the action is and where it comes from. That decision determines whether the model can plan, connect its imagined controls to deployment, and learn useful control variables from unlabeled video.