Actions, Inverse Dynamics, and Latent Actions › Learning Actions from Unlabeled Video

Now we hit the hard part. Video is everywhere, but action labels are rare. Even in robotics, a robot that has been running for years in a lab might have thousands of hours of video and only a few hours of logged commands. And most of the world's video was never on a robot at all.

The goal here is to discover the action space from unlabeled sequences of observations $o_1, o_2, \dots, o_T$, where $o_t$ is the observation at time $t$ and the underlying action $a_t$ is hidden. We will refer to a learned substitute for $a_t$ as a latent action, typically denoted $z_t$. The latent action is our best guess at what the action was, inferred from the transition alone. The hope is that the latent action is more than a bookkeeping trick: that it behaves like a real action when used for planning and control.

A few distinct practical reasons cause actions to be missing.

• Sensor mismatch. A camera records everything, but nobody instrumented the car's steering wheel. The video and the control signal were captured by different systems that were never synchronized.
• Privacy or commercial sensitivity. Action logs may be hidden even when the video is public. This is common when the video is released for research but the underlying telemetry is proprietary.
• Cross-embodiment transfer. A human hand and a robot gripper have different native actions, but you would like a shared abstraction. Here the labels are not missing so much as incommensurable across embodiments.
• Video was the point. Games played on Twitch or cooking demos on YouTube exist for entertainment, not for training. Actions were never captured because capturing them was never the goal.
• Task-shift. A dataset may be labeled with high-level intent but not low-level controls. "Make dinner" is a label, but it does not tell a motor-level model what to do.

In all of these cases you still have transitions. You just do not have the labels that explain them. The question becomes: what can be recovered about the actions from the transitions alone?

The simplest practical approach is not really unsupervised at all. Train an inverse dynamics model on a smaller, labeled dataset from the same or a similar domain, then apply it to the unlabeled data.

This is the strategy behind OpenAI's Video PreTraining (VPT) for Minecraft. The authors trained an inverse dynamics model on human-played Minecraft with recorded keyboard and mouse actions, applied it to a much larger collection of unlabeled Minecraft videos, and trained a behavioral prior on the resulting pseudo-labels. The IDM need not be perfect for this pipeline to help, but its errors matter: systematic bias, class imbalance, or domain shift can be inherited by the downstream policy.

The same trick works for robot data. If you have a few hours of robot video with joint commands, train an IDM, apply it to thousands of hours of robot video without commands, and now you have a huge dataset of pseudo-labeled actions. This tends to work best when the unlabeled domain is close to the labeled domain in dynamics, sensor characteristics, and visuals.

The weakness of the shortcut is obvious: it captures only the action structure present in the labeled data. If the unlabeled videos feature actions that never appeared in the small labeled set, the IDM may predict something plausible but wrong. If the labeled data has too little action variety, the IDM may collapse toward common actions and produce biased pseudo-labels. This is a distribution-shift problem: the labeled action-transition distribution does not cover the unlabeled transition distribution.

A practical mitigation is to add a small amount of on-policy data collection during pseudo-labeling, so the IDM gets queries on states and transitions it did not see in the labeled set, and can be corrected with a small amount of fresh human or scripted labels. This is a form of active learning over the input space of the IDM.

The fully unsupervised alternative is to learn a latent action model. The idea is to posit a deterministic decoder $f$ such that the transition at time $t$ can be explained as $s_{t+1} = f(s_t, z_t)$, where $s_t$ is the current state and $z_t$ is a low-dimensional latent action. Because $z_t$ is not observed, we also introduce an encoder $\phi(z_t \mid s_t, s_{t+1})$ that infers it from the observed transition. Then $f$ and $\phi$ are trained jointly by minimizing the next-state reconstruction error, averaged over the empirical data distribution.

If you allow $f$ to be arbitrarily expressive and $z_t$ to be arbitrary, the problem is underdetermined: any invertible reparametrization of a valid latent is equally good, and the encoder may also pass target information that has nothing to do with action. A low-dimensional $z_t$, structured decoder, prior, or other regularizer can narrow the solutions, but reconstruction loss alone does not identify physical actions.

It is helpful to think of this as a factor analysis problem with a temporal twist. Classical factor analysis says that observations can be explained as a linear map from a low-dimensional latent plus noise. Here the observation is a transition, and the latent is the action, but the identifiability issues are the same. Without constraints on the factor loading matrix, the latent factors are only identified up to rotation, which is exactly the same gauge freedom we will revisit when discussing alignment.

Common regularizers fall into a few families.

• Prior-matching: penalize divergence between the encoder distribution $q(z_t \mid s_t, s_{t+1})$ and a prior, usually standard normal or a mixture. This is the KL term $\mathrm{KL}(q(z_t \mid s_t, s_{t+1}) \| p(z_t))$ in a VAE-style objective. It limits information capacity and gives a downstream policy a sampling distribution, but does not by itself prevent target leakage or guarantee action semantics.
• Discrete codebooks: quantize $z_t$ to one of $K$ learnable vectors, as in VQ-VAE. This imposes a bounded discrete interface, though the codes are not inherently interpretable or aligned with real controls.
• Contrastive or diversity terms: encourage different transitions to map to different latents and similar transitions to map to similar latents. These terms fight the collapse mode directly by making the encoder's outputs distinguishable.
• Predictive consistency: require the transition model to predict the next state using $z_t$. This makes the latent useful for reconstruction, but additional constraints are needed to distinguish action effects from other transition information.
• Temporal smoothness: prefer latent actions that change slowly across a short window, mirroring how real control signals behave. The tradeoff is that smoothness suppresses rapid changes that are also real, so the regularizer strength has to be tuned.

Genie, a DeepMind model for interactive game video, learns a discrete latent action space. LAPA uses a VQ-VAE-style action quantizer to learn discrete latent actions for vision-language-action pretraining. In both cases the latent is trained to help explain or predict transitions; whether its codes acquire useful control semantics is an empirical question.

A practical choice with strong downstream consequences is whether $z_t$ is continuous or discrete.

Continuous latent actions are convenient for differentiable control and can be compared with real actions using probes when the relationship is sufficiently simple. They also fit naturally into MPC and continuous policy networks. Their weakness is that the encoder can output a smooth continuum that is only locally useful, leaving downstream planners to discover which directions in $z$-space affect the modeled transition. Continuous latents can also be poorly conditioned, with very different sensitivities along different axes.

Discrete latent actions are natural for interfaces, games, and any domain with a small set of distinct moves. They also make downstream learning easier because the action space is bounded. Their weakness is commitment: if the underlying action space is continuous, a discrete codebook will bucket distinct actions together, and a deterministic mean decoder may average across them. Discrete latents can also suffer from codebook collapse, where only a handful of codes are used and the rest are dead, which effectively shrinks the action space below the intended size.

Structured latent actions, where $z_t$ decomposes into named dimensions or hierarchical levels, are an active research area. An object-centric world model might learn one latent action per object, so that $z_t = (z_t^{(1)}, \dots, z_t^{(K)})$ and each $z_t^{(k)}$ controls the $k$-th slot. This encourages a factorized interface but does not guarantee interpretability and requires the right architecture; we touched on the relevant machinery in Part III: Representing Agents and Worlds.

The choice between discrete and continuous is not absolute. Some systems use continuous latents during training for smooth gradients and then quantize at deployment. Others use a continuous backbone with a discretized head. The right answer depends on whether the downstream consumer is a differentiable optimizer, a discrete-action RL agent, or a human reading the codes.

A few failure modes recur in latent action learning.

• Latent collapse. The encoder outputs a constant, and the decoder learns to predict $s_{t+1}$ from $s_t$ alone. This is perfect on the training loss but useless for control. Diversity regularizers exist to prevent it.
• Latent overfitting. The encoder outputs something close to $s_{t+1}$, letting the decoder nearly copy the target. This wastes the latent's low-dimensionality and makes downstream alignment impossible. The problem is that a non-causal latent can pass reconstruction loss by information leakage.
• Noncausal latents. The encoder picks up on incidental features, like motion blur or camera jitter, that correlate with transitions but are not actions. Latents then become unreliable under distribution shift.
• Temporal misalignment. If the video framerate does not match the true action rate, multiple frames correspond to one action or one frame to several actions. The learned latent action then has no consistent relationship to real control.
• Domain dependence. Latent actions learned on one environment need not transfer to another; success requires shared structure, paired anchors, or an explicit adaptation mechanism.

None of these are fatal. Most working systems combine several of the tricks above plus a small amount of labeled data for calibration. The point is that "learning actions from unlabeled video" is not a solved generic problem. It is a design space with many local optima, and choosing the right one depends on how much labeled data, structure, and prior you are willing to supply.

A useful mental model is blind system identification with weak priors. Classical identification observes inputs and outputs; latent-action learning withholds the inputs and asks whether transition-explaining variables can be recovered. Identifiability then depends on the model class and assumptions about the inputs, disturbances, observations, and dynamics. Conditions such as non-Gaussian independent sources can help in particular formulations, but there is no single dimension-count rule for the general nonlinear problem.