Actions, Inverse Dynamics, and Latent Actions

Every world model has to answer a deceptively simple question: given what the world looks like right now, and something you do next, what happens after? The "something you do" is an action. It is the torque you apply to a joint, the key you press, the token you emit, or the force you apply to a robot arm. In a well-defined simulated environment like a MuJoCo hopper or a game of Doom, that knob is labeled: a three-dimensional torque vector, or one of a dozen key combinations. Learning $p(s_{t+1} \mid s_t, a_t)$ in those settings is a supervised regression problem with clean labels. If you want a bit more intuition, imagine the model as a function that has two knobs to turn: one is fixed by what the world is doing, and the other is fixed by what you are doing. Only the second one is yours to adjust.

Real data is rarely so tidy. Most of the world's video has no action labels. A chef chopping onions, a car merging on the highway, a person opening a door. These are all state transitions, but the underlying actions are invisible. If you want a world model that can be controlled from this kind of data, you first have to infer the missing actions. The labels that would tell you which torque, which key, which token, are simply absent. What remains is a sequence of observations and the transitions they imply.

This chapter is about that gap. It moves in four steps, each with a classical analogue. First, we look at the clean case: what does it mean for a transition to be action-conditioned, and why does the action input deserve a special role compared to other inputs? Second, we look at inverse dynamics, the problem of recovering the action from a state transition, and at controllability, the structural property that determines which transitions are even achievable. Third, we look at how latent actions can be discovered from unlabeled video, where the real actions are never observed. Fourth, we look at how those latent actions can be aligned back to real controls, so a policy learned on top of the world model does something useful when deployed. Each step has a classical analogue in filtering, system identification, planning, and control, and keeping those connections visible helps clarify why the modern formulations look the way they do.

By the end of the chapter you will see that "the action" is not a fixed primitive that a world model inherits from its environment. It is a concept the model has to construct, and the choices you make about how to define it have consequences that ripple through planning, transfer, and controllability. In Part V: World-Model Architectures we saw how different architectures represent states. Here we ask what gets to sit in the action slot of those architectures.

It is worth saying up front that "model quality" is not one thing. A world model can be good at predicting future observations while still failing as a controllable object. It can represent the state well for downstream classification while representing the action poorly for planning. It can produce calibrated uncertainty in one region of state space and miscalibrated certainty in another. And it can be decision-useful for a particular task while being spectacularly wrong about the world in ways the task never probes. As we go through this chapter, try to keep these four notions of quality separate: predictive fidelity, representation quality, uncertainty quality, and decision usefulness. Most of the interesting failures in the rest of the book come from confusing one of these for another.

The core object of this chapter is the action-conditioned transition model. It is the function that says, given a state and an action, how the world evolves.

The role of the action is easy to get wrong. It looks like just another input. Stack it next to the state, feed the concatenation into a network, and out comes a prediction. In an ordinary sensor-fusion sense, that is what it is. But the action is conceptually distinct from the state in a way that matters for what you can do with the model afterward.

Consider two features in the state: the color of a traffic light and the steering angle of your car. Both are numbers. But only one of them responds to your decisions. The traffic light is observed. The steering angle is chosen. A world model that treats these as interchangeable covariates can predict the future, but it cannot plan, because it has no way to ask "what happens if I steer left instead of right?" The action input is the entry point for intervention, not just for correlation.

This resembles the distinction between conditioning and intervention in causal inference. Watching a traffic light turn red is not the same as making it turn red. An action-conditioned model exposes the interface needed to compare hypothetical action sequences, but conditioning alone does not make those comparisons causal. They are valid intervention forecasts only to the extent that the action variable has the intended semantics and the training data and model account for relevant confounding, hidden state, and distribution shift. Models that treat actions as decorative inputs fail even the weaker predictive test.

An action has a few properties that distinguish it from a state feature.

• It indexes a decision. A policy or controller chooses it, although a learner may later observe it in a log. Passive video usually does not expose that decision variable.
• It has an addressable interface. Many engineered systems use a lower-dimensional action than state, such as joint commands or game controls, but structured actions can also be large. What matters for planning is that the interface can be enumerated, sampled, or optimized.
• It is applied at a specific time. Actions carry a time index and have a rate. The rate may not match the observation rate, and it may not even be constant across time.
• It is assigned to a transition. In an idealized Markov model, $a_t$ is the control applied during the transition from $s_t$ to $s_{t+1}$. Real actuators can have delay, smoothing, and persistent effects; the state or action history must represent those effects if the model is to remain Markovian.

Some systems blur these lines. The joint torque commanded by a controller is an action; the resulting joint velocity is a state. But velocity is also what the low-level controller reacts to. Where the boundary lies depends on which loop you are modeling. In a hierarchical world model, high-level actions might be subgoals (target positions), and low-level actions might be motor commands. The same physical robot can therefore have several valid action interfaces, and which one you pick determines what kind of planning and learning you can do above it.

This layering mirrors how many real controllers are organized. A high-level planner chooses a waypoint, a mid-level controller produces a reference trajectory, and a low-level servo loop produces torque. Each level sees the level below it as "the action." The same world model could be trained at any of these levels, and the latent action space it discovers will inherit the semantics of whichever level the data was logged at. A model trained on high-rate torque commands has a different action interface from one trained on lower-rate waypoint commands. Its dimension may be larger or smaller depending on the controller, but the semantics and temporal resolution are not interchangeable.

The set of possible actions, denoted $\mathcal{A}$, comes in several flavors, and each one has consequences for how you build the transition model.

• Discrete: $a_t \in \{1, \dots, K\}$, where $K$ is the number of discrete actions. Typical for Atari and Minecraft. Represented as a one-hot vector or an index. A categorical policy can treat action selection as classification, while the transition model conditions on the selected category.
• Continuous: $a_t \in \mathbb{R}^m$, where $m$ is the action dimension. Typical for robotic torque or velocity control. Represented directly as real numbers. A differentiable model can provide action gradients where it is locally smooth, but continuous spaces remain difficult to cover uniformly.
• Mixed: some dimensions discrete, some continuous. Common in real interfaces like keyboard-plus-mouse, where the keyboard is discrete and mouse motion is continuous.
• Structured: sequences, images, or graphs. Auto-regressive interfaces and API calls live here. Here the "action" is a program or a message, and the transition model has to interpret it.
• Action chunks: sometimes the natural unit is a short sequence $a_{t:t+H}$ rather than a single step, where $H$ is the chunk horizon. Action Chunking with Transformers predicts such sequences. Chunking reduces the effective decision rate and can represent temporally coordinated behavior, although the best chunk length depends on the data and controller.

The representation matters because scaling and geometry affect optimization. Continuous actions are often rescaled to a fixed range. Discrete actions are one-hot or embedded. Continuous action sequences are sometimes quantized into discrete tokens so the same transformer backbone used for language or images can process them, an idea we will revisit in Part IX: Foundation and World-Action Models.

In practice, most latent world models condition on actions in one of a few standard ways.

• Concatenation: the action vector is concatenated with the latent state and passed through the transition network. The network then has to learn how to use the action, which is easier when actions explain a large share of the target variation.
• FiLM / conditioning: the action modulates intermediate activations through learned scale and shift parameters. This feature-wise affine transformation can gate, amplify, suppress, or shift intermediate features.
• Cross-attention: the action is treated as a token and the transition attends to it. This shows up in transformer world models. Attention lets the model decide, per state, how much to listen to the action, which is helpful when the action's relevance varies with context.
• Action as an event: instead of conditioning, the action drives a step function on the state, useful for switch-like effects such as discrete modes.

The choice rarely changes what the model can represent, but it changes what it tends to learn, and it changes training stability. Conditioning too weakly, for example by treating the action as one among many inputs, often leads to a model that ignores actions entirely because ignoring them is a good local optimum when most of the variance in the next state comes from the state itself. This local optimum is easy to fall into, because the residual action-driven variance is often a small fraction of the total next-state variance.

This failure mode is worth watching for. A quick test is to shuffle actions across time steps and see whether predictions degrade. If they do not, the model has no measurable predictive dependence on the recorded actions in that evaluation distribution. A second test is to feed alternative actions at the same state and check whether the predicted futures diverge. Divergence shows sensitivity, not causal correctness; agreement with held-out interventions is the stronger test. A third test is to compute the gradient of the predicted next state with respect to the action; near-zero gradients diagnose local insensitivity in differentiable models.

The following diagnostic turns the shuffle test into a visual comparison. It simulates a small linear system, computes next-state prediction error under three conditions, and shows that breaking the action-transition correspondence removes most of the action's predictive benefit.

If the forward model is $p(s_{t+1} \mid s_t, a_t)$, the inverse model runs the other direction.

Forward models say what an action does. Inverse models say what action was done. The two are mirror images, but they behave very differently under noise, ambiguity, and distribution shift. In classical control, the forward model is the plant, and the inverse model is what a tracking controller implicitly inverts to compute a command that achieves a reference. The key difference is that in classical control, the inverse is usually solved numerically over a short horizon with a known plant, whereas in modern latent world models, the inverse has to be learned from data and may never be exactly solvable.

Inverse dynamics shows up in several distinct roles in world-model and robotics pipelines.

• Goal-conditioned low-level control: given a current observation and a desired next observation, an inverse model can propose an action that connects them. It becomes a policy only when another component supplies feasible targets; ordinary behavioral cloning instead predicts actions from the observations available at deployment.
• Auxiliary representation learning: add an inverse-dynamics head to a representation network as an auxiliary task. Pathak et al.'s curiosity model used such a head to learn features for forward prediction. Predicting actions encourages retention of action-relevant information, although high inverse accuracy is neither necessary nor sufficient for a generally useful representation.
• Pseudo-labeling: when you have a small labeled dataset with actions and a large unlabeled dataset without, train the inverse model on the small set and use it to label the large one. Video PreTraining (VPT) for Minecraft uses this pattern.
• Hierarchical control: in a hierarchical agent, the high-level policy picks a subgoal (a desired next state), and the low-level controller queries the inverse model for the action that gets you there. This gives a clean interface between levels, and it matches how classical hierarchical planners decompose long-horizon problems.
• Diagnosing action information: poor prediction from $(s_t, s_{t+1})$ can mean that the representation lost the action effect, but it can also reflect non-injective dynamics, disturbances, or noisy labels. Inverse accuracy therefore diagnoses one-step action identifiability, not controllability by itself.

Inverse models are also a natural way to reconstruct an action trace from a video for downstream purposes. If a robot demonstrates something on camera and you have a competent inverse model for that domain, you have recovered a labeled trajectory from a bare video. This is a specific instance of a broader idea: turning observation-only data into action-labeled data by supplying an inductive bias about how actions leave traces on the state.

Inverse models are harder to learn than forward models because the action is often not uniquely determined by the transition.

Imagine a robotic finger touching a table. The finger moves down by 2 cm. Was that an action pushing down, or the finger being pushed down by an external force? The same state transition can be explained by multiple actions. If the model cannot distinguish environmental causes from agent causes, the inverse problem has multiple valid answers. A deterministic model trained with squared error tends toward their conditional mean; a probabilistic inverse model can instead represent multiple modes. In classical system identification, correlation between a regressor and an unobserved disturbance is a confounding or endogeneity problem.

This ambiguity has formal structure. If the forward map $a_t \mapsto s_{t+1}$ is not injective at a given $s_t$, then $I(a_t \mid s_t, s_{t+1})$ is not a point mass. The problem is identifying a distribution over actions. In a linear time-invariant system

where:

• $s_t \in \mathbb{R}^n$: the state at time $t$
• $s_{t+1} \in \mathbb{R}^n$: the next state
• $a_t \in \mathbb{R}^m$: the action at time $t$
• $A \in \mathbb{R}^{n \times n}$: the state transition matrix
• $B \in \mathbb{R}^{n \times m}$: the action matrix, mapping actions into state changes
• $A s_t + B a_t$: the deterministic next state under linear dynamics

in the deterministic, disturbance-free case with known $A$, a feasible transition identifies a unique action precisely when $B$ has full column rank. Then $a_t = B^{+} (s_{t+1} - A s_t)$, where $B^{+} = (B^\top B)^{-1} B^\top$ is the left inverse (Moore-Penrose pseudoinverse) of $B$. Here $B$ must have full column rank ($\mathrm{rank}(B) = m$) for $(B^\top B)$ to be invertible. If $B$ has more columns than rows, or is rank-deficient, multiple actions yield the same next state. The same linear-algebraic ambiguity appears when a redundant mechanism has more control coordinates than task-space constraints, although inverse kinematics and torque control solve different physical problems.

The controllability rank condition has a geometric reading. The next figure draws the action direction and its propagated version for the linear system, showing that the two directions span the two-dimensional state space in this example.

The dimension count clarifies the situation. If $m < n$, the action space is smaller than the state space. Given full column rank, the action map is injective, but a residual $s_{t+1}-As_t$ is exactly reachable only when it lies in the column space of $B$. If $m > n$, the inverse is underdetermined: even with full row rank, there is a whole affine subspace of actions that produce the same transition, and the pseudoinverse picks the minimum-norm solution. If $\mathrm{rank}(B) < \min(m, n)$, some action directions are redundant and the reachable residual subspace is smaller still.

There is a second kind of ambiguity that is subtler. The environment may inject perturbations that look exactly like actions. A robot on ice may velocity-match its commanded joint positions even when it is slipping. The inverse model has no way to know this without more context, and it will produce a confident but wrong estimate of the commanded action. This is why inverse models are usually paired with a dynamics model that can account for external effects, and why injecting a hidden state into the inverse model often causes more harm than good unless it is well-regularized. A hidden state can absorb the perturbation, which sounds desirable, but it can also absorb the action itself, and then the inverse head learns nothing useful.

Controllability is the structural question of which states the agent can reach using actions. For a finite-dimensional linear system with unconstrained inputs, it is a property of $(A,B)$ rather than of a particular policy: any state can be driven to any other state in finite time. With bounded inputs, obstacles, or nonlinear dynamics, reachability depends on the horizon, admissible controls, and region of state space.

For linear time-invariant systems, controllability has a clean rank condition. Form the controllability matrix

where:

• $\mathcal{C}$: the controllability matrix, formed by stacking the action directions after successive applications of the dynamics
• $A \in \mathbb{R}^{n \times n}$: the state transition matrix, and $n$ its (state) dimension
• $B \in \mathbb{R}^{n \times m}$: the action matrix from the linear dynamics, and $m$ the action dimension
• $A^k B$: the direction reached by taking an action and letting the dynamics propagate it for $k$ steps, for $k = 0, 1, \dots, n-1$

The system $(A, B)$ is controllable if and only if $\mathcal{C}$ has rank $n$. The intuition is that $B$ tells you the "instantaneous direction" the action pushes you in, $AB$ tells you where that push gets you a step later, $A^2 B$ two steps, and so on. If you can reach enough distinct directions through the physics iterating on the action, you can cover the whole space. Stopping at $A^{n-1}$ is enough because of the Cayley-Hamilton theorem: any longer product $A^k B$ for $k \ge n$ is already a linear combination of the earlier ones.

A closely related idea is the Kalman decomposition: any linear system can be partitioned into a controllable subspace and an uncontrollable complement. The uncontrollable component cannot be changed by the input. It cannot be chosen as a reachable target, but it may still matter for reward, constraints, safety, or prediction. A planner should therefore distinguish what must be predicted from what can be steered.

In learned representations, controllability is not guaranteed by architecture. A representation can collapse onto a subspace that loses action-relevant information, or it can mix controllable and uncontrollable features into one latent dimension. When this happens, planners struggle because they cannot steer the latent state where they want. Diagnosing this in neural models is harder than in linear systems, but the basic intuition carries over: some directions in latent space respond to the action and some do not, and the ones that respond are the ones a planner can use.

For learned world models, a practical question is whether the latent state responds predictably to actions in a way that supports planning.

You can operationalize one part of this with an inverse model over the latent space, $I(a_t \mid h_t, h_{t+1})$. Near-perfect held-out prediction means the latent retains enough information to identify the recorded one-step action under that data distribution. It does not establish multi-step reachability or controllability. Performance no better than the action prior is evidence that the latent transition lacks usable action information, subject to the ambiguity and noise caveats above.

A second diagnostic is the local action-sensitivity span: the Jacobian $\partial h_{t+1} / \partial a_t$. If this Jacobian is rank-deficient at typical states, fewer independent action directions have a first-order effect there than the raw action space suggests. Its singular values reveal locally weak directions, but this one-step differential test is not a nonlinear controllability certificate.

A stronger empirical test is to run a small planner in latent space and execute its action sequences in held-out rollouts. Success requires both a useful coordinate system and accurate multi-step dynamics. Failure does not isolate the cause, but it directly exposes whether the model supports the planning task.

A representation useful for control preserves predictable action effects and supports reaching task-relevant latent states over the planning horizon. Local response need not be proportional in a nonlinear system, and large one-step sensitivity alone does not guarantee reachability.

Now we hit the hard part. Video is everywhere, but action labels are rare. Even in robotics, a robot that has been running for years in a lab might have thousands of hours of video and only a few hours of logged commands. And most of the world's video was never on a robot at all.

The goal here is to discover the action space from unlabeled sequences of observations $o_1, o_2, \dots, o_T$, where $o_t$ is the observation at time $t$ and the underlying action $a_t$ is hidden. We will refer to a learned substitute for $a_t$ as a latent action, typically denoted $z_t$. The latent action is our best guess at what the action was, inferred from the transition alone. The hope is that the latent action is more than a bookkeeping trick: that it behaves like a real action when used for planning and control.

A few distinct practical reasons cause actions to be missing.

• Sensor mismatch. A camera records everything, but nobody instrumented the car's steering wheel. The video and the control signal were captured by different systems that were never synchronized.
• Privacy or commercial sensitivity. Action logs may be hidden even when the video is public. This is common when the video is released for research but the underlying telemetry is proprietary.
• Cross-embodiment transfer. A human hand and a robot gripper have different native actions, but you would like a shared abstraction. Here the labels are not missing so much as incommensurable across embodiments.
• Video was the point. Games played on Twitch or cooking demos on YouTube exist for entertainment, not for training. Actions were never captured because capturing them was never the goal.
• Task-shift. A dataset may be labeled with high-level intent but not low-level controls. "Make dinner" is a label, but it does not tell a motor-level model what to do.

In all of these cases you still have transitions. You just do not have the labels that explain them. The question becomes: what can be recovered about the actions from the transitions alone?

The simplest practical approach is not really unsupervised at all. Train an inverse dynamics model on a smaller, labeled dataset from the same or a similar domain, then apply it to the unlabeled data.

This is the strategy behind OpenAI's Video PreTraining (VPT) for Minecraft. The authors trained an inverse dynamics model on human-played Minecraft with recorded keyboard and mouse actions, applied it to a much larger collection of unlabeled Minecraft videos, and trained a behavioral prior on the resulting pseudo-labels. The IDM need not be perfect for this pipeline to help, but its errors matter: systematic bias, class imbalance, or domain shift can be inherited by the downstream policy.

The same trick works for robot data. If you have a few hours of robot video with joint commands, train an IDM, apply it to thousands of hours of robot video without commands, and now you have a huge dataset of pseudo-labeled actions. This tends to work best when the unlabeled domain is close to the labeled domain in dynamics, sensor characteristics, and visuals.

The weakness of the shortcut is obvious: it captures only the action structure present in the labeled data. If the unlabeled videos feature actions that never appeared in the small labeled set, the IDM may predict something plausible but wrong. If the labeled data has too little action variety, the IDM may collapse toward common actions and produce biased pseudo-labels. This is a distribution-shift problem: the labeled action-transition distribution does not cover the unlabeled transition distribution.

A practical mitigation is to add a small amount of on-policy data collection during pseudo-labeling, so the IDM gets queries on states and transitions it did not see in the labeled set, and can be corrected with a small amount of fresh human or scripted labels. This is a form of active learning over the input space of the IDM.

The fully unsupervised alternative is to learn a latent action model. The idea is to posit a deterministic decoder $f$ such that the transition at time $t$ can be explained as $s_{t+1} = f(s_t, z_t)$, where $s_t$ is the current state and $z_t$ is a low-dimensional latent action. Because $z_t$ is not observed, we also introduce an encoder $\phi(z_t \mid s_t, s_{t+1})$ that infers it from the observed transition. Then $f$ and $\phi$ are trained jointly by minimizing the next-state reconstruction error, averaged over the empirical data distribution.

If you allow $f$ to be arbitrarily expressive and $z_t$ to be arbitrary, the problem is underdetermined: any invertible reparametrization of a valid latent is equally good, and the encoder may also pass target information that has nothing to do with action. A low-dimensional $z_t$, structured decoder, prior, or other regularizer can narrow the solutions, but reconstruction loss alone does not identify physical actions.

It is helpful to think of this as a factor analysis problem with a temporal twist. Classical factor analysis says that observations can be explained as a linear map from a low-dimensional latent plus noise. Here the observation is a transition, and the latent is the action, but the identifiability issues are the same. Without constraints on the factor loading matrix, the latent factors are only identified up to rotation, which is exactly the same gauge freedom we will revisit when discussing alignment.

Common regularizers fall into a few families.

• Prior-matching: penalize divergence between the encoder distribution $q(z_t \mid s_t, s_{t+1})$ and a prior, usually standard normal or a mixture. This is the KL term $\mathrm{KL}(q(z_t \mid s_t, s_{t+1}) \| p(z_t))$ in a VAE-style objective. It limits information capacity and gives a downstream policy a sampling distribution, but does not by itself prevent target leakage or guarantee action semantics.
• Discrete codebooks: quantize $z_t$ to one of $K$ learnable vectors, as in VQ-VAE. This imposes a bounded discrete interface, though the codes are not inherently interpretable or aligned with real controls.
• Contrastive or diversity terms: encourage different transitions to map to different latents and similar transitions to map to similar latents. These terms fight the collapse mode directly by making the encoder's outputs distinguishable.
• Predictive consistency: require the transition model to predict the next state using $z_t$. This makes the latent useful for reconstruction, but additional constraints are needed to distinguish action effects from other transition information.
• Temporal smoothness: prefer latent actions that change slowly across a short window, mirroring how real control signals behave. The tradeoff is that smoothness suppresses rapid changes that are also real, so the regularizer strength has to be tuned.

Genie, a DeepMind model for interactive game video, learns a discrete latent action space. LAPA uses a VQ-VAE-style action quantizer to learn discrete latent actions for vision-language-action pretraining. In both cases the latent is trained to help explain or predict transitions; whether its codes acquire useful control semantics is an empirical question.

A practical choice with strong downstream consequences is whether $z_t$ is continuous or discrete.

Continuous latent actions are convenient for differentiable control and can be compared with real actions using probes when the relationship is sufficiently simple. They also fit naturally into MPC and continuous policy networks. Their weakness is that the encoder can output a smooth continuum that is only locally useful, leaving downstream planners to discover which directions in $z$-space affect the modeled transition. Continuous latents can also be poorly conditioned, with very different sensitivities along different axes.

Discrete latent actions are natural for interfaces, games, and any domain with a small set of distinct moves. They also make downstream learning easier because the action space is bounded. Their weakness is commitment: if the underlying action space is continuous, a discrete codebook will bucket distinct actions together, and a deterministic mean decoder may average across them. Discrete latents can also suffer from codebook collapse, where only a handful of codes are used and the rest are dead, which effectively shrinks the action space below the intended size.

Structured latent actions, where $z_t$ decomposes into named dimensions or hierarchical levels, are an active research area. An object-centric world model might learn one latent action per object, so that $z_t = (z_t^{(1)}, \dots, z_t^{(K)})$ and each $z_t^{(k)}$ controls the $k$-th slot. This encourages a factorized interface but does not guarantee interpretability and requires the right architecture; we touched on the relevant machinery in Part III: Representing Agents and Worlds.

The choice between discrete and continuous is not absolute. Some systems use continuous latents during training for smooth gradients and then quantize at deployment. Others use a continuous backbone with a discretized head. The right answer depends on whether the downstream consumer is a differentiable optimizer, a discrete-action RL agent, or a human reading the codes.

A few failure modes recur in latent action learning.

• Latent collapse. The encoder outputs a constant, and the decoder learns to predict $s_{t+1}$ from $s_t$ alone. This is perfect on the training loss but useless for control. Diversity regularizers exist to prevent it.
• Latent overfitting. The encoder outputs something close to $s_{t+1}$, letting the decoder nearly copy the target. This wastes the latent's low-dimensionality and makes downstream alignment impossible. The problem is that a non-causal latent can pass reconstruction loss by information leakage.
• Noncausal latents. The encoder picks up on incidental features, like motion blur or camera jitter, that correlate with transitions but are not actions. Latents then become unreliable under distribution shift.
• Temporal misalignment. If the video framerate does not match the true action rate, multiple frames correspond to one action or one frame to several actions. The learned latent action then has no consistent relationship to real control.
• Domain dependence. Latent actions learned on one environment need not transfer to another; success requires shared structure, paired anchors, or an explicit adaptation mechanism.

None of these are fatal. Most working systems combine several of the tricks above plus a small amount of labeled data for calibration. The point is that "learning actions from unlabeled video" is not a solved generic problem. It is a design space with many local optima, and choosing the right one depends on how much labeled data, structure, and prior you are willing to supply.

A useful mental model is blind system identification with weak priors. Classical identification observes inputs and outputs; latent-action learning withholds the inputs and asks whether transition-explaining variables can be recovered. Identifiability then depends on the model class and assumptions about the inputs, disturbances, observations, and dynamics. Conditions such as non-Gaussian independent sources can help in particular formulations, but there is no single dimension-count rule for the general nonlinear problem.

Even a well-trained latent action model has no reason to place its latent actions on the same axis as the real ones. The learned $z_t$ is a knob that happens to be useful for prediction. Your downstream planner, on the other hand, has to issue a real command.

Alignment is the process of connecting a learned latent action to a real action. It sounds like a small engineering step, but it is a fundamental limitation. Unlabeled latent action discovery alone leaves a free reparametrization, and alignment must break that symmetry.

This is worth pausing on. The need for alignment is not a failure of any particular algorithm. It is a mathematical property of the problem. From the perspective of unlabeled data alone, the latent action space is defined only up to a symmetry group, and any element of the orbit is as good as any other. Breaking the symmetry requires information that is not in the unlabeled data, which is why every alignment method must import something external: a small labeled set, an assumption about statistics, a shared structure across views, or a prior on the action space.

Linear systems make the problem crisp. If the true dynamics are

where:

• $s_t$: the true state at time $t$
• $a_t$: the true action at time $t$
• $A$: the true state transition matrix
• $B$: the true action matrix
• $s_{t+1}$: the true next state at time $t+1$

and a latent action model learns $s_{t+1} = A s_t + \tilde{B} z_t$, then any $z_t = M a_t$ works for any invertible matrix $M$ that keeps $\tilde{B} M = B$. In particular, the two models agree when we can write

where:

• $A$: the true state transition matrix
• $B$: the true action matrix
• $s_t$: the true state at time $t$
• $a_t$: the true action at time $t$
• $\tilde{B}$: the action matrix learned by the latent action model
• $z_t$: the learned latent action at time $t$
• $M$: an invertible matrix relating the learned and true action spaces, satisfying $\tilde{B} M = B$

The latent action is therefore identified only up to a linear reparametrization: any invertible $M$ satisfying $\tilde{B} M = B$ yields an equally valid latent action $z_t = M a_t$. On nonlinear systems the gauge freedom is even larger. If the true transition is $s_{t+1} = F(s_t, a_t)$ and the latent model learns $s_{t+1} = \tilde{F}(s_t, z_t)$, then any diffeomorphism $\psi$ of the action space gives another valid model via $\tilde{F}(s_t, \cdot) = F(s_t, \psi^{-1}(\cdot))$, with $z_t = \psi(a_t)$.

where:

• $F$: the true transition function
• $\tilde{F}$: the learned latent transition function
• $\psi$: a diffeomorphism of the action space
• $z_t = \psi(a_t)$: the latent action induced by the reparametrization

The practical consequences are concrete. Two different random seeds training the same architecture on the same data may produce latent actions that are sign-flipped, permuted, or scaled versions of each other. If the model is then used for planning, the policy that was trained on top of one version will fail on the other. If you have two robots, each with a latent action space, they may not be directly comparable even if the physical systems are pixel-identical.

For discrete latent actions, the analogue is a permutation of the codebook: code 7 might mean "jump forward" in one training run and "duck" in another. For continuous latent actions, the ambiguity is both a linear map and a smooth bijection, which is even harder to detect from data alone. Estimating the misalignment from data is possible when enough paired samples exist, but the number of samples grows with the dimension of the action space and with the richness of the diffeomorphism class you are trying to fit.

The terminology "gauge freedom" comes from physics: when two descriptions of the same physical system are related by a transformation that has no measurable consequences, the transformation is called a gauge transformation, and the space of such transformations is the gauge group. In latent action modeling, the gauge group is whatever class of reparametrizations leaves the reconstruction loss invariant. Identifying the gauge group tells you exactly what additional data or assumption is needed to fix a unique latent action space.

The most direct way to break the symmetry is to pay for a little supervision. Suppose you have a small paired dataset with both latent actions $z_t$ and real actions $a_t$. Two standard approaches follow.

• Linear probe. Fit a linear map $W$ such that $a_t \approx W z_t$. This tests whether the observed relationship is close to linear over the sampled operating range. It is cheap to fit, but it can still overfit, so alignment must be measured on held-out transitions and, ideally, under relevant distribution shifts.
• Decoder head. Add a small head $g(z_t) \to a_t$ to the latent action model and train it on the paired data while the rest of the model is frozen or slowly updated. The head absorbs the nonlinear part of the misalignment. Decoder heads are more expressive but need more paired data and can overfit if the paired set is tiny.

Both approaches work well when the paired data has the same distribution as the unlabeled data. If it does not, the alignment map will be biased. A common practical variant is to include a small amount of paired data in the original training objective, so the model learns a latent action space that is already partially aligned. This is sometimes called semi-supervised latent action learning, and it can be more sample-efficient than fitting the probe as a post-hoc step.

How many paired transitions a probe needs depends on latent dimension, conditioning, noise, model class, and distribution coverage. Learning curves on held-out trajectories are more reliable than a fixed sample-count rule. A low-dimensional linear map can require far less data than the world model, but only if the representation already exposes the relevant action structure.

When labeled pairs are scarce, distributional constraints can supplement alignment. Matching marginal statistics of latent and real actions can remove selected ambiguities, but it generally cannot recover a pointwise correspondence: many reparametrizations preserve the same distribution.

A few techniques are relevant.

• Invariance to nuisance factors. Require $z_t$ to be unchanged when the same transition is seen from a different viewpoint, background, or sensor. This forbids the encoder from encoding nuisance variables, which shrinks the gauge group.
• Cross-modal alignment. If the same action was performed, its latent action should match across two different views of the same event. This is a contrastive task with positive pairs defined by actions. Instead of labeling actions directly, you label which transitions should have the same latent action, which is often easier.
• Canonicalization. Normalize the latent action space so that $\mathbb{E}[z_t] = 0$ and $\operatorname{Cov}(z_t)$ is identity. This does not fix the rotation but it removes scale. Canonicalization is a small, cheap operation that eliminates an entire class of gauge freedom.
• Mutual information maximization. Encourage $I(z_t; a_t)$ to be high with respect to whatever partial labels exist, which is a soft version of alignment. The mutual information term is a general formulation that subsumes several specific alignment losses as special cases.

Distributional alignment is weaker than paired supervision in what it guarantees. It can leave rotations, permutations, or nonlinear symmetries unresolved and can match marginals while pairing individual actions incorrectly. Paired anchors or structural assumptions are needed when the downstream system requires a specific real-control mapping.

Discrete latent actions are a special case because alignment becomes a matching problem. If you have $K$ latent codes and $M$ real actions, and some paired data, you can solve an optimal assignment problem: match each latent code to the real action it most often co-occurs with. When $K = M$ and the mapping is bijective, this is a clean linear assignment that can be solved exactly in polynomial time by the Hungarian algorithm.

Discrete alignment is a matching problem, and the next figure shows it as a co-occurrence heatmap. The selected assignment is marked on the matrix, making the mapping from latent codes to real actions explicit.

When $K \neq M$, you have to decide whether to merge or split codes, and this is essentially a form of action abstraction.

An overcomplete latent codebook can represent several modes within one nominal real action, but simply ignoring infrequent codes may discard meaningful behavior. Utilization and transition semantics should be measured before codes are merged or pruned. When $K \neq M$, alignment is no longer a one-to-one assignment and requires an explicit many-to-one, one-to-many, or abstraction policy.

It is tempting to demand perfect alignment between latent and real actions. In practice the bar is set by the downstream task.

• Planning inside a learned simulator. An unaligned latent may suffice to optimize imagined trajectories. If the plan must be executed in the real environment, however, the chosen latent still needs a reliable map to executable controls.
• Behavior cloning. A policy can imitate latent labels without naming their semantics, but deployment still requires either a latent-to-real action decoder or an environment that accepts those same latent controls.
• Deployment on hardware. The policy has to output a valid real action. Some alignment strategy is mandatory; whether a linear probe is sufficient must be established on held-out interventions.
• Transfer across robots. Alignment has to produce compatible action spaces, which is much stronger than alignment to a single robot's actions. It requires a shared coordinate system that survives domain shift, in addition to an alignment map for each embodiment.

The required alignment depends on the deployment interface. A planner that only explores imagined rollouts has different needs from a policy that must actuate hardware. Deciding that interface early determines whether pretraining needs paired anchors or can defer the mapping to a downstream decoder.

We can see all of the above on a small linear system that we fully control. In the noiseless equation, the true action is recoverable in closed form, so the supervised inverse model can be checked against ground truth. The learned latent model is deliberately less constrained: its result shows why reconstruction does not by itself reveal the true action coordinate.

The system is a 2D state (position and velocity) driven by a scalar force. Position integrates velocity, and velocity decays by a fixed damping factor $\gamma$ each step:

where:

• $s_t = (p_t, v_t)^\top$: the 2D state at time $t$, with position $p_t$ and velocity $v_t$
• $s_{t+1}$: the next 2D state
• $a_t \in \mathbb{R}$: the scalar force action at time $t$
• $\Delta t$: the simulation timestep, used both for position integration and for scaling the force into a velocity change
• $\gamma = 0.93$: the per-step velocity damping factor, so that $v_{t+1} = \gamma v_t + \Delta t \, a_t$
• $A = \begin{bmatrix} 1 & \Delta t \\ 0 & \gamma \end{bmatrix}$: the state transition matrix, which advances position by velocity and damps velocity
• $B = \begin{bmatrix} 0 \\ \Delta t \end{bmatrix}$: the action matrix, which injects force only into the velocity component

The action enters only through velocity. The vector $B$ has rank one, so the effective action space is one-dimensional. In the stated deterministic dynamics, $B$ has full column rank and the feasible residual lies in its column space, so $a_t = B^{+}(s_{t+1} - A s_t)$ recovers the action exactly. The simulation below adds process noise, so recovery from each realized transition is approximate rather than exact.

We start by importing libraries and generating a synthetic dataset of 400 trajectories, each 50 steps long, with random Gaussian actions. The random actions act as a natural excitation signal. Without them, the state would not explore enough of its reachable set to identify the dynamics, a point that matters in any real data collection problem.

The results illustrate several points at once. A linear inverse dynamics model recovers the true action accurately in this controlled setting. A latent action model trained only on next-state reconstruction reaches low reconstruction error while remaining weakly correlated with the true action. A linear probe measures that weak alignment without retraining the encoder.

The same experiment becomes less identifiable with nonlinear dynamics, unobserved disturbances, or a rank-deficient action map. A second action input with a smaller transition effect may be underrepresented by a reconstruction objective even when it matters for a downstream task.

The key parameters for the worked example are:

• dt (0.1): The simulation timestep. It scales how velocity integrates into position and how force converts into a velocity change.
• gamma (0.93): The per-step velocity damping factor. It controls how quickly velocity decays and therefore how far past actions influence the current state.
• Effective action rank (1): The action matrix has a single independent control column, so its noiseless action map is injective. Adding independent action columns does not create ambiguity unless the resulting action map loses column rank.
• latent_dim (1): The dimensionality of the learned latent action. The bottleneck limits capacity, but even one dimension can encode target information instead of the physical action.
• hidden (64): The width of the encoder and decoder hidden layers.
• learning rate (1e-3): The Adam step size used to train the latent action model.
• epochs (500): The number of full-batch training passes over the transitions.
• alpha (1e-6 for the IDM, 1e-3 for the probe): The Ridge regularization strength. Small values keep the fit close to an exact linear solution.
• train fraction (0.8): The share of complete trajectories assigned to training, with the remainder held out for evaluation.

The techniques in this chapter are what let world models move beyond environments where every action is pre-labeled. They are also where most of the practical difficulty lives, and it is worth being honest about the limitations before we move on.

Inverse dynamics is limited by identifiability. A state transition does not always reveal the underlying action. When the forward map is not injective, or when external perturbations mimic agent effects, a deterministic mean-regression model can average incompatible hypotheses; a probabilistic model can preserve several hypotheses but cannot identify the true one without more information. Pseudo-labels from IDM-based methods can carry these errors into downstream training.

Latent actions are not unique. Different seeds, architectures, or datasets can produce different latent spaces with the same reconstruction loss. Reconstruction-only formulations retain a reparametrization symmetry unless the model class or assumptions break it. Paired supervision is one way to choose real-control coordinates; invariance, canonicalization, and shared structure add alternative assumptions, but none supplies semantics for free.

Latent-action transfer across domains is not guaranteed. It becomes plausible when environments share action semantics, paired anchors, or jointly learned structure. Arbitrary cross-embodiment transfer remains an open problem because visually similar transitions can require different controls and identical controls can have different effects.

Controllability in learned representations is fragile. Even with clean action labels, a latent state can conflate action-relevant and action-irrelevant features. Inverse prediction and Jacobian-rank analysis catch selected one-step failures, while held-out planning rollouts test the combined model more directly. Linear systems have a clean controllability-rank condition; general learned nonlinear representations do not have an equally simple global test.

Passive data biases the model toward observed behavior. Video shows what the recorded agents happened to do, not necessarily the full range of feasible interventions. The learned latent action space is therefore shaped by the behavior policy and coverage of the dataset. The next chapter, Interaction Data and Passive Observation, takes this up as a first-class concern.

These ideas have nevertheless produced concrete systems. VPT used an IDM to pseudo-label online Minecraft video; Genie learned a latent action interface for interactive video generation; and LAPA used latent-action pretraining before adaptation to robot controls. Each result depends on its domain, data, and alignment procedure rather than establishing generic action recovery from arbitrary video.

The clearest way to read the field at this point is to see the action slot as a design decision rather than a given. When you build a world model, you are deciding what the action is and where it comes from. That decision determines whether the model can plan, connect its imagined controls to deployment, and learn useful control variables from unlabeled video.

The action slot of a world model is where control enters. Unlike an ordinary state feature, it is the interface through which a planner selects among modeled futures.

We covered four ideas in this chapter.

• Action-conditioned transitions. The forward model $p(s_{t+1} \mid s_t, a_t)$ is where actions live. They enter via concatenation, FiLM conditioning, cross-attention, or as event inputs, and the choice affects what the model tends to learn. A model that ignores its actions can still predict, but it cannot plan.
• Inverse dynamics and controllability. The inverse model $p(a_t \mid s_t, s_{t+1})$ is a mirror of the forward model. It is used for goal-conditioned control, auxiliary representation learning, pseudo-labeling, and hierarchical control. Controllability is the structural property of the transition pair that says which states are reachable with actions; learned representations require separate one-step diagnostics and rollout tests.
• Learning actions from unlabeled video. When actions are missing, you can train an inverse model on a small labeled set and apply it to the rest, or learn a latent variable directly from transitions. Bottlenecks and regularizers constrain the solutions but do not guarantee physical action semantics.
• Latent action alignment with real controls. Learned latent actions are only defined up to an invertible reparametrization. Aligning them with real actions takes a small amount of supervision, applied as a linear probe, a decoder head, a contrastive objective, or a discrete codebook assignment. The amount of alignment needed depends on what you plan to do downstream.

The next chapter, Interaction Data and Passive Observation, turns to the data side of this story. Where does the transition data come from? What does it look like when it was produced by an agent actively trying to learn versus by a passive observer, and what can and cannot be inferred from each? The methods in this chapter are the tools you use once you know what data you have. The next one is about looking at the data itself.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about actions, inverse dynamics, and latent actions in world models.