Actions, Inverse Dynamics, and Latent Actions › Inverse Dynamics and Controllability

If the forward model is $p(s_{t+1} \mid s_t, a_t)$, the inverse model runs the other direction.

Forward models say what an action does. Inverse models say what action was done. The two are mirror images, but they behave very differently under noise, ambiguity, and distribution shift. In classical control, the forward model is the plant, and the inverse model is what a tracking controller implicitly inverts to compute a command that achieves a reference. The key difference is that in classical control, the inverse is usually solved numerically over a short horizon with a known plant, whereas in modern latent world models, the inverse has to be learned from data and may never be exactly solvable.

Inverse dynamics shows up in several distinct roles in world-model and robotics pipelines.

• Goal-conditioned low-level control: given a current observation and a desired next observation, an inverse model can propose an action that connects them. It becomes a policy only when another component supplies feasible targets; ordinary behavioral cloning instead predicts actions from the observations available at deployment.
• Auxiliary representation learning: add an inverse-dynamics head to a representation network as an auxiliary task. Pathak et al.'s curiosity model used such a head to learn features for forward prediction. Predicting actions encourages retention of action-relevant information, although high inverse accuracy is neither necessary nor sufficient for a generally useful representation.
• Pseudo-labeling: when you have a small labeled dataset with actions and a large unlabeled dataset without, train the inverse model on the small set and use it to label the large one. Video PreTraining (VPT) for Minecraft uses this pattern.
• Hierarchical control: in a hierarchical agent, the high-level policy picks a subgoal (a desired next state), and the low-level controller queries the inverse model for the action that gets you there. This gives a clean interface between levels, and it matches how classical hierarchical planners decompose long-horizon problems.
• Diagnosing action information: poor prediction from $(s_t, s_{t+1})$ can mean that the representation lost the action effect, but it can also reflect non-injective dynamics, disturbances, or noisy labels. Inverse accuracy therefore diagnoses one-step action identifiability, not controllability by itself.

Inverse models are also a natural way to reconstruct an action trace from a video for downstream purposes. If a robot demonstrates something on camera and you have a competent inverse model for that domain, you have recovered a labeled trajectory from a bare video. This is a specific instance of a broader idea: turning observation-only data into action-labeled data by supplying an inductive bias about how actions leave traces on the state.

Inverse models are harder to learn than forward models because the action is often not uniquely determined by the transition.

Imagine a robotic finger touching a table. The finger moves down by 2 cm. Was that an action pushing down, or the finger being pushed down by an external force? The same state transition can be explained by multiple actions. If the model cannot distinguish environmental causes from agent causes, the inverse problem has multiple valid answers. A deterministic model trained with squared error tends toward their conditional mean; a probabilistic inverse model can instead represent multiple modes. In classical system identification, correlation between a regressor and an unobserved disturbance is a confounding or endogeneity problem.

This ambiguity has formal structure. If the forward map $a_t \mapsto s_{t+1}$ is not injective at a given $s_t$, then $I(a_t \mid s_t, s_{t+1})$ is not a point mass. The problem is identifying a distribution over actions. In a linear time-invariant system

where:

• $s_t \in \mathbb{R}^n$: the state at time $t$
• $s_{t+1} \in \mathbb{R}^n$: the next state
• $a_t \in \mathbb{R}^m$: the action at time $t$
• $A \in \mathbb{R}^{n \times n}$: the state transition matrix
• $B \in \mathbb{R}^{n \times m}$: the action matrix, mapping actions into state changes
• $A s_t + B a_t$: the deterministic next state under linear dynamics

in the deterministic, disturbance-free case with known $A$, a feasible transition identifies a unique action precisely when $B$ has full column rank. Then $a_t = B^{+} (s_{t+1} - A s_t)$, where $B^{+} = (B^\top B)^{-1} B^\top$ is the left inverse (Moore-Penrose pseudoinverse) of $B$. Here $B$ must have full column rank ($\mathrm{rank}(B) = m$) for $(B^\top B)$ to be invertible. If $B$ has more columns than rows, or is rank-deficient, multiple actions yield the same next state. The same linear-algebraic ambiguity appears when a redundant mechanism has more control coordinates than task-space constraints, although inverse kinematics and torque control solve different physical problems.

The controllability rank condition has a geometric reading. The next figure draws the action direction and its propagated version for the linear system, showing that the two directions span the two-dimensional state space in this example.

The dimension count clarifies the situation. If $m < n$, the action space is smaller than the state space. Given full column rank, the action map is injective, but a residual $s_{t+1}-As_t$ is exactly reachable only when it lies in the column space of $B$. If $m > n$, the inverse is underdetermined: even with full row rank, there is a whole affine subspace of actions that produce the same transition, and the pseudoinverse picks the minimum-norm solution. If $\mathrm{rank}(B) < \min(m, n)$, some action directions are redundant and the reachable residual subspace is smaller still.

There is a second kind of ambiguity that is subtler. The environment may inject perturbations that look exactly like actions. A robot on ice may velocity-match its commanded joint positions even when it is slipping. The inverse model has no way to know this without more context, and it will produce a confident but wrong estimate of the commanded action. This is why inverse models are usually paired with a dynamics model that can account for external effects, and why injecting a hidden state into the inverse model often causes more harm than good unless it is well-regularized. A hidden state can absorb the perturbation, which sounds desirable, but it can also absorb the action itself, and then the inverse head learns nothing useful.

Controllability is the structural question of which states the agent can reach using actions. For a finite-dimensional linear system with unconstrained inputs, it is a property of $(A,B)$ rather than of a particular policy: any state can be driven to any other state in finite time. With bounded inputs, obstacles, or nonlinear dynamics, reachability depends on the horizon, admissible controls, and region of state space.

For linear time-invariant systems, controllability has a clean rank condition. Form the controllability matrix

where:

• $\mathcal{C}$: the controllability matrix, formed by stacking the action directions after successive applications of the dynamics
• $A \in \mathbb{R}^{n \times n}$: the state transition matrix, and $n$ its (state) dimension
• $B \in \mathbb{R}^{n \times m}$: the action matrix from the linear dynamics, and $m$ the action dimension
• $A^k B$: the direction reached by taking an action and letting the dynamics propagate it for $k$ steps, for $k = 0, 1, \dots, n-1$

The system $(A, B)$ is controllable if and only if $\mathcal{C}$ has rank $n$. The intuition is that $B$ tells you the "instantaneous direction" the action pushes you in, $AB$ tells you where that push gets you a step later, $A^2 B$ two steps, and so on. If you can reach enough distinct directions through the physics iterating on the action, you can cover the whole space. Stopping at $A^{n-1}$ is enough because of the Cayley-Hamilton theorem: any longer product $A^k B$ for $k \ge n$ is already a linear combination of the earlier ones.

A closely related idea is the Kalman decomposition: any linear system can be partitioned into a controllable subspace and an uncontrollable complement. The uncontrollable component cannot be changed by the input. It cannot be chosen as a reachable target, but it may still matter for reward, constraints, safety, or prediction. A planner should therefore distinguish what must be predicted from what can be steered.

In learned representations, controllability is not guaranteed by architecture. A representation can collapse onto a subspace that loses action-relevant information, or it can mix controllable and uncontrollable features into one latent dimension. When this happens, planners struggle because they cannot steer the latent state where they want. Diagnosing this in neural models is harder than in linear systems, but the basic intuition carries over: some directions in latent space respond to the action and some do not, and the ones that respond are the ones a planner can use.

For learned world models, a practical question is whether the latent state responds predictably to actions in a way that supports planning.

You can operationalize one part of this with an inverse model over the latent space, $I(a_t \mid h_t, h_{t+1})$. Near-perfect held-out prediction means the latent retains enough information to identify the recorded one-step action under that data distribution. It does not establish multi-step reachability or controllability. Performance no better than the action prior is evidence that the latent transition lacks usable action information, subject to the ambiguity and noise caveats above.

A second diagnostic is the local action-sensitivity span: the Jacobian $\partial h_{t+1} / \partial a_t$. If this Jacobian is rank-deficient at typical states, fewer independent action directions have a first-order effect there than the raw action space suggests. Its singular values reveal locally weak directions, but this one-step differential test is not a nonlinear controllability certificate.

A stronger empirical test is to run a small planner in latent space and execute its action sequences in held-out rollouts. Success requires both a useful coordinate system and accurate multi-step dynamics. Failure does not isolate the cause, but it directly exposes whether the model supports the planning task.

A representation useful for control preserves predictable action effects and supports reaching task-relevant latent states over the planning horizon. Local response need not be proportional in a nonlinear system, and large one-step sensitivity alone does not guarantee reachability.