Vision-Language-Action and World-Action Models › Joint perception, language, and control

A short robot instruction such as "put the red mug in the sink" leaves the mug's location, the scene geometry, and the motion details unspecified. A controller must recover them from observations, memory, and learned priors. The engineering question is where that information is represented and how the pieces are joined. Designs range from separate modules exchanging symbols to models that fuse vision and language before producing actions. Their interpretability, data requirements, and capability depend on the implementation and task.

One case for fusion is an information-bottleneck argument, related to the sufficient-state discussion in Part III: Representing Agents and Worlds. A symbolic modular pipeline may pass selected labels, poses, and predicates across interfaces while omitting image detail. Other modular systems pass images, maps, or uncertainty estimates, so the bottleneck depends on the interface. A bounding box may be sufficient for one grasp and omit the handle orientation needed for another. Targeted interface tests can expose such omissions before physical failure, though finding every task-relevant detail in advance remains difficult.

If the visual encoder is unfrozen and trained end-to-end with the policy, action loss can update its features. Errors on translucent objects, for example, may encourage features that distinguish useful optical cues, provided the training data expose that failure and the optimization can correct it. This is one meaning of joint training: action prediction can shape perception instead of relying only on an intermediate label objective. Frozen-encoder VLAs do not have this gradient path. Even with end-to-end training, better sample efficiency or worse interpretability relative to a modular baseline is an empirical question, not a consequence of fusion alone.

Language enters for a different reason. Natural language is a flexible, compositional interface for many task specifications. An instruction can name objects, spatial relations, and ordering constraints, and can qualify an action with words such as "carefully." Unlike a flat classifier with one label per task, it can express new combinations without adding a class for each one. Structured task languages can be compositional too, and an unfamiliar phrase still requires grounding and may require new training data. Part IV: Space, Physics, and Causality discusses how to test such combinations. Broad language pretraining may already include phrases such as "red mug," but that exposure does not establish that a robot can execute the corresponding instruction.

The cleanest formalization reuses machinery we have already seen. Take a partially observable Markov decision process with state $s_t \in \mathcal{S}$, camera observation $o_t \in \mathcal{O}$, proprioceptive observation $q_t \in \mathcal{Q}$, action $a_t \in \mathcal{A}$, transition kernel $T(s_{t+1} \mid s_t, a_t)$, and joint observation kernel $O(o_t, q_t \mid s_t)$. Add a fixed instruction $\ell = (w_1, \dots, w_L)$ drawn from a distribution over natural-language strings. A task-specific reward $R_\ell(s_t,a_t)$ or success condition specifies what achieving that instruction means; the behavior-cloning objective below instead learns from demonstrations of that task. The task is to learn a policy

$$\pi_\theta(a_t \mid o_{\leq t}, q_{\leq t}, \ell)$$

where:

• $\pi_\theta$: the policy, parameterized by $\theta$
• $a_t$: the action at time $t$, in the action space $\mathcal{A}$
• $o_{\leq t}$: the camera-observation history up to time $t$, i.e. $o_1, \dots, o_t$, each in $\mathcal{O}$
• $q_{\leq t}$: the proprioceptive-observation history up to time $t$, where $q_t \in \mathcal{Q}$ includes measurements such as joint angles and gripper width
• $\ell$: the fixed natural-language instruction, shared across the episode. The instruction conditions the whole thing but never changes across an episode; it is a parameter of the task, not part of the state. Written as a token sequence $\ell = (w_1, \dots, w_L)$ of $L$ tokens, it is fixed for the whole episode and conditions every action.

Two limitations of the simplified policy setup are worth stating because their effects can be mistaken for model errors.

• Markov assumption. The written form $o_{\leq t}$ admits history, but many VLA implementations condition on a limited set of frames plus the instruction. A policy without another memory input cannot recover a past event absent from that window, as discussed in Part III, Chapter 4: Temporal State and Memory. A short-context policy is not inherently defective; it is suited to tasks where its available observations carry enough information.
• Train–deployment distribution shift. A policy fitted to demonstrations may encounter different tasks, objects, and scenes at deployment. This is distinct from whether the POMDP's transition and observation kernels are stationary. Distribution shift also affects other supervised learners, but robotics can attach physical costs to the resulting errors.

Behavior cloning turns this into a supervised learning problem with a simple objective over a dataset $\mathcal{D}$ of demonstrations:

$$\mathcal{L}_{\text{BC}}(\theta) = -\mathbb{E}_{(o, q, \ell, a) \sim \mathcal{D}} \left[ \log \pi_\theta(a \mid o, q, \ell) \right]$$

where:

• $\mathcal{D}$: the dataset of expert demonstrations, each an observation-instruction-action tuple
• $(o, q, \ell, a) \sim \mathcal{D}$: a single demonstration drawn from the dataset
• $\pi_\theta(a \mid o, q, \ell)$: the conditional density of action $a$ for continuous actions, or probability mass for discrete action bins, given the observation, proprioception, and instruction
• $\mathcal{L}_{\text{BC}}(\theta)$: the negative log-likelihood loss, the expectation under the demonstration distribution

Everything distinctive about VLA architectures is downstream of how you represent the conditioning inputs and how you parameterize $\pi_\theta$ over a continuous, high-precision action space. The objective itself is unremarkable; what makes the problem interesting is the representation choice around it.

Two input-fusion choices and one action-output choice show how these models are assembled. They are design axes, not three mutually exclusive families: a model can fuse visual and language tokens and also emit actions as text tokens.

Late fusion with a shared embedding. Separate encoders process images and language, then their embeddings are concatenated and fed to an action head. This is a relatively simple design, though its data and compute costs depend on the encoders and which weights are trained. The visual encoder can be changed independently. Cross-modal interaction begins after each encoder has formed its features, so the visual features themselves cannot be conditioned on whether the instruction says "pick up the mug" or "avoid the mug." A sufficiently expressive downstream head may still distinguish the two tasks.

Token-level joint fusion. Visual features and language tokens enter a shared transformer context, sometimes with proprioceptive tokens. In OpenVLA specifically, DINOv2 and SigLIP visual features are fused, projected into the Llama 2 input-embedding space, and processed with text by the language-model backbone. This is joint self-attention over the resulting token sequence, not a dedicated cross-attention module at every layer, and OpenVLA does not require proprioception as an input. RT-2 also builds on pretrained vision-language backbones, but details differ across its variants. Token-level fusion permits task-dependent interaction between modalities at computational cost; an attention pattern by itself does not prove that a phrase is correctly grounded.

Action-as-text output. Independently of the input-fusion choice, a model can discretize each action dimension and produce action tokens through its language decoder rather than through a separate continuous-action head. RT-2 used existing integer tokens for action bins in its PaLI-X variant and repurposed rarely used tokens in its PaLM-E variant. In both cases, the output is decoded as an action when the model is prompted for robot control. This makes instruction understanding, visual question answering, and motor control next-token prediction over a shared backbone.

RT-2's combination of pretrained multimodal inputs and action-token output is worth dwelling on, because it supports behaviors absent from the robot demonstrations. When RT-2 was co-fine-tuned on web-scale vision-language data alongside robot trajectories, it retained vision-language capabilities that could inform control. The RT-2 paper tested instructions involving symbols, reasoning, and recognition. In its reported unseen-scenario evaluation, RT-2 reached 62% success versus RT-1's 32%; a separate Language Table simulation evaluation reported 90% for RT-2, so those figures are not endpoints of one comparison. The results are consistent with transfer from web pretraining through shared weights, although the evaluation alone does not isolate a single causal mechanism.

The pretrained vision-language backbone contributes three distinct things that are easy to conflate, and separating them clarifies why the same architectural choice helps some tasks and not others.

• Semantic priors. Object categories, attribute terms, common-sense relations, and some spatial relationships can be learned from broad image-text data. Such pretraining may help resolve "the fruit next to the bowl" in a new scene, but the target robot still needs to localize the actual objects and geometry.
• Broadly pretrained perceptual features. Encoders such as CLIP, SigLIP, and DINOv2 have seen much broader visual data than a small task-specific encoder trained on a few hundred demonstrations. Their features may help under variation in lighting, viewpoint, and clutter, but robustness relative to a task-specific baseline must be measured on the target setting. This is the motivation for the vision-only world models in Chapter 2: Pretrained Visual Models and DINO-WM, not a guarantee of superior control.
• A shared interface across tasks and embodiments. Language can express a new task without changing a fixed classifier vocabulary. Whether the policy executes that task still depends on its training coverage, perception, and action capabilities; new demonstrations or adaptation may be needed.

The costs are less discussed but shape architecture decisions just as strongly, and in practice they are the constraints that architects argue about.

The first is spatial precision. Image encoders vary in patch size and training objective; an encoder optimized for broad semantic recognition may not preserve every detail needed for a particular grasp. The tolerable localization error depends on the object, gripper, and controller. Some systems combine visual pathways with complementary strengths. OpenVLA fuses DINOv2 features, which retain patch-level spatial structure, with language-aligned SigLIP features before projection into its language backbone. That design does not by itself prove that either pathway supplies sufficient grasp precision; the target task must test it.

The second is latency. At an illustrative decoding rate of 100 tokens per second, an autoregressive backbone emits one token every 10 milliseconds, already using an entire 100 Hz control period. A command with eight scalar action tokens takes about 80 milliseconds before prefill, communication, and execution overhead. We will spend a large part of the next section on action-tokenization tricks that reduce this delay, and a large part of the section after that on architectures that split the slow reasoner from the fast actor.

The third is fine-tuning instability. Fine-tuning a large pretrained backbone on a narrow robot dataset can degrade capabilities that made it valuable, a catastrophic-forgetting risk that task training loss may not expose. Parameter-efficient adaptation, such as low-rank adapters on a frozen backbone, and co-fine-tuning with a mixture of robot and pretraining data are two responses. Their trade-offs depend on the task and training regime rather than guaranteeing preservation of every pretrained capability.

A reactive VLA using one frame is memoryless with respect to its input. Its observation stream can be non-Markov even when the underlying POMDP state is Markov. Two consequences follow.

First, some tasks require information absent from the current frame. "Put the apple where the cup was" requires the earlier cup location after the cup moves. A strictly one-frame policy cannot recover that hidden event from its current image alone unless the information is available through another input or prior. The written form $o_{\leq t}$ admits history; a short frame window helps while the relevant event remains within it, while an explicit memory module can retain information longer, as discussed in Part III, Chapter 4. Neither approach guarantees that the policy will use the information correctly.

Second, occlusion can make a single view ambiguous. A gripper may block a camera's view of the object during a grasp, while another camera might still see it. Recent frames can help recover what was visible before the occlusion, but a short window will not always disambiguate the scene. The benefit depends on camera placement, geometry, and how long the relevant information remains available.

The practical compromise used by many current systems is action chunking with a short observation window. The model sees the last one or two frames and emits a block of future actions. Chunking amortizes policy inference across several control steps, while the window supplies recent context for partially observed scenes.

A question worth asking of any instruction-conditioned policy: is the reported success rate coming from visual grounding, or could the model be relying on familiar scene configurations and a weak instruction prior? Aggregate success can hide either mechanism.

Useful diagnostics are interventions rather than attention pictures alone. Shuffle images across otherwise matched trials and measure whether performance changes; replace the instruction with a conflicting one and check whether actions change in the intended direction. Attention visualizations can help locate candidate failure modes, but attention weights are not causal proof of visual grounding. A policy may also solve some benchmark tasks from proprioception and scene priors, so image and instruction ablations are needed before attributing success to either modality.