Pretraining, Post-Training, Scaling, and Adaptation › Transfer across embodiments and domains

An additional scaling problem in world modeling is transfer: making a model trained on one robot or simulator work under a different embodiment or domain. This section covers the main strategies and their conditions.

A world model trained on one embodiment has implicitly learned that embodiment's dynamics. The learned dynamics are a function of the robot's mass distribution, actuator limits, sensor noise, and contact properties. When you move to a new embodiment, some of these change dramatically and some stay the same. The model must distinguish the two. Because the model has no way of knowing in advance which aspects of dynamics are shared and which are specific to the training embodiment, it has to learn that distinction from data. This is what makes transfer both possible (because there is shared structure) and hard (because separating shared from specific is not trivial).

The general problem is a distribution shift over the joint distribution of states, actions, and transitions. There are three qualitatively different cases.

1. Observation-distribution shift. The physical dynamics stay fixed, but the distribution or encoding of observations changes. A new sensor-noise process is one example. If only the input marginal changes while the predictive conditional remains fixed, this is the narrower statistical case usually called covariate shift.
2. Dynamics shift. The transition function changes but the state and action spaces do not. Example: a robot with the same body but a different payload. Adaptation requires updating the dynamics.
3. Structural shift. The state space, action space, or morphology changes. Example: moving from a wheeled robot to a quadruped. Adaptation requires reinterpreting the state and action representations, which is qualitatively harder.

The cases suggest different interventions, but their difficulty is empirical. An observation shift may be handled by recalibration when the underlying state remains recoverable. A dynamics shift may suit a compact adapter or task descriptor when the source representation already spans the target variation. Structural shift usually requires explicit interface alignment or architectures that support heterogeneous inputs and outputs.

The Open X-Embodiment collaboration pooled data from 22 robot embodiments. Its RT-X experiments reported positive transfer from cross-embodiment co-training on several evaluated robots, while also documenting cases where co-training hurt particular tasks. This is evidence that a shared mixture can help, not a guarantee for an unseen platform.

For cross-embodiment learning to work, three things need to be aligned.

• State representation. Different robots report different state vectors. The model must map them into a common space, either by hand-designed normalization or by a learned encoder.
• Action space. A robot arm has a different action space than a mobile base. Common strategies include using a shared low-level action interface, learning a latent action space, or conditioning the model on an embodiment descriptor so it can interpret actions differently per embodiment.
• Observation space. Camera images are easier to place in a common tensor format than robot-specific proprioception, although viewpoint, calibration, and appearance still vary. Proprioceptive inputs require their own alignment or embodiment-specific encoding.

The action-space question connects directly to the latent action material from Part VI: Learning World Models at Scale. A learned latent action space is one way to share structure across heterogeneous interfaces. Its success depends on paired behavior that identifies comparable effects; different embodiments need not admit the same latent control axes. A standardized low-level interface or embodiment-specific decoder may be more reliable when such correspondence is weak.

When the target domain is the real world and the source domain is a simulator, domain randomization samples simulator parameters such as masses, friction, sensor noise, or appearance during training so that deployment may fall within the trained distribution (Tobin et al. 2017).

Domain randomization is more likely to transfer when:

• The randomization covers the true parameters.
• The model has enough capacity to represent the whole randomized family.
• The task is robust to the specific realization of the parameters.

Parameter randomization cannot cover structural errors that are absent from the simulator family—for example, a contact mode the simulator never represents. Real-world adaptation can correct residual error that the chosen model class can express, but it does not guarantee recovery from arbitrary simulator misspecification. The simulator ranges and the amount of real data should therefore be validated against deployment behavior.

A simple and effective technique for cross-embodiment transfer is to condition the model on a compact description of the embodiment. This is the same idea as task conditioning, but the conditioning signal is specifically about the robot.

Embodiment descriptors can be:

• Hand-designed. Mass, dimensions, degrees of freedom, actuator types.
• Learned. A small embedding trained per embodiment, shared across tasks.
• Inferred. Estimated from a few transitions using an encoder, so that a brand-new robot can be described without any explicit metadata.

Inferred descriptors avoid requiring complete metadata: an encoder maps a short history of transitions to a latent system representation. This is a form of latent system identification. The inferred factors are identifiable only to the extent that the observed actions excite the relevant dynamics and the history contains enough information. They may absorb nuisance correlations or miss hidden factors, so calibration on held-out interventions remains important.

Transfer is not always positive. When source representations or priors are poorly matched to the target, a pretrained model can perform worse than a target-only baseline under the same adaptation budget. The outcome depends on initialization, optimizer, regularization, and data, so it is not necessarily slower than scratch at every point; the comparison must be measured on the target protocol.

The main defenses against negative transfer are:

• Data mixing. Train on source and target together, with a mixing ratio that reflects the importance of each.
• Rehearsal. Keep a small buffer of source data and replay it during target adaptation.
• Elastic weight consolidation. Penalize changes to parameters that were important for the source task.
• Conditioning. Give the model enough task information that it can choose to ignore the source prior when the target demands it.

These defenses address different failure modes. Rehearsal preserves measured source behavior, conditioning separates task-specific regimes when the descriptor is informative, and regularization limits selected parameter changes. Their value depends on the target-data budget and the kind of shift, so a target-only baseline and source-retention evaluation should accompany the choice.