Pretraining, Post-Training, Scaling, and Adaptation

Real-world models are rarely trained once on one task from one data source. They are built in stages. A large pretraining phase consumes heterogeneous data across environments, embodiments, and modalities. A post-training phase reshapes the model so its rollouts support the decisions an agent will make. An adaptation phase specializes the pretrained model to a particular robot, task, or user, often with only a small amount of new data. This chapter is about that pipeline.

Why use a pipeline at all? Training a separate model for every application repeats representation learning and prevents data sharing. Pretraining can amortize that cost when source and target systems share useful structure. Post-training then changes what the model is optimized to do, while adaptation fits the remaining target-specific differences. Keeping those operations conceptually separate also makes failures easier to diagnose.

Suppose a team has simulation, passive video, and a small amount of target-robot interaction. A staged workflow lets it test three separate questions: whether the broad mixture learns a transferable representation, whether a control-oriented objective improves decisions, and how much target data adaptation needs. Source diversity helps only when it contains structure relevant to the target, so each stage needs its own held-out evaluation.

The three stages have different objectives and different failure modes.

• Pretraining is a data-hungry, compute-hungry phase. Its job is to absorb broad structure: how objects move, how contact works, how different sensor modalities relate to one another. Its failure modes are data contamination, memorization of spurious correlations, and evaluation that rewards surface-level fit rather than transferable structure.
• Post-training is a preference-and-goal-driven phase. Its job is to shape a general predictor into a model whose rollouts are useful for control, not merely accurate on average. Its failure modes are reward hacking, distribution collapse, and loss of the very diversity that made the pretrained model valuable.
• Adaptation is a specialization phase. Its job is to make the model work on a specific target distribution with few samples. Its failure modes are overfitting, catastrophic forgetting, and negative transfer when the target is far from the pretraining distribution.

Each stage optimizes a different quantity. Pretraining minimizes average prediction error over a broad data mixture. Post-training minimizes a scalar downstream signal, such as reward, cost, or preference. Adaptation minimizes error on a narrow target distribution using few samples. These quantities can conflict: a model optimized for one stage may be suboptimal for another.

The stages are not always separated in practice. Some models are pretrained and adapted directly; others undergo reinforcement-learning post-training without later adaptation. The conceptual separation still matters because the objectives differ. A post-trained model can have worse held-out prediction error yet produce better closed-loop decisions, so both quantities need to be measured.

This chapter develops the vocabulary and design tradeoffs needed to reason about the pipeline. It covers scaling, task conditioning and adaptation, preference and control-oriented post-training, and transfer across embodiments. A worked implementation then pretrains a small task-conditioned dynamics model, adapts it to a new system with a few trajectories, and demonstrates a preference loss.

The material builds on Part VI: Learning World Models at Scale and prepares you for Part VII: Planning and Agency, where an optimizer consumes the model's predictions.

For autoregressive language models studied within a fixed model and data family, held-out loss often follows a smooth empirical trend with the number of model parameters $N$ and training tokens $D$. This is not a theorem, and the fitted law does not automatically transfer to a different architecture, modality, or world-model objective. For world models, it is best treated as a budget-planning hypothesis to test with a small scaling ladder.

If a fitted law predicts loss within its measured range, it can compare candidate runs before paying for each full experiment. For example, it can test whether a fixed FLOP budget favors more parameters or more data. The fit makes the assumed tradeoff explicit; short runs near the proposed allocation must still verify it.

A useful form, closely related to the empirical language-model scaling laws studied by Kaplan et al. (2020), is a sum of power-law terms:

where:

• $N$: the number of model parameters
• $D$: the number of training tokens (or transitions)
• $N_c$: a critical model size that sets the scale of the parameter term
• $D_c$: a critical dataset size that sets the scale of the data term
• $\alpha_N$: the exponent governing how quickly the parameter term decays as the model grows
• $\alpha_D$: the exponent governing how quickly the data term decays as the dataset grows
• $L_\infty$: the irreducible loss floor

Each term has a meaning.

• The parameter term $\left(N_c/N\right)^{\alpha_N}$ is the fitted contribution that shrinks as the model grows. It may reflect limited model capacity, but the empirical decomposition does not identify a unique causal source of error.
• The data term $\left(D_c/D\right)^{\alpha_D}$ is the fitted contribution that shrinks as the dataset grows. It may reflect finite-data effects, including estimation error and incomplete coverage.
• The floor $L_\infty$ is the asymptote of this fitted model. Its value depends on the data distribution, representation, model family, optimizer, and loss; it is not necessarily an intrinsic constant of the environment.

The separation is a diagnostic within the fitted regime. If the parameter-dependent contribution dominates, a larger model is one candidate intervention. If the data-dependent contribution dominates, more or better-covered data is one candidate. If measured loss is close to the fitted floor, simply extending the same scaling path is predicted to yield little improvement; changing the representation, data distribution, model family, or loss may move that floor. These are hypotheses for follow-up experiments, not causal conclusions supplied by the curve fit.

In Kaplan et al.'s fitted form, $\alpha_N \approx 0.076$ and $\alpha_D \approx 0.095$. Exponents of that size imply slow improvement in each isolated term: a tenfold increase in the associated resource reduces that term by about 10 to 20%, and halving it requires a thousandfold to millionfold increase. Other fits use different definitions and exponents, so compare the predicted loss change rather than the raw exponent alone.

The sum-of-power-laws form makes the two fitted contributions trade off. For a fixed compute budget $C \approx 6 N D$ (a common approximation for dense transformer training), you can choose a larger model trained on fewer tokens or a smaller model trained on more. Hoffmann et al. (2022) estimated approximately equal scaling of model size and training tokens in their studied language-model regime. Their 70-billion-parameter Chinchilla model used 1.4 trillion tokens, a ratio of about 20 tokens per parameter. That ratio describes one empirical regime and should not be treated as a universal constant; “too much” or “too little” data is meaningful only relative to a fixed compute budget and a validated scaling fit.

The Chinchilla result changed how language-model researchers reason about fixed-compute allocation: in those experiments, rebalancing compute toward more training tokens produced a smaller model with better loss than larger under-trained alternatives. Its durable lesson is methodological rather than numerical. Estimate the balance point for the architecture, data, and objective at hand instead of importing a token ratio from another setting.

For world models, the analogue of “tokens per parameter” is not obvious. A transition $(s_t, a_t, s_{t+1})$ has different structure from a text token, adjacent transitions are dependent, and interaction data may be costly to collect. Here $s_t$ is the state at time $t$, $a_t$ is the action, and $s_{t+1}$ is the next state. Collection cost and training FLOPs are separate constraints: an expensive transition does not by itself change the compute optimum, but it can make the data-optimal point infeasible. Applying scaling ideas responsibly therefore requires a domain-specific definition of $D$, a coverage measure, and an explicit collection budget.

The scaling picture for world models differs from language in several important ways.

Sequential dependence changes the meaning of data scale. Both text and interaction corpora contain dependent observations, but trajectories can exhibit especially strong temporal correlation and policy-dependent coverage. Adjacent transitions may add less information than transitions from distinct states, tasks, or embodiments. Consequently, raw transition count is an incomplete proxy for effective dataset size. A useful scaling study should report $D$ together with trajectory diversity, state-action coverage, and the data-collection policy.

Context length is a separate experimental axis. A longer history can expose information needed for partially observed or delayed dynamics, but it also changes memory cost, optimization, and the distribution of training windows. More context helps only when the additional history is informative and the model and dataset support it. For that reason, context length should be varied and measured alongside $N$ and $D$, rather than assumed to translate directly into a particular planning horizon.

Action and embodiment coverage are additional axes. Beyond $N$, $D$, and $C$, a world model must cover relevant action spaces, embodiments, and dynamics regimes. Diverse pretraining can improve transfer when the target shares structure with the source mixture, although diversity is not a substitute for target-relevant coverage. Scaling studies should therefore distinguish the number of transitions from the breadth and balance of environments that generated them.

Stochasticity and multimodality matter. Next-token prediction and next-state prediction can both be multimodal. For a proper probabilistic loss, irreducible uncertainty under the chosen observations and targets contributes to the attainable loss; for point losses, multimodality can instead produce averaged predictions. A fitted $L_\infty$ also absorbs representation and model misspecification. Distributional prediction, better observations, or a different target can therefore change the measured floor.

Scaling curves are usually plotted on log-log axes, where a single power-law term appears linear. Thresholded capability metrics can nevertheless show step-like jumps. For example, if a planning benchmark awards credit only after a rollout crosses a fixed success threshold, a smooth change in predictive quality can look discontinuous. Such a plot is evidence about the metric and experiment, not proof that planning ability itself appeared discontinuously.

The emergence picture is contested. Schaeffer et al. (2023) showed that some reported jumps can arise from nonlinear or discontinuous metrics applied to smoothly improving models. That result does not rule out every discontinuity. For world models, test both continuous quantities and task success across several scales before assigning a mechanism to an apparent threshold.

In practice, a pretraining run is constrained by a compute budget $C$ and a wall-clock budget. The standard prescription is:

• Estimate the scaling exponents on a small set of runs (the "scaling ladder").
• Fit the exponents $\alpha_N$, $\alpha_D$, $L_\infty$, $N_c$, $D_c$.
• Choose $N$ and $D$ to minimize the predicted loss subject to $6 N D \le C$.
• Spend the remaining budget on a longer run at the chosen $(N, D)$.

The fitted exponents are only trustworthy within the range of the ladder. Extrapolating a power law fitted on $10^6$ to $10^8$ parameters out to $10^{11}$ parameters is an act of faith, not a calculation. Still, the exercise is useful because it forces you to state your assumptions explicitly and to notice when a proposed training run is far from the predicted frontier. Beyond that, the process is a mechanism for organizing disagreement. When two engineers propose different training runs, they can point at the fitted law and ask which run is closer to the frontier and which assumption is being challenged. That conversational clarity, more than the numerical prediction, is often the real return on investing in a scaling ladder before committing to a large run.

Pretraining gives you a general model. Adaptation makes that model specific. The central question of adaptation is how to inject task information into a pretrained model without destroying what the pretraining learned. There are three broad families of answers: full fine-tuning, parameter-efficient fine-tuning, and conditioning. Each family represents a different bet about how much of the pretrained model needs to change for the target task, and each has different costs in memory, compute, and risk of forgetting.

The naive approach is to take a pretrained model and continue training it on data from the new task. This works, but it has two well-known problems.

Catastrophic forgetting. Target-only updates can reduce performance on previously learned tasks, especially when their data are absent during adaptation. The amount of forgetting depends on task similarity and the training protocol; it should be measured on retained source evaluations rather than inferred from target performance alone. Methods such as elastic weight consolidation were developed to reduce this failure mode.

Sample efficiency. Full fine-tuning exposes every weight to the target loss. With a small target dataset, this gives the optimizer many ways to fit idiosyncrasies rather than reusable structure. Whether overfitting occurs depends on the initialization, regularization, learning rate, and validation protocol, but early stopping and source-retention checks are especially important in this regime.

Both problems motivate restricting which parameters are updated. The natural question becomes: how much of the model needs to change? If the answer is "much less than all of it", then the two problems above collapse into a much smaller version of themselves.

The idea behind parameter-efficient fine-tuning (PEFT) is to update a small number of parameters while keeping the pretrained backbone frozen. A widely used method is LoRA (Hu et al. 2021). For a frozen linear layer with weight matrix $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, LoRA adds a low-rank update:

where:

• $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$: the frozen pretrained weight matrix
• $W$: the adapted weight matrix
• $d_{\text{in}}$: the input dimension of the frozen layer
• $d_{\text{out}}$: the output dimension of the frozen layer
• $A \in \mathbb{R}^{r \times d_{\text{in}}}$: the first trainable low-rank factor
• $B \in \mathbb{R}^{d_{\text{out}} \times r}$: the second trainable low-rank factor
• $r \ll \min(d_{\text{in}}, d_{\text{out}})$: the rank of the update
• $\alpha$: a scaling constant
• $\frac{\alpha}{r} B A$: the low-rank update added to the frozen weights

Only $A$ and $B$ are updated. The forward pass becomes

where:

• $h$: the output of the adapted linear layer
• $u$: the input vector to the linear layer
• $W_0 u$: the frozen pretrained transformation
• $\frac{\alpha}{r} B A u$: the low-rank adaptation applied to the input

LoRA assumes that a useful weight update can be represented at low rank. Hu et al. (2021) found this parameterization effective for adapting large language models, but it is not guaranteed for every layer or domain. A physical system may vary along only a few latent factors, such as mass or damping, yet a nonlinear network can still require a higher-rank weight change to represent their effect. Rank is therefore an empirical capacity choice, not a consequence of the physical parameter count.

Other PEFT methods include:

• Adapters: small bottleneck MLPs inserted between frozen layers.
• Prefix and prompt tuning: learnable vectors prepended to the input or to attention keys and values.
• Bias-only tuning: updating only the bias terms.
• IA$^3$ and related methods: scaling the activations of frozen weights by learned vectors.

Each has different tradeoffs in expressiveness, memory, and inference latency. LoRA is a common default because it adds no inference latency when merged back into $W_0$. Adapters are easy to stack and interpret, prefix tuning avoids touching the weight matrices entirely, and bias-only tuning is the cheapest of all but also the least expressive. In practice, the choice between them is often dominated by the deployment constraints: if you need to serve dozens of adapted variants from a single base model, LoRA's ability to merge into the base weights or sit as a small sidecar is usually decisive.

A complementary approach is to condition the model on task information rather than modifying its weights. This is what language models do when they are prompted. The model is unchanged, but the input carries a description of what to do. Conditioning is appealing because it can be applied without any gradient updates at all, provided the pretrained model has learned to interpret the conditioning signal during pretraining.

For world models, instruction conditioning comes in several flavors.

• Task identifiers. A learned embedding per task, concatenated to the model's input. This is the simplest form, and it is what our worked example uses.
• Natural language instructions. A text description such as "this robot has a soft gripper and a heavy base", encoded and used as a conditioning vector. This can describe a new task without fitting a task ID, provided pretraining taught the model how those descriptions relate to dynamics.
• Goal specifications. A target state, a set of constraints, or a reward function, passed to the model as input. This is the foundation of goal-conditioned policies and goal-conditioned world models.
• Demonstrations. A few example transitions in the context window. This is in-context learning, and it is the most flexible form of adaptation because it requires no weight updates at all.

Instruction conditioning shifts work into pretraining. The model must see enough paired descriptions and dynamics to learn how the signal should affect predictions. If the signal omits a task-relevant distinction, two different systems become indistinguishable to the predictor. Conversely, a high-capacity descriptor is useful only when the data identify how each part should be used; otherwise the model may ignore it or fit spurious correlations. Generalization to a newly described task must be measured rather than assumed.

Another approach is meta-learning: training an initialization so that a small number of gradient steps can adapt it to a task. The classic formulation is MAML (Finn et al. 2017), which optimizes the initialization through one or more task-specific gradient updates. For world models, this directly targets a regime where interaction data is scarce, provided the meta-training task distribution represents the deployment tasks.

Meta-learning depends on the distribution of tasks used during meta-training. A target outside that distribution can erase its advantage over ordinary fine-tuning. Original MAML differentiates through the inner adaptation step, while first-order variants approximate or omit those second-order terms. The general lesson is that rapid adaptation is itself learned under a task distribution and must be evaluated under the shifts expected at deployment.

Pretraining gives you a model that predicts the world. Post-training gives you a model that predicts the world in a way that is useful. The difference is subtle but has practical consequences. A pretrained model optimizes average prediction error. A post-trained model optimizes whatever the downstream decision-maker cares about. These are rarely the same objective, and the gap between them is what post-training exists to close.

Consider a world model used inside a model predictive controller. The controller samples action sequences, rolls them out under the model, and picks the sequence that minimizes a cost. The model must rank candidate sequences well near the optimum, not merely minimize average one-step error. Correlated model bias is especially dangerous because an optimizer can repeatedly exploit it. Zero-mean noise may average out under repeated sampling, but it can still harm ranking when samples are few or costs are nonlinear.

This is the same mismatch that motivates the decision-aware learning literature and the model-based RL objectives we discussed in Part II: Inference, Decisions, and Control. The point here is that post-training is the stage where you close that mismatch, by optimizing the model against a decision-relevant signal. Recognizing which quantity the downstream controller is sensitive to, and shaping the model's loss accordingly, is the central design question of post-training. Sometimes this means reweighting the loss; sometimes it means adding an entirely new term; sometimes it means training against a preference signal that is only available through comparison. All of these techniques are ways of pushing the model's errors into regions the planner does not care about.

A post-training pipeline can begin with supervised fine-tuning (SFT) on curated data. For a world model, SFT data might be:

• High-quality trajectories from expert demonstrations.
• Transitions from tasks that resemble the target deployment.
• Synthetic rollouts from a trusted simulator.
• Instruction-conditioned examples, where the model learns to map a task description to the right dynamics.

Plain output-likelihood SFT teaches the conditional distribution represented by its examples. It does not explicitly express a pairwise preference between two plausible outputs unless the training format includes that comparison. Preference data adds this supervision, although curated demonstrations can encode some of the same behavior indirectly.

To teach preferences, one pipeline is reinforcement learning from human feedback (RLHF), demonstrated for complex behaviors by Christiano et al. (2017) and later applied to instruction-following language models by Ouyang et al. (2022). The pipeline has three stages.

1. Collect comparisons. Show a human two candidate model outputs (two rollouts, two predicted futures, two action sequences) and ask which is better.
2. Fit a reward model. Train a scalar function $r_\phi(x, y)$ that predicts the human preference.
3. Optimize the policy against the reward model. Use reinforcement learning (typically PPO) to update the model so it produces outputs the reward model likes.

For a generative world model, compared outputs could be trajectories or predicted futures, and “better” must be defined for the deployment. Human comparisons can supply that label. Automatic signals such as task success, controller cost, or constraint violations instead create proxy-feedback preference data; they are not human feedback merely because the same optimization template is used. Either source can be misspecified, so held-out closed-loop evaluation remains necessary.

RLHF requires fitting a reward model and then running an RL optimization loop. Direct Preference Optimization (DPO, Rafailov et al. 2023) shows that, under its reward-model and KL-regularized policy assumptions, you can optimize the corresponding preference objective without fitting an explicit reward model.

Given a preferred output $y_w$ and a rejected output $y_l$ for the same input $c$, and a reference model $\pi_{\text{ref}}$ (usually the SFT model), the DPO loss is

where:

• $\mathcal{L}_{\text{DPO}}(\theta)$: the DPO loss for policy parameters $\theta$
• $\theta$: the parameters of the policy being trained
• $\mathbb{E}_{(c, y_w, y_l)}$: the expectation under the data distribution over conditioning input $c$, preferred output $y_w$, and rejected output $y_l$
• $c$: the conditioning input (for example, a task description or state history)
• $y_w$: the preferred (winning) output
• $y_l$: the rejected (losing) output
• $\sigma$: the logistic sigmoid function
• $\beta$: a temperature-like parameter controlling the strength of the implicit KL penalty
• $\pi_\theta(y \mid c)$: the probability of output $y$ under the current policy given conditioning input $c$
• $\pi_{\text{ref}}(y \mid c)$: the probability of output $y$ under the frozen reference model given conditioning input $c$

The ratio $\pi_\theta(y_w \mid c)/\pi_{\text{ref}}(y_w \mid c)$ compares how much more the current model prefers the winning output than the reference does. The same ratio for $y_l$ does this for the losing output. The difference between these log-ratios is scaled by $\beta$ and passed through a sigmoid, so the loss is small when the model already prefers $y_w$.

The intuition is that DPO increases the relative log-probability of preferred outputs over rejected ones, while a KL-like penalty (implicit in the reference model) keeps the policy from drifting too far from the pretrained model. The parameter $\beta$ controls the strength of that penalty. A small $\beta$ imposes a weak implicit KL penalty and lets the model separate winning from losing outputs more aggressively; a large $\beta$ imposes a stronger penalty and keeps the policy closer to the reference model. In practice, $\beta$ is one of the most consequential hyperparameters in a DPO training run, and it is worth tuning as carefully as the learning rate.

DPO avoids a separately fitted reward model and can be implemented with a supervised-style loop. Applying it to world-model outputs requires a normalized conditional likelihood, or an explicitly justified surrogate, for each compared sequence. Preference pairs might come from expert and baseline rollouts or human comparisons; how they are labeled determines what the model will learn.

The most direct form of post-training is to optimize the model against the actual control objective. Several variants exist.

• Value-aware training. Train the model so that the value function computed from its rollouts matches the true value. This makes the model useful for policy optimization even if its per-step predictions are imperfect.
• Cost-aware training. Weight the prediction loss by the sensitivity of the downstream cost to prediction error. Regions where the planner is sensitive get more weight.
• Adversarial and robust training. Train the model to be accurate on the worst-case action sequence, not the average one. This reduces model exploitation, where the planner finds a sequence that looks good under the model but fails in reality.
• Constraint-aware training. Penalize predictions that violate known physical or safety constraints, even when the data does not contain such violations.

All of these objectives treat the model as a component of a decision system. Closed-loop performance is therefore the primary test of decision usefulness, while open-loop prediction metrics remain valuable diagnostics for calibration, coverage, and failure analysis. A model with worse average prediction error can still produce better decisions if its errors are better aligned with the planner's sensitivities.

A final distinction that matters for post-training is process versus outcome supervision.

• Outcome supervision labels the final result. It can be easy or difficult to collect depending on the task, and it leaves ambiguity about which intermediate decisions caused the outcome.
• Process supervision labels intermediate predictions or decisions. When reliable intermediate targets are observable, it can provide denser credit assignment, but collecting or validating those targets may be costly.

For world models, an observed transition supplies a next-observation target, so logged trajectories can provide dense supervised signals. Those signals are not free: interaction may be expensive, sensors may be partial or noisy, and the observed next state need not reveal the latent state a planner requires. Dense one-step supervision also does not remove the evaluation mismatch: a model can predict one step well and still accumulate harmful rollout error.

An additional scaling problem in world modeling is transfer: making a model trained on one robot or simulator work under a different embodiment or domain. This section covers the main strategies and their conditions.

A world model trained on one embodiment has implicitly learned that embodiment's dynamics. The learned dynamics are a function of the robot's mass distribution, actuator limits, sensor noise, and contact properties. When you move to a new embodiment, some of these change dramatically and some stay the same. The model must distinguish the two. Because the model has no way of knowing in advance which aspects of dynamics are shared and which are specific to the training embodiment, it has to learn that distinction from data. This is what makes transfer both possible (because there is shared structure) and hard (because separating shared from specific is not trivial).

The general problem is a distribution shift over the joint distribution of states, actions, and transitions. There are three qualitatively different cases.

1. Observation-distribution shift. The physical dynamics stay fixed, but the distribution or encoding of observations changes. A new sensor-noise process is one example. If only the input marginal changes while the predictive conditional remains fixed, this is the narrower statistical case usually called covariate shift.
2. Dynamics shift. The transition function changes but the state and action spaces do not. Example: a robot with the same body but a different payload. Adaptation requires updating the dynamics.
3. Structural shift. The state space, action space, or morphology changes. Example: moving from a wheeled robot to a quadruped. Adaptation requires reinterpreting the state and action representations, which is qualitatively harder.

The cases suggest different interventions, but their difficulty is empirical. An observation shift may be handled by recalibration when the underlying state remains recoverable. A dynamics shift may suit a compact adapter or task descriptor when the source representation already spans the target variation. Structural shift usually requires explicit interface alignment or architectures that support heterogeneous inputs and outputs.

The Open X-Embodiment collaboration pooled data from 22 robot embodiments. Its RT-X experiments reported positive transfer from cross-embodiment co-training on several evaluated robots, while also documenting cases where co-training hurt particular tasks. This is evidence that a shared mixture can help, not a guarantee for an unseen platform.

For cross-embodiment learning to work, three things need to be aligned.

• State representation. Different robots report different state vectors. The model must map them into a common space, either by hand-designed normalization or by a learned encoder.
• Action space. A robot arm has a different action space than a mobile base. Common strategies include using a shared low-level action interface, learning a latent action space, or conditioning the model on an embodiment descriptor so it can interpret actions differently per embodiment.
• Observation space. Camera images are easier to place in a common tensor format than robot-specific proprioception, although viewpoint, calibration, and appearance still vary. Proprioceptive inputs require their own alignment or embodiment-specific encoding.

The action-space question connects directly to the latent action material from Part VI: Learning World Models at Scale. A learned latent action space is one way to share structure across heterogeneous interfaces. Its success depends on paired behavior that identifies comparable effects; different embodiments need not admit the same latent control axes. A standardized low-level interface or embodiment-specific decoder may be more reliable when such correspondence is weak.

When the target domain is the real world and the source domain is a simulator, domain randomization samples simulator parameters such as masses, friction, sensor noise, or appearance during training so that deployment may fall within the trained distribution (Tobin et al. 2017).

Domain randomization is more likely to transfer when:

• The randomization covers the true parameters.
• The model has enough capacity to represent the whole randomized family.
• The task is robust to the specific realization of the parameters.

Parameter randomization cannot cover structural errors that are absent from the simulator family—for example, a contact mode the simulator never represents. Real-world adaptation can correct residual error that the chosen model class can express, but it does not guarantee recovery from arbitrary simulator misspecification. The simulator ranges and the amount of real data should therefore be validated against deployment behavior.

A simple and effective technique for cross-embodiment transfer is to condition the model on a compact description of the embodiment. This is the same idea as task conditioning, but the conditioning signal is specifically about the robot.

Embodiment descriptors can be:

• Hand-designed. Mass, dimensions, degrees of freedom, actuator types.
• Learned. A small embedding trained per embodiment, shared across tasks.
• Inferred. Estimated from a few transitions using an encoder, so that a brand-new robot can be described without any explicit metadata.

Inferred descriptors avoid requiring complete metadata: an encoder maps a short history of transitions to a latent system representation. This is a form of latent system identification. The inferred factors are identifiable only to the extent that the observed actions excite the relevant dynamics and the history contains enough information. They may absorb nuisance correlations or miss hidden factors, so calibration on held-out interventions remains important.

Transfer is not always positive. When source representations or priors are poorly matched to the target, a pretrained model can perform worse than a target-only baseline under the same adaptation budget. The outcome depends on initialization, optimizer, regularization, and data, so it is not necessarily slower than scratch at every point; the comparison must be measured on the target protocol.

The main defenses against negative transfer are:

• Data mixing. Train on source and target together, with a mixing ratio that reflects the importance of each.
• Rehearsal. Keep a small buffer of source data and replay it during target adaptation.
• Elastic weight consolidation. Penalize changes to parameters that were important for the source task.
• Conditioning. Give the model enough task information that it can choose to ignore the source prior when the target demands it.

These defenses address different failure modes. Rehearsal preserves measured source behavior, conditioning separates task-specific regimes when the descriptor is informative, and regularization limits selected parameter changes. Their value depends on the target-data budget and the kind of shift, so a target-only baseline and source-retention evaluation should accompany the choice.

Let's make the pipeline concrete with a small example. We will work with a family of one-dimensional driven damped point masses. Each "embodiment" is defined by a mass $m$ and a damping coefficient $b$, and the state is a two-vector $(x, v)$ with position and velocity. The transition, using semi-implicit Euler with step size $\Delta t$, is

where:

• $x_t$: the position at time step $t$
• $v_t$: the velocity at time step $t$
• $a_t$: the action (force) applied at time step $t$
• $m$: the mass of the point mass
• $b$: the damping coefficient
• $\Delta t$: the integration step size
• $v_{t+1}$: the velocity after the update
• $x_{t+1}$: the position after the update

Different $(m, b)$ pairs behave differently. In the absence of applied force, velocity decays at a rate set by $b/m$. Because the update contains no restoring force proportional to position, these systems are not oscillators. Our goal is to build a single model that can predict the family and then adapt it to a new parameter pair from a small target dataset.

The pipeline has three stages.

1. Pretrain a task-conditioned model on four embodiments. The model sees the state, the action, and a task identifier, and predicts the next state.
2. Adapt to a fifth, held-out embodiment using a small number of trajectories. We compare three adaptation strategies: doing nothing (zero-shot, using the mean of the training task embeddings), training only a new task embedding while freezing the network, and full fine-tuning.
3. Compare against training a fresh model from scratch on the same small amount of target data.

This example is small enough to run in a few seconds on a CPU, but it captures the essential tradeoffs. The pretrained model has seen many dynamics regimes, so its internal representation is already close to what the new embodiment needs. The from-scratch model has to learn everything from a handful of samples, and it will do poorly. The embedding-only adaptation gets most of the benefit at a fraction of the parameter cost of full fine-tuning.

The scaling-law experiment from the next section gives you the tools to reason about how this picture changes as the pretraining dataset grows. The DPO example at the end shows how to shape the model's outputs with a preference signal, which is the post-training half of the story.

Let's build the pipeline end to end. We start with a scaling-law fit, then move to the embodiment experiment, then finish with parameter-efficient adaptation and preference-based post-training.

We start by generating synthetic loss measurements from a known scaling law, then fitting the exponents back. This mirrors the practical workflow: you measure loss at a small ladder of $(N, D)$ points and fit a power law to extrapolate. The points are chosen to span multiple orders of magnitude in both axes, since a power law can only be identified if the fit data covers enough dynamic range.

Moving along the ray gives a sequence of allocations with a fixed data-to-parameter ratio. To locate this chapter's fitted optimum, you would instead minimize the fitted loss separately along each compute contour and confirm the answer with runs near that point.

Now let's build the embodiment family. We define a rollout function for a driven damped point mass, then a dataset builder that stacks transitions across parameter settings.

The key contrast is between the pretraining data (four embodiments, 80 trajectories) and the adaptation data (one embodiment, two trajectories). The pretrained model has seen a lot of dynamics; the adaptation data is tiny.

We now define a task-conditioned dynamics model. The network takes a state, an action, and a task embedding, and predicts the next state.

The pretrained model has learned a shared dynamics representation plus four task embeddings. Now we adapt.

We evaluate five strategies on the held-out embodiment:

1. Persistence baseline: predict $\hat{s}_{t+1} = s_t$. This is the trivial baseline that any model must beat.
2. Zero-shot: use the pretrained network with the mean of the training embeddings. This represents the case where you have no target data at all.
3. Embedding-only adaptation: freeze the network and the four training embeddings, and fit a single new embedding to the two target trajectories.
4. Full fine-tuning: unfreeze everything and continue training on the target data.
5. From scratch: train a fresh model with a single embedding on the target data alone.

Let's compute each.

The bar chart shows the tradeoff for this seed. Embedding-only adaptation uses eight trainable values and reaches the same order of one-step error as full fine-tuning. The comparison should be repeated across seeds and target systems before drawing a general performance conclusion.

Embedding-only adaptation is a special case of PEFT: it updates a small number of parameters. A more general version is LoRA, which we can apply to the network's linear layers. Let's implement a minimal LoRA wrapper.

For this seed, the three strategies land within a small range of one-step error but use very different trainable parameter counts. A frozen-backbone design can store many small adapted variants without duplicating the backbone.

In this example, LoRA reaches the same order of error as full fine-tuning while updating only a few hundred parameters. The adapter is smaller than a full model copy. Whether accuracy remains comparable is task- and rank-dependent.

Finally, let's look at the post-training half. We want to shape the model so it prefers useful rollouts over harmful ones, using a preference signal. We will use the DPO loss as a conceptual demonstration.

The DPO loss takes log-probabilities of preferred and rejected outputs under the current model and a reference model, and increases the margin between them.

At a zero margin every curve passes through $\log 2$. As the margin grows, the loss approaches zero. Larger $\beta$ makes the loss and its gradient transition more sharply as a function of this fixed margin; the resulting parameter update also depends on how the margin itself changes with the model parameters.

The key parameters for the task-conditioned dynamics model and its adaptation are:

• emb_dim: Dimensionality of each task embedding. Larger embeddings can describe more distinct dynamics regimes but add parameters and increase overfitting risk on small adaptation datasets.
• hidden: Width of the hidden layers in the dynamics network. This sets the capacity of the shared dynamics representation that all tasks reuse.
• epochs: Number of optimization passes over the training data. More passes can reduce training loss, but held-out error may worsen on a small target dataset, so select this value with validation data.
• lr: Learning rate for Adam. The example tunes different values for each stage; dataset size alone does not determine which stage should use the larger rate.
• rank and alpha (LoRA): The rank sets how many trainable parameters the low-rank update adds, and alpha scales that update relative to the frozen weights.
• beta (DPO): The temperature-like parameter that controls the strength of the implicit KL penalty keeping the policy near the reference model.

The pretraining, post-training, and adaptation pipeline is powerful, but each stage has limitations that are easy to overlook when you read only the success stories.

Scaling laws are descriptive, not prescriptive. The exponents you fit on a small ladder may not extrapolate. They depend on architecture, tokenizer, data distribution, and optimizer. A scaling law fitted on text does not transfer to trajectories, and a law fitted on one family of robots may not transfer to another. Treating a fitted exponent as ground truth leads to badly miscalibrated budgets. The most useful role of scaling laws is not to predict the exact loss of a $10^{11}$-parameter run, but to force you to state your assumptions and to notice when a proposed run is far off the frontier.

Post-training can reduce generality. Optimizing a narrow control objective can worsen performance on other prediction distributions. Retaining the pretrained checkpoint, evaluating source tasks, and applying explicit regularization are practical safeguards. DPO is derived from a KL-regularized objective, but finite preference data and optimization do not by themselves guarantee a bound on diversity or out-of-domain behavior.

Adaptation is constrained by the conditioning signal and backbone. A descriptor must distinguish target-relevant regimes, and the pretrained network must know how to use it. Fitting a new embedding can sometimes interpolate or recombine learned behavior, but it cannot guarantee representation of an arbitrary new dynamics family. Natural language expands the descriptor vocabulary without removing this identifiability and coverage requirement.

Negative transfer is possible. A source-trained initialization can underperform a target-only model under the same target-data and optimization budget. Include a target-only baseline and report the adaptation protocol, source retention, and uncertainty across seeds; otherwise a transfer claim has no controlled counterfactual.

Evaluation depends on the decision system. A lower open-loop prediction error need not imply a better controller, and no single planner-independent scalar captures every decision consequence. Report both diagnostic prediction metrics and closed-loop results with the actual planner, horizon, and cost. Cross-paper comparisons are fragile when those components differ.

Compute and data are not the only constraints. Scaling a world model requires not just GPUs and tokens but also access to diverse environments, reliable simulators, and the engineering infrastructure to run long experiments. The scaling curves hide the enormous cost of data collection and curation, which we covered in Part VI: Learning World Models at Scale. A team without access to this infrastructure can still benefit from the conceptual pipeline, but the specific numbers in any scaling law will not apply.

Stage-specific diagnosis. Pretraining can amortize representation learning across targets; post-training changes the objective; adaptation specializes the resulting model. This decomposition provides useful diagnostic questions: is the shared representation inadequate, is the conditioning signal uninformative, is the post-training objective misaligned, or is target data insufficient? The stages can interact, so the diagnosis still requires ablations and stage-specific evaluation.

This chapter developed the three-stage pipeline that underlies modern world-model training.

• Pretraining scales data, parameters, context, and compute to build a broad representation. Empirical scaling laws can describe loss within a measured regime; their fitted floor depends on the observations, targets, model family, and loss as well as irreducible uncertainty. Context, action-space coverage, and embodiment diversity are additional experimental axes.
• Task adaptation and instruction conditioning inject task-specific information without destroying the pretrained representation. Full fine-tuning risks forgetting and sample inefficiency; parameter-efficient methods like LoRA update a small number of parameters; conditioning is cheap at adaptation time but requires the model to have been trained to interpret the signal. Meta-learning targets the few-shot regime directly but is sensitive to task distribution shift.
• Preference and control-oriented post-training shapes a general predictor for planning. RLHF and DPO use preference data; control-oriented objectives weight errors by downstream consequences; process labels can provide denser credit when reliable intermediate targets are observable. A post-trained model can be worse on average prediction error and better in a particular closed loop.
• Transfer across embodiments and domains uses cross-embodiment training, domain randomization, embodiment descriptors, adapters, and rehearsal. Because source training can cause negative transfer, compare it with a target-only baseline under the same evaluation protocol.

The worked example applied the pipeline to driven damped point masses. In one fixed-seed run, pretraining plus embedding-only adaptation outperformed target-only training, and LoRA reached the same order of one-step error as full fine-tuning while updating fewer parameters. The DPO section then showed how relative sequence likelihoods enter a preference objective.

The pipeline prepares you for the planning and agency material in Part VII: Planning and Agency, where a planner queries the model built here. The next chapter turns to sampling-based planning and model predictive control.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about pretraining, post-training, scaling, and adaptation in world-model pipelines.