Pretraining, Post-Training, Scaling, and Adaptation › Limitations & Impact

The pretraining, post-training, and adaptation pipeline is powerful, but each stage has limitations that are easy to overlook when you read only the success stories.

Scaling laws are descriptive, not prescriptive. The exponents you fit on a small ladder may not extrapolate. They depend on architecture, tokenizer, data distribution, and optimizer. A scaling law fitted on text does not transfer to trajectories, and a law fitted on one family of robots may not transfer to another. Treating a fitted exponent as ground truth leads to badly miscalibrated budgets. The most useful role of scaling laws is not to predict the exact loss of a $10^{11}$-parameter run, but to force you to state your assumptions and to notice when a proposed run is far off the frontier.

Post-training can reduce generality. Optimizing a narrow control objective can worsen performance on other prediction distributions. Retaining the pretrained checkpoint, evaluating source tasks, and applying explicit regularization are practical safeguards. DPO is derived from a KL-regularized objective, but finite preference data and optimization do not by themselves guarantee a bound on diversity or out-of-domain behavior.

Adaptation is constrained by the conditioning signal and backbone. A descriptor must distinguish target-relevant regimes, and the pretrained network must know how to use it. Fitting a new embedding can sometimes interpolate or recombine learned behavior, but it cannot guarantee representation of an arbitrary new dynamics family. Natural language expands the descriptor vocabulary without removing this identifiability and coverage requirement.

Negative transfer is possible. A source-trained initialization can underperform a target-only model under the same target-data and optimization budget. Include a target-only baseline and report the adaptation protocol, source retention, and uncertainty across seeds; otherwise a transfer claim has no controlled counterfactual.

Evaluation depends on the decision system. A lower open-loop prediction error need not imply a better controller, and no single planner-independent scalar captures every decision consequence. Report both diagnostic prediction metrics and closed-loop results with the actual planner, horizon, and cost. Cross-paper comparisons are fragile when those components differ.

Compute and data are not the only constraints. Scaling a world model requires not just GPUs and tokens but also access to diverse environments, reliable simulators, and the engineering infrastructure to run long experiments. The scaling curves hide the enormous cost of data collection and curation, which we covered in Part VI: Learning World Models at Scale. A team without access to this infrastructure can still benefit from the conceptual pipeline, but the specific numbers in any scaling law will not apply.

Stage-specific diagnosis. Pretraining can amortize representation learning across targets; post-training changes the objective; adaptation specializes the resulting model. This decomposition provides useful diagnostic questions: is the shared representation inadequate, is the conditioning signal uninformative, is the post-training objective misaligned, or is target data insufficient? The stages can interact, so the diagnosis still requires ablations and stage-specific evaluation.