Pretraining, Post-Training, Scaling, and Adaptation › Preference and control-oriented post-training

Pretraining gives you a model that predicts the world. Post-training gives you a model that predicts the world in a way that is useful. The difference is subtle but has practical consequences. A pretrained model optimizes average prediction error. A post-trained model optimizes whatever the downstream decision-maker cares about. These are rarely the same objective, and the gap between them is what post-training exists to close.

Consider a world model used inside a model predictive controller. The controller samples action sequences, rolls them out under the model, and picks the sequence that minimizes a cost. The model must rank candidate sequences well near the optimum, not merely minimize average one-step error. Correlated model bias is especially dangerous because an optimizer can repeatedly exploit it. Zero-mean noise may average out under repeated sampling, but it can still harm ranking when samples are few or costs are nonlinear.

This is the same mismatch that motivates the decision-aware learning literature and the model-based RL objectives we discussed in Part II: Inference, Decisions, and Control. The point here is that post-training is the stage where you close that mismatch, by optimizing the model against a decision-relevant signal. Recognizing which quantity the downstream controller is sensitive to, and shaping the model's loss accordingly, is the central design question of post-training. Sometimes this means reweighting the loss; sometimes it means adding an entirely new term; sometimes it means training against a preference signal that is only available through comparison. All of these techniques are ways of pushing the model's errors into regions the planner does not care about.

A post-training pipeline can begin with supervised fine-tuning (SFT) on curated data. For a world model, SFT data might be:

• High-quality trajectories from expert demonstrations.
• Transitions from tasks that resemble the target deployment.
• Synthetic rollouts from a trusted simulator.
• Instruction-conditioned examples, where the model learns to map a task description to the right dynamics.

Plain output-likelihood SFT teaches the conditional distribution represented by its examples. It does not explicitly express a pairwise preference between two plausible outputs unless the training format includes that comparison. Preference data adds this supervision, although curated demonstrations can encode some of the same behavior indirectly.

To teach preferences, one pipeline is reinforcement learning from human feedback (RLHF), demonstrated for complex behaviors by Christiano et al. (2017) and later applied to instruction-following language models by Ouyang et al. (2022). The pipeline has three stages.

1. Collect comparisons. Show a human two candidate model outputs (two rollouts, two predicted futures, two action sequences) and ask which is better.
2. Fit a reward model. Train a scalar function $r_\phi(x, y)$ that predicts the human preference.
3. Optimize the policy against the reward model. Use reinforcement learning (typically PPO) to update the model so it produces outputs the reward model likes.

For a generative world model, compared outputs could be trajectories or predicted futures, and “better” must be defined for the deployment. Human comparisons can supply that label. Automatic signals such as task success, controller cost, or constraint violations instead create proxy-feedback preference data; they are not human feedback merely because the same optimization template is used. Either source can be misspecified, so held-out closed-loop evaluation remains necessary.

RLHF requires fitting a reward model and then running an RL optimization loop. Direct Preference Optimization (DPO, Rafailov et al. 2023) shows that, under its reward-model and KL-regularized policy assumptions, you can optimize the corresponding preference objective without fitting an explicit reward model.

Given a preferred output $y_w$ and a rejected output $y_l$ for the same input $c$, and a reference model $\pi_{\text{ref}}$ (usually the SFT model), the DPO loss is

$$\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(c, y_w, y_l)} \left[ \log \sigma \left( \beta \left( \log \frac{\pi_\theta(y_w \mid c)}{\pi_{\text{ref}}(y_w \mid c)} - \log \frac{\pi_\theta(y_l \mid c)}{\pi_{\text{ref}}(y_l \mid c)} \right) \right) \right]$$

where:

• $\mathcal{L}_{\text{DPO}}(\theta)$: the DPO loss for policy parameters $\theta$
• $\theta$: the parameters of the policy being trained
• $\mathbb{E}_{(c, y_w, y_l)}$: the expectation under the data distribution over conditioning input $c$, preferred output $y_w$, and rejected output $y_l$
• $c$: the conditioning input (for example, a task description or state history)
• $y_w$: the preferred (winning) output
• $y_l$: the rejected (losing) output
• $\sigma$: the logistic sigmoid function
• $\beta$: a temperature-like parameter controlling the strength of the implicit KL penalty
• $\pi_\theta(y \mid c)$: the probability of output $y$ under the current policy given conditioning input $c$
• $\pi_{\text{ref}}(y \mid c)$: the probability of output $y$ under the frozen reference model given conditioning input $c$

The ratio $\pi_\theta(y_w \mid c)/\pi_{\text{ref}}(y_w \mid c)$ compares how much more the current model prefers the winning output than the reference does. The same ratio for $y_l$ does this for the losing output. The difference between these log-ratios is scaled by $\beta$ and passed through a sigmoid, so the loss is small when the model already prefers $y_w$.

The intuition is that DPO increases the relative log-probability of preferred outputs over rejected ones, while a KL-like penalty (implicit in the reference model) keeps the policy from drifting too far from the pretrained model. The parameter $\beta$ controls the strength of that penalty. A small $\beta$ imposes a weak implicit KL penalty and lets the model separate winning from losing outputs more aggressively; a large $\beta$ imposes a stronger penalty and keeps the policy closer to the reference model. In practice, $\beta$ is one of the most consequential hyperparameters in a DPO training run, and it is worth tuning as carefully as the learning rate.

DPO avoids a separately fitted reward model and can be implemented with a supervised-style loop. Applying it to world-model outputs requires a normalized conditional likelihood, or an explicitly justified surrogate, for each compared sequence. Preference pairs might come from expert and baseline rollouts or human comparisons; how they are labeled determines what the model will learn.

The most direct form of post-training is to optimize the model against the actual control objective. Several variants exist.

• Value-aware training. Train the model so that the value function computed from its rollouts matches the true value. This makes the model useful for policy optimization even if its per-step predictions are imperfect.
• Cost-aware training. Weight the prediction loss by the sensitivity of the downstream cost to prediction error. Regions where the planner is sensitive get more weight.
• Adversarial and robust training. Train the model to be accurate on the worst-case action sequence, not the average one. This reduces model exploitation, where the planner finds a sequence that looks good under the model but fails in reality.
• Constraint-aware training. Penalize predictions that violate known physical or safety constraints, even when the data does not contain such violations.

All of these objectives treat the model as a component of a decision system. Closed-loop performance is therefore the primary test of decision usefulness, while open-loop prediction metrics remain valuable diagnostics for calibration, coverage, and failure analysis. A model with worse average prediction error can still produce better decisions if its errors are better aligned with the planner's sensitivities.

A final distinction that matters for post-training is process versus outcome supervision.

• Outcome supervision labels the final result. It can be easy or difficult to collect depending on the task, and it leaves ambiguity about which intermediate decisions caused the outcome.
• Process supervision labels intermediate predictions or decisions. When reliable intermediate targets are observable, it can provide denser credit assignment, but collecting or validating those targets may be costly.

For world models, an observed transition supplies a next-observation target, so logged trajectories can provide dense supervised signals. Those signals are not free: interaction may be expensive, sensors may be partial or noisy, and the observed next state need not reveal the latent state a planner requires. Dense one-step supervision also does not remove the evaluation mismatch: a model can predict one step well and still accumulate harmful rollout error.