Pretraining, Post-Training, Scaling, and Adaptation › Key Parameters
Michael BrenndoerferJuly 6, 20261 min read
World Models HandbookAll chapters Part 6 of 12Learning World Models at Scale
Pretraining, Post-Training, Scaling, and Adaptation › Key ParametersOne section · Sections in this chapter8 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The key parameters for the task-conditioned dynamics model and its adaptation are:
- emb_dim: Dimensionality of each task embedding. Larger embeddings can describe more distinct dynamics regimes but add parameters and increase overfitting risk on small adaptation datasets.
- hidden: Width of the hidden layers in the dynamics network. This sets the capacity of the shared dynamics representation that all tasks reuse.
- epochs: Number of optimization passes over the training data. More passes can reduce training loss, but held-out error may worsen on a small target dataset, so select this value with validation data.
- lr: Learning rate for Adam. The example tunes different values for each stage; dataset size alone does not determine which stage should use the larger rate.
- rank and alpha (LoRA): The rank sets how many trainable parameters the low-rank update adds, and alpha scales that update relative to the frozen weights.
- beta (DPO): The temperature-like parameter that controls the strength of the implicit KL penalty keeping the policy near the reference model.