Pretraining, Post-Training, Scaling, and Adaptation › Worked Example: Pretraining and Adapting a Family of Dynamics Models

Let's make the pipeline concrete with a small example. We will work with a family of one-dimensional driven damped point masses. Each "embodiment" is defined by a mass $m$ and a damping coefficient $b$, and the state is a two-vector $(x, v)$ with position and velocity. The transition, using semi-implicit Euler with step size $\Delta t$, is

$$\begin{aligned} v_{t+1} &= v_t + \Delta t \left(-\frac{b}{m} v_t + \frac{a_t}{m}\right), \\ x_{t+1} &= x_t + \Delta t\, v_{t+1} \end{aligned}$$

where:

• $x_t$: the position at time step $t$
• $v_t$: the velocity at time step $t$
• $a_t$: the action (force) applied at time step $t$
• $m$: the mass of the point mass
• $b$: the damping coefficient
• $\Delta t$: the integration step size
• $v_{t+1}$: the velocity after the update
• $x_{t+1}$: the position after the update

Different $(m, b)$ pairs behave differently. In the absence of applied force, velocity decays at a rate set by $b/m$. Because the update contains no restoring force proportional to position, these systems are not oscillators. Our goal is to build a single model that can predict the family and then adapt it to a new parameter pair from a small target dataset.

The pipeline has three stages.

1. Pretrain a task-conditioned model on four embodiments. The model sees the state, the action, and a task identifier, and predicts the next state.
2. Adapt to a fifth, held-out embodiment using a small number of trajectories. We compare three adaptation strategies: doing nothing (zero-shot, using the mean of the training task embeddings), training only a new task embedding while freezing the network, and full fine-tuning.
3. Compare against training a fresh model from scratch on the same small amount of target data.

This example is small enough to run in a few seconds on a CPU, but it captures the essential tradeoffs. The pretrained model has seen many dynamics regimes, so its internal representation is already close to what the new embodiment needs. The from-scratch model has to learn everything from a handful of samples, and it will do poorly. The embedding-only adaptation gets most of the benefit at a fraction of the parameter cost of full fine-tuning.

The scaling-law experiment from the next section gives you the tools to reason about how this picture changes as the pretraining dataset grows. The DPO example at the end shows how to shape the model's outputs with a preference signal, which is the post-training half of the story.