Pretraining, Post-Training, Scaling, and Adaptation › Scaling data, parameters, context, and compute

For autoregressive language models studied within a fixed model and data family, held-out loss often follows a smooth empirical trend with the number of model parameters $N$ and training tokens $D$. This is not a theorem, and the fitted law does not automatically transfer to a different architecture, modality, or world-model objective. For world models, it is best treated as a budget-planning hypothesis to test with a small scaling ladder.

If a fitted law predicts loss within its measured range, it can compare candidate runs before paying for each full experiment. For example, it can test whether a fixed FLOP budget favors more parameters or more data. The fit makes the assumed tradeoff explicit; short runs near the proposed allocation must still verify it.

A useful form, closely related to the empirical language-model scaling laws studied by Kaplan et al. (2020), is a sum of power-law terms:

$$L(N, D) = \left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D} + L_\infty$$

where:

• $N$: the number of model parameters
• $D$: the number of training tokens (or transitions)
• $N_c$: a critical model size that sets the scale of the parameter term
• $D_c$: a critical dataset size that sets the scale of the data term
• $\alpha_N$: the exponent governing how quickly the parameter term decays as the model grows
• $\alpha_D$: the exponent governing how quickly the data term decays as the dataset grows
• $L_\infty$: the irreducible loss floor

Each term has a meaning.

• The parameter term $\left(N_c/N\right)^{\alpha_N}$ is the fitted contribution that shrinks as the model grows. It may reflect limited model capacity, but the empirical decomposition does not identify a unique causal source of error.
• The data term $\left(D_c/D\right)^{\alpha_D}$ is the fitted contribution that shrinks as the dataset grows. It may reflect finite-data effects, including estimation error and incomplete coverage.
• The floor $L_\infty$ is the asymptote of this fitted model. Its value depends on the data distribution, representation, model family, optimizer, and loss; it is not necessarily an intrinsic constant of the environment.

The separation is a diagnostic within the fitted regime. If the parameter-dependent contribution dominates, a larger model is one candidate intervention. If the data-dependent contribution dominates, more or better-covered data is one candidate. If measured loss is close to the fitted floor, simply extending the same scaling path is predicted to yield little improvement; changing the representation, data distribution, model family, or loss may move that floor. These are hypotheses for follow-up experiments, not causal conclusions supplied by the curve fit.

In Kaplan et al.'s fitted form, $\alpha_N \approx 0.076$ and $\alpha_D \approx 0.095$. Exponents of that size imply slow improvement in each isolated term: a tenfold increase in the associated resource reduces that term by about 10 to 20%, and halving it requires a thousandfold to millionfold increase. Other fits use different definitions and exponents, so compare the predicted loss change rather than the raw exponent alone.

The sum-of-power-laws form makes the two fitted contributions trade off. For a fixed compute budget $C \approx 6 N D$ (a common approximation for dense transformer training), you can choose a larger model trained on fewer tokens or a smaller model trained on more. Hoffmann et al. (2022) estimated approximately equal scaling of model size and training tokens in their studied language-model regime. Their 70-billion-parameter Chinchilla model used 1.4 trillion tokens, a ratio of about 20 tokens per parameter. That ratio describes one empirical regime and should not be treated as a universal constant; “too much” or “too little” data is meaningful only relative to a fixed compute budget and a validated scaling fit.

The Chinchilla result changed how language-model researchers reason about fixed-compute allocation: in those experiments, rebalancing compute toward more training tokens produced a smaller model with better loss than larger under-trained alternatives. Its durable lesson is methodological rather than numerical. Estimate the balance point for the architecture, data, and objective at hand instead of importing a token ratio from another setting.

For world models, the analogue of “tokens per parameter” is not obvious. A transition $(s_t, a_t, s_{t+1})$ has different structure from a text token, adjacent transitions are dependent, and interaction data may be costly to collect. Here $s_t$ is the state at time $t$, $a_t$ is the action, and $s_{t+1}$ is the next state. Collection cost and training FLOPs are separate constraints: an expensive transition does not by itself change the compute optimum, but it can make the data-optimal point infeasible. Applying scaling ideas responsibly therefore requires a domain-specific definition of $D$, a coverage measure, and an explicit collection budget.

The scaling picture for world models differs from language in several important ways.

Sequential dependence changes the meaning of data scale. Both text and interaction corpora contain dependent observations, but trajectories can exhibit especially strong temporal correlation and policy-dependent coverage. Adjacent transitions may add less information than transitions from distinct states, tasks, or embodiments. Consequently, raw transition count is an incomplete proxy for effective dataset size. A useful scaling study should report $D$ together with trajectory diversity, state-action coverage, and the data-collection policy.

Context length is a separate experimental axis. A longer history can expose information needed for partially observed or delayed dynamics, but it also changes memory cost, optimization, and the distribution of training windows. More context helps only when the additional history is informative and the model and dataset support it. For that reason, context length should be varied and measured alongside $N$ and $D$, rather than assumed to translate directly into a particular planning horizon.

Action and embodiment coverage are additional axes. Beyond $N$, $D$, and $C$, a world model must cover relevant action spaces, embodiments, and dynamics regimes. Diverse pretraining can improve transfer when the target shares structure with the source mixture, although diversity is not a substitute for target-relevant coverage. Scaling studies should therefore distinguish the number of transitions from the breadth and balance of environments that generated them.

Stochasticity and multimodality matter. Next-token prediction and next-state prediction can both be multimodal. For a proper probabilistic loss, irreducible uncertainty under the chosen observations and targets contributes to the attainable loss; for point losses, multimodality can instead produce averaged predictions. A fitted $L_\infty$ also absorbs representation and model misspecification. Distributional prediction, better observations, or a different target can therefore change the measured floor.

Scaling curves are usually plotted on log-log axes, where a single power-law term appears linear. Thresholded capability metrics can nevertheless show step-like jumps. For example, if a planning benchmark awards credit only after a rollout crosses a fixed success threshold, a smooth change in predictive quality can look discontinuous. Such a plot is evidence about the metric and experiment, not proof that planning ability itself appeared discontinuously.

The emergence picture is contested. Schaeffer et al. (2023) showed that some reported jumps can arise from nonlinear or discontinuous metrics applied to smoothly improving models. That result does not rule out every discontinuity. For world models, test both continuous quantities and task success across several scales before assigning a mechanism to an apparent threshold.

In practice, a pretraining run is constrained by a compute budget $C$ and a wall-clock budget. The standard prescription is:

• Estimate the scaling exponents on a small set of runs (the "scaling ladder").
• Fit the exponents $\alpha_N$, $\alpha_D$, $L_\infty$, $N_c$, $D_c$.
• Choose $N$ and $D$ to minimize the predicted loss subject to $6 N D \le C$.
• Spend the remaining budget on a longer run at the chosen $(N, D)$.

The fitted exponents are only trustworthy within the range of the ladder. Extrapolating a power law fitted on $10^6$ to $10^8$ parameters out to $10^{11}$ parameters is an act of faith, not a calculation. Still, the exercise is useful because it forces you to state your assumptions explicitly and to notice when a proposed training run is far from the predicted frontier. Beyond that, the process is a mechanism for organizing disagreement. When two engineers propose different training runs, they can point at the fitted law and ask which run is closer to the frontier and which assumption is being challenged. That conversational clarity, more than the numerical prediction, is often the real return on investing in a scaling ladder before committing to a large run.