Training and Inference Systems › Training and Inference Systems

A robot controller has 20 milliseconds to choose its next action. If its world-model query spends 25 milliseconds waiting in a queue, a fast forward pass cannot recover the missed deadline. That is an illustrative budget, not a benchmark of a particular robot, but it gives us a concrete systems problem: accurate predictions must arrive within the time and resource limits of their consumer. Up to this point we have talked about what world models are (Part I), how they represent state (Part III and Part IV), which architectures realize that representation (Part V), how they are learned (Part VI), and how they are used to plan and act (Part VII and Part VIII). Here we connect those ideas to training infrastructure, recoverable experiments and rollout serving. We will separate analytical cost models from local CPU measurements rather than infer deployment performance from model names.

The distinction between training and inference loops matters. Training advances through data and gradient updates. Autoregressive inference advances through predicted time steps, sometimes in lockstep with physical equipment. Both depend on numerical quality, but also on the machinery delivering the right inputs, state and outputs within a budget. A late accurate prediction and an inaccurate timely prediction are different failures; either can make a controller miss its requirements. Instrumenting both quality and timing lets us tell them apart.

A world-model training and inference stack includes the model's predict() call and the infrastructure around it. We will examine four components that need to work together:

• Training infrastructure that distributes gradient computation when needed, preserves optimizer state across updates, and saves snapshots for recovery.
• Data, checkpoint, and experiment infrastructure that move trajectory data and manifests between storage and compute, and record the artifacts needed to compare or reconstruct runs.
• Rollout serving that exposes the learned dynamics as a stateful, low-latency service, with the caching and batching strategies specific to autoregressive latent rollouts.
• Latency, memory, and energy optimization that turns an algorithm into something a robot, a browser, or a data center can run within its power and time budget.

These components are not independent. Their boundaries can introduce failures that component-only tests miss. Data availability can enter the training-step critical path. Checkpoint overhead and retention determine the recovery points available after a failure. The rollout server's cache footprint constrains concurrent sessions and feasible batches, which in turn affect response time. After changing one component, measure the whole path again: the limiting resource may have moved.

Consider three failures. A rollout service substitutes a default when overloaded, changing the training data without raising a model exception. A checkpoint mixes tensors from before and after an optimizer update, changing the resumed computation. A cache key omits episode identity and returns another episode's state. These failures can produce plausible numbers rather than an obvious crash. Explicit state identities, checksums, invariants and continuation tests can expose them; silence is a risk to test for, not an inevitable property of every failure.

This chapter assumes the model families from Part V, learning methods from Part VI, and decision-centric uses from Part VIII. We treat the model as a potentially expensive, stateful function and ask how to execute it correctly and within budget. Many techniques are shared with language and vision models. The workloads considered here can add long autoregressive rollouts, sensor timing and planning consumers that amplify some prediction errors. Those features are not universal to every world model, and a contractive transition can attenuate rather than amplify perturbations.

Let's begin with the training side.