Training and Inference Systems › Limitations and Impact

The techniques in this chapter are powerful, and they also have hard limits. If a fixed model class cannot represent the needed dynamics, distributing its training does not remove that structural limitation. This differs from inaccurate fitted parameters that further training can improve. A cache that serves the wrong episode's state can also produce fast but incorrect rollouts. Systems infrastructure enables hypotheses to be tested; it does not validate them. A frequently overlooked consequence is that systems bugs tend to hide in the same corridors that reliability metrics use. If rollout latency grows when the queue is full, a poorly instrumented system may interpret the slower response as a longer-horizon measurement, not a delay. If a cache returns a stale latent, the rollout's apparent smoothness may look better than the actual dynamics. Tests must therefore be structured to probe the systems layer as explicitly as they probe the model. In particular, a well-designed system distinguishes model failures from infrastructure failures, and only then lets the evaluation ladder from The World-Model Evaluation Ladder be applied.

Distributed training introduces its own hazards. Incorrect loss normalization for unequal worker batches, mismatched optimizer state or incompatible numerical settings can compromise updates. Independent local sampling seeds alone do not make correctly synchronized replicas diverge. A slower rank can delay exposed synchronization; its percentage slowdown is not necessarily the whole job's percentage slowdown because overlap and other work matter. Per-rank timing, valid-token counts, losses and gradient norms make these effects observable. An aggregate loss can hide one faulty contribution, so inspect disaggregated evidence rather than assume every rank is healthy.

Checkpoint bugs include fresh optimizer moments on resume, repeated or skipped windows and missing auxiliary learning state. Define the state and test uninterrupted training against checkpointed continuation, comparing the next losses, parameter tensors and relevant optimizer or sampler state under a fixed deterministic configuration. Passing that test covers the exercised configuration only. Distributed shards, interruptions during writes, changed worker counts and device RNG state need additional tests. A mismatch calls for diagnosis: an incomplete snapshot and allowed numerical nondeterminism are different causes.

Three serving failures deserve explicit tests. First, non-preemptive batch work can block interactive requests sharing capacity. Protect their deadlines with admission control and an appropriate scheduling or reservation policy; separate queues are one implementation, not the only one. Second, a changed model, configuration or sampling policy must not reuse obsolete cache identities. Versioned keys can make old entries unreachable without physically deleting every response, while fresh-sample requests must not accidentally reuse a previous draw. Third, retained session state needs an owner and release policy: explicit close, expiry or eviction can prevent orphaned state from accumulating without bound. Test these lifecycle transitions and their errors as well as successful forward calls.

Measure energy per useful action as well as model throughput. If a planner evaluates thousands of candidates to choose one action, its decision energy includes every executed evaluation, not just the selected rollout. Valid prefix reuse or tested pruning can reduce discarded computation after accounting for overhead and planning-quality changes. This connects systems measurements to the algorithmic budgets discussed in Sampling-Based Planning and Model Predictive Control and Policies Learned in Imagination.

Finally, the biggest limitation is that systems quality is not transferable across model families without care. A serving stack optimized for a transformer world model with KV caches does not translate directly to a recurrent world model with hidden state. Variable-length episodes can supply fixed-length training windows. A training stack using those windows still needs correct temporal associations and episode-boundary handling; padding with appropriate masks is another design option. The general principles hold (distributed, cached, batched, latency-tuned), but the recipes are family-specific and frequently subtle. What works today for one flagship architecture is a starting point, not a finished design, for the next one.

The systems layer also supplies evidence for governance: dataset provenance, access logs, checkpoint retention and reproducible experiment records help an organization audit what it trained and deployed. Specific legal obligations depend on jurisdiction and use, and are outside this chapter's systems tutorial. Security, Ethics, Privacy, and Governance covers those governance concerns. The planned next chapter, Sim-to-Real Deployment and Continual Correction, considers noisy sensors, feedback delays and deployment shifts; it is not yet published.