Training and Inference Systems › Summary

Training and inference systems turn a world model from a mathematical object into something that behaves as a system under real constraints. The key ideas in this chapter:

• Distributed-training axes include data (batch), tensor (layer operations), pipeline (depth) and context partitioning. Megatron's selected-activation sequence parallelism is distinct from context parallelism. Large models can combine axes; choose the mixture from memory constraints, interconnect and the actual workload.
• Temporal partitioning is architecture-specific, not unique to world models. Ring exchange distributes attention over known context. Efficient recurrent scans require suitable compact associative transition structure; an arbitrary nonlinear recurrence still needs valid boundary states. Batching independent trajectories, checkpointing, recomputation or explicitly approximate training remain alternatives.
• Data, checkpoint and experiment records support reproducible continuation and interpretable comparisons. Capture the algorithm's actual parameters, optimizer, sampler, random and auxiliary state; an EMA is required only if the algorithm uses it. Distinguish atomic publication from durable storage, and test recovery under the intended configuration.
• Rollout serving carries causal and stochastic state. KV caches, session lifetimes and cache-aware batching need explicit ownership. Result-cache identity includes model version, initial state, action prefix, configuration and any reproducible RNG stream identity; fresh independent samples must not accidentally reuse a cached draw.
• Latency, memory and energy require distinct measurement boundaries. A serial control-loop budget includes capture, queues, transport, input preparation, model work, output handling, encoding, decision and actuation; overlapping paths need critical-path accounting. Memory includes tensor categories plus runtime overhead, and fixed recurrent state differs from growing KV history. Energy per inference, per useful action and average power answer different budget questions.
• Measure throughput and end-to-end latency together. A local batch microbenchmark need not show a knee and does not measure queue delay or energy. Choose settings against actual workload, memory and deadline constraints.

The systems layer supplies tests and records needed to assess reliability; it does not by itself certify prediction quality, safety or hard deadlines. The next chapter's deployment questions require checking these mechanisms against actual sensors, delays and shifts, not only the controlled CPU examples here.