Training and Inference Systems › Latency, Memory, and Energy Optimization

We now turn from correct model execution to three resource questions: response time, memory and energy. Their tradeoffs depend on the workload. Batching can change amortized compute cost while adding admission waiting; quantization can change storage and arithmetic while also changing predictions. Neither gives a fixed energy or latency factor without measurement. Choose settings against the deployment's actual quality and resource constraints, which differ between a data-center training job and a battery-powered robot.

Choose the timing boundary before decomposing latency. For an illustrative serial control loop, from the start of sensor acquisition to completion of the commanded actuator response, use disjoint stages:

$$\begin{aligned} T_{\text{loop}} ={}& T_{\text{capture}} + T_{\text{queue}} + T_{\text{transport}} + T_{\text{pre}} + T_{\text{model}} \\ &+ T_{\text{post}} + T_{\text{encode}} + T_{\text{decision}} + T_{\text{actuate}} \end{aligned}$$

where:

• $T_{\text{capture}}$: acquisition of the sensor data used for this decision.
• $T_{\text{queue}}$: all waiting for admission, batching and resources along the measured path, excluding active work counted elsewhere.
• $T_{\text{transport}}$: movement of inputs, outputs and commands between components, excluding their encoding and queue waiting.
• $T_{\text{pre}}$: input preparation such as resizing and normalization; learned encoder computation is counted under the model instead.
• $T_{\text{model}}$: the model computation, including any learned observation encoder and dynamics calls used for this decision.
• $T_{\text{post}}$: output interpretation or uncertainty calculations not already in the model.
• $T_{\text{encode}}$: serialization/deserialization of messages, not network transit.
• $T_{\text{decision}}$: remaining planner/controller work, excluding model calls already counted above.
• $T_{\text{actuate}}$: command processing and actuator response within the chosen endpoint.

This sum assumes serial stages without overlap. Pipelined capture, asynchronous transfers or overlapping computation require critical-path timestamp accounting, not addition of every raw service duration. Count an operation once even if software names put it in two components. Optimizing model time still reduces the serial sum, but may yield only a small relative improvement when other terms dominate. The local benchmarks below cover a much narrower model/bookkeeping boundary, not this complete loop.

In a synchronous one-decision-per-period design at 30 Hz, the period is about 33.3 ms, and the scheduled work needs margin inside it. Asynchronous control may use different observation-age and stability requirements. A browser's display interval and input-to-display latency are distinct targets: 60 Hz means about 16.7 ms between frames, not a universal 100 ms frame budget. Offline imagination may prioritize throughput while still having completion deadlines, memory limits or feedback-staleness constraints. Choose the metric and its boundary for the actual consumer.

Concrete techniques:

• Precision strategy. A tensor stored in fp16 or bf16 uses half the scalar bytes of fp32; that does not halve the entire footprint or guarantee a throughput factor. Long rollouts can amplify numerical perturbations in sensitive transitions, while contractions can attenuate them. Evaluate precision at one-step, rollout and decision levels on the target hardware. fp32 accumulation or a higher-precision reference may help diagnose an error, but periodic reference recomputation has its own cost and is not a generic cure. Quantization sensitivity depends on the operation and dynamics, not a universal transformer-versus-recurrence ordering.
• Kernel fusion. Compatible fused kernels can reduce launches and intermediate memory traffic while preserving the intended computation. Fusing execution is not the same as algebraically folding input-dependent normalization into a fixed linear weight matrix. Measure launch and memory costs on the target runtime, and validate numerical differences introduced by a replacement kernel.
• Sequence-length reduction. Dense full-prefix attention has quadratic arithmetic in context length; a cached one-token query still performs work linear in retained context, as discussed earlier. A fixed window or fixed-size recurrent state can bound per-step work at fixed model size. Chunking full-context attention bounds temporary working memory, not all pairwise arithmetic. Truncating context or changing architecture requires prediction and decision-quality validation.
• Speculative decoding. Exact language-model sampling with a draft model requires a specified acceptance and correction procedure preserving the target distribution, as in Leviathan, Kalman and Matias. A fast approximate world-model transition checked against another model or incoming sensors is a different heuristic; it does not automatically inherit that exactness. Define the property being checked, the correction or fallback and the remaining approximation error. Bounded approximations can be intentional controller designs, but a timing gain alone does not establish their acceptability.

Four useful tensor categories contribute memory pressure, before additional runtime overhead:

• Weights. Parameters and any additional network copies actually loaded for inference. Their fraction of the footprint depends on model size and the other terms, not batch size alone.
• Activations. Transient inference computation buffers; their sizes depend on the operation, batch and context. Ordinary inference does not retain backward activations.
• Session state. Full-context KV storage grows with retained tokens and concurrent sessions. A fixed-width recurrent state grows with concurrent sessions but not with rollout duration unless extra history is retained.
• Input/output buffers. These hold observations, actions, latent representations and predictions passed between encoders, dynamics models, decoders and policies. Multimodal inputs and outputs can make these buffers substantial.

Managing session state can be important, but profile the dominant term rather than assume it. For an explicitly disjoint tensor accounting, add weights, inference activations, retained session state and I/O, then account separately for runtime, allocator and workspace memory:

$$\begin{aligned} M_{\text{total}} &= M_{\text{weights}} + M_{\text{activations}} + M_{\text{sessions}} + M_{\text{io}} + M_{\text{overhead}} \end{aligned}$$

where:

• $M_{\text{weights}}$: bytes of loaded parameters and any required additional network copies
• $M_{\text{activations}}$: transient tensors for the current inference computation
• $M_{\text{sessions}}$: retained state for each session, with growth determined by its recurrent or attention representation
• $M_{\text{io}}$: input and output buffers for encoders, decoders, and policies
• $M_{\text{overhead}}$: other allocated or reserved memory not assigned to the preceding terms, including runtime and workspaces

Strategies include:

• Eviction policies. An evicted session can explicitly expire, with the next call returning an error, or it can be recovered by a defined protocol. Recovery may need a saved boundary state or observation context, subsequent actions, matching model/configuration and stochastic state; actions alone generally do not reconstruct it. LRU is only a selection policy, not a proof of recovery or request ordering. Silently replacing lost causal state with a default is the correctness hazard.
• State compression. Quantization or a low-rank representation can reduce payload bytes, but is generally lossy; omitted information cannot simply be recomputed from the compressed state. Under a quantified contraction with the same future inputs, an initial-state perturbation can decay, but ongoing compression injects new error and control constraints may still be sensitive to it. Measure reconstruction, rollout and decision effects before accepting the memory tradeoff.
• Sliding-window attention with action replay. When old observations become irrelevant, drop their KV entries but track the drop in a per-session ledger. Any subsequent request that references the dropped range is refused or reconstructed. This makes the memory/latency tradeoff explicit and auditable.

Length-aware grouping can change padding and allocation patterns. In an allocator requiring contiguous blocks, unusable free fragments can prevent an allocation even when total free bytes would suffice. Paged caches, preallocation and allocator policy can change that behavior. Monitor allocated, reserved and usable memory under changing session lengths; this CPU tutorial neither measures GPU fragmentation nor predicts an OOM time.

Energy needs its own measurement. Battery capacity limits total energy, while electrical and thermal constraints limit power; data centers can have hard power and cooling limits as well as cost budgets. For a declared measurement boundary, energy is $E=\int P(t)\,dt$, where power $P$ is in watts and elapsed time is in seconds, giving joules. Runtime alone does not determine it. Account for the actual model lifecycle:

• Training and adaptation may repeat. Model updates, imagined transitions and deployment queries all contribute; which dominates depends on their counts and measured costs, not whether the learner is called PPO or imagination-based.
• Rollout length multiplies serial work. At a constant 10 ms per step, 10,000 sequential steps take 100 seconds before other overhead. Both horizon and per-step implementation are design choices to evaluate against decision quality. Time is not yet energy without power measurement.
• Batching can amortize fixed overhead, but can also change active power, memory, waiting and unused work. Establish joules per useful inference from measured power and time, and test mobile deadlines and thermal limits rather than assume batching is unambiguously beneficial.

The common optimizations are:

• Precision choices, assessed for numerical quality and measured energy on the target hardware.
• Kernel fusion and reduced memory traffic, assessed with power instrumentation rather than treating bytes moved as an energy meter.
• Avoiding redundant evaluations. Sampling-based planners may score many candidates. Valid repeated-prefix reuse can avoid computation when lookup and storage costs are lower; independent stochastic samples must retain their distinct sample identities. Pruning can reduce work but also change planning quality.

Energy is observable during development using a suitable meter or validated device counters and a declared scope. Measure idle and active intervals consistently, and include the components relevant to the decision, not only a convenient accelerator kernel. Energy per inference and per useful action answer different questions from average power; record the metrics the deployment budget actually constrains.

We measure a small CPU workload at several vectorized batch sizes. Each row times a different number of independent states advanced by one forward pass. Dividing that time by batch size gives amortized compute time per state, not end-to-end request latency. These timings vary with the CPU, runtime, thread settings and system load; they are not GPU throughput or energy measurements. Energy requires measured power integrated over time, as in the MLCommons power measurement rules.

Compare the actual table: batching may reduce amortized compute time even when a whole batch takes longer. The experiment does not simulate an arrival queue or establish a universal optimal batch size. A deployment must jointly measure throughput and end-to-end latency at its own request sizes and load.

The next experiment isolates history-buffer bookkeeping during a recursive rollout. Both models use only the current state and action, so trimming the stored history cannot change predictions here. A transformer that attends to its history is different: truncating its cache may change the prediction distribution.