Model Serving › Summary

Model serving turns a trained model into a production service. The key concepts from this chapter are:

• Model loading involves copying weights from disk to GPU memory, with large models requiring tensor parallelism across multiple GPUs. Cold start latency is high and must be managed through proactive capacity planning.

• Memory budgeting requires accounting for model weights, activations, and KV cache. KV cache consumption grows with both sequence length and concurrent request count, making careful configuration essential. PagedAttention eliminates fragmentation and dramatically improves KV cache efficiency.

• Request handling encompasses queuing policies (FCFS, SJF, priority), batching strategies (static, dynamic, continuous), and rate limiting to protect server capacity. Continuous batching provides the highest GPU utilization for variable-length generation workloads.

• The prefill-decode asymmetry shapes every serving design decision. Prefill is compute-bound and spikes GPU utilization; decode is memory-bandwidth-bound and generates one token at a time. Chunked prefill and disaggregated serving architectures address this asymmetry by preventing long prefills from disrupting decode latency.

• Serving frameworks like vLLM, TGI, and TensorRT-LLM provide production-ready implementations of these concepts with different trade-offs in performance, flexibility, and ease of use.

• Serving configuration requires explicitly trading throughput for latency based on application requirements. Key parameters include gpu_memory_utilization, max_num_seqs, batch size limits, and chunked prefill settings. Tracking latency as a distribution (p50, p95, p99) rather than an average reveals tail behavior that average metrics hide.

• Deployment patterns such as canary rollouts and blue-green deployments provide safe paths for model updates without service interruption. Autoscaling for LLMs requires proactive capacity management because cold start times are too long for reactive scaling.

The next chapter covers latency optimization techniques specific to serving infrastructure: profiling request latency, reducing time-to-first-token through prefill optimization, and tuning decode throughput.