Model Serving › Model Serving

Training a language model is only half the work. The other half is getting it into the hands of users reliably, at scale, and within latency budgets that real applications demand. Model serving is the discipline of taking a trained model and turning it into a running service that handles requests from clients, manages computational resources, and behaves predictably under load.

The challenges of serving language models differ substantially from serving traditional machine learning models. A sentiment classifier produces one prediction per input in a fixed amount of compute. A language model generates tokens one at a time in a loop, making each request variable in length, variable in cost, and fundamentally sequential in its output. A request for a two-sentence summary consumes far less GPU time and memory than a request for a detailed report, yet both enter the same request queue and compete for the same hardware. Serving infrastructure must account for this variability at every layer.

This variability drives the design of every major serving technique. When requests have predictable, uniform cost, you can plan capacity with simple arithmetic. When each request has an unknown compute cost, you need adaptive queuing, batching, and memory management to keep the GPU busy without exhausting memory or exceeding latency budgets.

This chapter covers the full stack of model serving: loading and initializing models in memory, accepting and routing requests, configuring serving parameters, and the frameworks that tie it all together. As we saw in Part XXXI on training infrastructure, getting software and hardware to work together efficiently requires careful configuration at every layer. Serving is no different, but the optimization objectives shift from training throughput to inference latency and concurrency.