LLM Inference Serving: Architecture, Routing & Auto-Scaling › Inference Server Architecture

An inference server sits between client applications and the underlying model. It manages requests and batches them for scheduling, then streams the generated responses. Modern LLM inference servers have evolved specialized architectures to address the unique challenges of autoregressive generation. Understanding these architectural patterns is needed for anyone building or operating production language model services, because the choices made at this layer determine the performance ceiling for everything built on top.

Think of the inference server as the operating system for your model. Just as an operating system multiplexes CPU time across multiple processes, allocates memory across competing programs, and provides a unified interface to hardware, the inference server multiplexes GPU compute across requests, manages the KV cache across concurrent sessions, and presents a clean HTTP/gRPC interface to applications. The analogy extends further: just as operating systems evolved from simple batch schedulers to sophisticated time-sharing systems that could support many concurrent users, LLM inference servers have evolved from simple request queues to continuous batching systems that can fluidly interleave prefill and decode phases across dozens of concurrent requests.

The design of an LLM inference server differs fundamentally from a traditional REST API server. A traditional API server is stateless: each request arrives, executes a function, returns a result, and frees all associated memory. An LLM server is stateful during generation: each request allocates a growing block of KV cache memory that persists for the duration of token generation. A request generating 500 tokens holds memory for the entire duration, unlike a database query that returns immediately. This statefulness requires the inference server to actively manage memory, schedule when requests begin processing, and decide when to preempt long-running requests to make room for new arrivals.

Another key difference is that LLM requests are not uniform in cost. A request with a 4000-token prompt requires 40x more prefill computation than a 100-token prompt, and a request generating 500 tokens requires 500 individual decode steps while one generating 10 tokens requires only 10. A server that treats all requests as having equal cost (as naive implementations do) will chronically misallocate resources, sometimes starving short requests behind long ones or saturating one endpoint while others sit idle. The sophisticated scheduling and routing discussed throughout this chapter exists precisely to handle this non-uniformity.

The architecture of an LLM inference server differs significantly from traditional ML serving systems in basic ways that stem from the nature of autoregressive text generation. Classification models process requests independently with fixed compute costs: an image classifier performs the same computation regardless of the input image's content. Language models, by contrast, generate tokens iteratively, with each request potentially requiring hundreds of forward passes and memory that grows with sequence length. This iterative, variable-cost nature demands specialized architectural components that can efficiently manage the unique resource patterns of text generation.

A typical inference server contains several interconnected components, each playing a distinct role in the request processing pipeline:

• Request handler: Accepts incoming requests over HTTP/gRPC, validates inputs, and manages response streaming. For LLMs, this often involves Server-Sent Events (SSE) or WebSocket connections to stream tokens as they're generated. The request handler must maintain long-lived connections efficiently, as a single request may take several seconds to complete while tokens are progressively returned to the client.

• Tokenizer: Converts text to token IDs on input and back to text on output. While fast compared to model inference, tokenization can become a bottleneck at high throughput because it runs on the CPU while the model runs on the GPU. Servers often parallelize tokenization across CPU cores to prevent this component from limiting overall system throughput.

• Scheduler: Determines which requests to process together in each batch. As we discussed in the previous chapter on continuous batching, sophisticated schedulers dynamically add and remove requests from running batches. The scheduler must balance multiple objectives: maximizing GPU utilization, maintaining fair request ordering, and meeting latency targets for high-priority requests.

• Model executor: Manages the actual model inference, including memory allocation for KV caches, attention computation, and integration with optimized kernels like FlashAttention. This component coordinates the GPU computation and ensures that the model weights, activations, and cached attention states are efficiently arranged in memory.

• Memory manager: Tracks KV cache allocations, implements paged attention for efficient memory use, and handles memory pressure through eviction policies. Because KV cache memory requirements grow with sequence length and concurrent request count, intelligent memory management is necessary for maintaining high throughput without running out of GPU memory.

Several frameworks have emerged to handle LLM serving at scale, each with different strengths and design philosophies. Understanding the distinctions helps you choose the right tool for your deployment context, because switching frameworks later requires significant re-engineering effort.

vLLM implements PagedAttention for efficient KV cache management, letting high throughput with memory-constrained hardware. By treating KV cache memory like virtual memory pages, vLLM can handle more concurrent requests than naive implementations that allocate contiguous memory blocks. It supports continuous batching and integrates with popular model architectures, which makes it a popular choice for production deployments. Its open-source nature and active community mean that support for new models and hardware typically arrives quickly.

Text Generation Inference (TGI) from Hugging Face provides production-ready serving with tensor parallelism for multi-GPU deployments, quantization support, and optimized attention implementations. Its tight integration with the Hugging Face ecosystem makes it particularly convenient for teams already using Transformers for model development. TGI handles much of the operational complexity automatically, from model loading to request queuing, letting teams to reach production quickly.

TensorRT-LLM from NVIDIA offers highly optimized inference for NVIDIA GPUs, with custom CUDA kernels and support for quantization schemes like FP8 and INT4. When running on NVIDIA hardware, TensorRT-LLM often achieves the highest raw performance, though it requires more effort to set up than framework-agnostic alternatives. The performance gains are most pronounced for latency-necessary applications where every millisecond matters.

Triton Inference Server provides a model-agnostic serving platform that can host multiple models with different backends, useful for ensemble architectures or multi-model deployments. Its flexibility makes it well-suited for complex inference pipelines that combine multiple models or processing steps. For organizations running diverse model portfolios, Triton's unified interface reduces operational overhead.

The configuration balances memory allocation against concurrency in a careful trade-off. Higher gpu_memory_utilization allows more KV cache space for concurrent requests, but leaves less headroom for activation memory during computation. Setting this value too high can cause out-of-memory errors during prefill of long sequences; setting it too low wastes expensive GPU memory that could support additional concurrent requests. Most practitioners find values between 0.85 and 0.92 work well, with lower values giving a safety margin for unexpected memory spikes.

The key parameters for inference server configuration are:

• gpu_memory_utilization: Fraction of GPU memory to reserve for the model weights and KV cache. The remaining fraction is a safety margin for activation memory during computation and other runtime overheads.
• max_num_seqs: Maximum number of concurrent sequences the server can handle. Requests beyond this limit queue until a sequence slot becomes available.
• enable_chunked_prefill: Whether to split the prefill phase into chunks to prevent blocking the decode phase of other requests, letting interleaved progress across multiple active requests.

Understanding the journey of a request through the server helps identify optimization opportunities and debug performance issues. Each stage presents different bottlenecks and optimization levers. Think of the request lifecycle as a relay race: the baton (request) passes through multiple stages, and the slowest leg determines the overall time. Profiling which stage is slowest tells you exactly where to focus optimization effort.

1. Arrival: The request arrives via HTTP POST with prompt text and generation parameters (temperature, max tokens, etc.). The server validates the request format and assigns it a unique identifier for tracking.

2. Preprocessing: The server tokenizes the prompt and validates that the resulting sequence fits within model limits. Long prompts may be rejected or truncated depending on server configuration.

3. Scheduling: The scheduler decides when to begin processing. The request may queue if all batch slots are occupied. Priority and fairness policies determine ordering among waiting requests.

4. Prefill: The model processes all prompt tokens in parallel, populating the KV cache with attention states for all input positions. This phase is compute-bound because it performs dense matrix multiplications across the entire prompt length.

5. Decode Loop: Tokens are generated one at a time, with each new token streamed to the client. This phase is memory-bandwidth-bound because each forward pass must read the entire KV cache from GPU memory while creating only a single output token.

6. Completion: Generation stops when the model produces an end token or reaches the maximum length. The server releases KV cache memory and closes the response stream, making capacity available for new requests.

The key insight is that the prefill and decode phases have completely different performance profiles. Prefill is a matrix multiply problem where throughput scales with hardware compute capability. Decode is a memory-read problem where throughput scales with hardware memory bandwidth. This means that techniques that improve prefill speed (like tensor parallelism) may not help decode speed at all, and optimizations that target one phase can interfere with the other when they share GPU resources.