LLM Inference Serving: Architecture, Routing & Auto-Scaling

Deploying a language model involves much more than loading weights and running forward passes. Production systems must handle thousands of concurrent users, maintain low latency under varying load, and do so cost-effectively. Inference serving covers the system architecture and request-routing choices needed to scale a trained model into a reliable, performant service. This chapter explores the full infrastructure stack that sits between a trained model and the users who interact with it.

Think of inference serving as the logistics layer of a large restaurant kitchen. The chef (the model) has extraordinary skill, but without waitstaff to take orders, a host to manage seating, and a manager to schedule staffing based on expected dinner rush, the kitchen never serves customers efficiently. The food might be excellent, but poor front-of-house operations create bottlenecks, long waits, and unhappy guests. Inference serving is that front-of-house operation: the systems that accept requests, dispatch them to the right model instance at the right time, and return responses reliably at scale.

The previous chapters in this part covered techniques that make individual inference requests faster and more memory-efficient: KV caching reduces redundant computation, quantization shrinks memory footprints, speculative decoding accelerates token generation, and continuous batching maximizes GPU utilization. These per-request optimizations are necessary but not sufficient for production deployment. A single well-tuned model instance still cannot absorb thousands of simultaneous requests without a serving layer that multiplexes demand across instances, manages queues intelligently, and scales capacity to match load.

The key insight is that language model inference has fundamentally different characteristics from other computational workloads. Each request does not complete in a fixed time the way a database query might, each token generated depends on all prior tokens, and the memory footprint of a single request grows with every generation step. These properties mean that standard web service patterns, such as stateless request handling and simple round-robin load balancing, work poorly without LLM-specific adaptations. Understanding why standard approaches fail helps us understand the design choices behind modern serving frameworks.

Production serving also involves making explicit trade-offs between competing objectives that cannot all be maximized simultaneously. Lower latency typically requires either more hardware or smaller batch sizes, both of which increase cost. Higher throughput typically requires larger batches, which increases latency for individual requests. Serving many users cheaply requires high GPU utilization, but high utilization leaves no headroom for traffic spikes. The frameworks and techniques in this chapter help you choose among these trade-offs based on your requirements, traffic patterns, and budget constraints.

This chapter focuses on the serving layer above individual inference optimization: how to orchestrate multiple model instances into a production system that serves many users simultaneously while meeting latency and throughput requirements. We cover inference server architecture, request routing strategies, load balancing approaches adapted for LLM workloads, auto-scaling under variable demand, latency optimization techniques, and the monitoring infrastructure needed to maintain service quality over time.

An inference server sits between client applications and the underlying model. It manages requests and batches them for scheduling, then streams the generated responses. Modern LLM inference servers have evolved specialized architectures to address the unique challenges of autoregressive generation. Understanding these architectural patterns is needed for anyone building or operating production language model services, because the choices made at this layer determine the performance ceiling for everything built on top.

Think of the inference server as the operating system for your model. Just as an operating system multiplexes CPU time across multiple processes, allocates memory across competing programs, and provides a unified interface to hardware, the inference server multiplexes GPU compute across requests, manages the KV cache across concurrent sessions, and presents a clean HTTP/gRPC interface to applications. The analogy extends further: just as operating systems evolved from simple batch schedulers to sophisticated time-sharing systems that could support many concurrent users, LLM inference servers have evolved from simple request queues to continuous batching systems that can fluidly interleave prefill and decode phases across dozens of concurrent requests.

The design of an LLM inference server differs fundamentally from a traditional REST API server. A traditional API server is stateless: each request arrives, executes a function, returns a result, and frees all associated memory. An LLM server is stateful during generation: each request allocates a growing block of KV cache memory that persists for the duration of token generation. A request generating 500 tokens holds memory for the entire duration, unlike a database query that returns immediately. This statefulness requires the inference server to actively manage memory, schedule when requests begin processing, and decide when to preempt long-running requests to make room for new arrivals.

Another key difference is that LLM requests are not uniform in cost. A request with a 4000-token prompt requires 40x more prefill computation than a 100-token prompt, and a request generating 500 tokens requires 500 individual decode steps while one generating 10 tokens requires only 10. A server that treats all requests as having equal cost (as naive implementations do) will chronically misallocate resources, sometimes starving short requests behind long ones or saturating one endpoint while others sit idle. The sophisticated scheduling and routing discussed throughout this chapter exists precisely to handle this non-uniformity.

The architecture of an LLM inference server differs significantly from traditional ML serving systems in basic ways that stem from the nature of autoregressive text generation. Classification models process requests independently with fixed compute costs: an image classifier performs the same computation regardless of the input image's content. Language models, by contrast, generate tokens iteratively, with each request potentially requiring hundreds of forward passes and memory that grows with sequence length. This iterative, variable-cost nature demands specialized architectural components that can efficiently manage the unique resource patterns of text generation.

A typical inference server contains several interconnected components, each playing a distinct role in the request processing pipeline:

• Request handler: Accepts incoming requests over HTTP/gRPC, validates inputs, and manages response streaming. For LLMs, this often involves Server-Sent Events (SSE) or WebSocket connections to stream tokens as they're generated. The request handler must maintain long-lived connections efficiently, as a single request may take several seconds to complete while tokens are progressively returned to the client.

• Tokenizer: Converts text to token IDs on input and back to text on output. While fast compared to model inference, tokenization can become a bottleneck at high throughput because it runs on the CPU while the model runs on the GPU. Servers often parallelize tokenization across CPU cores to prevent this component from limiting overall system throughput.

• Scheduler: Determines which requests to process together in each batch. As we discussed in the previous chapter on continuous batching, sophisticated schedulers dynamically add and remove requests from running batches. The scheduler must balance multiple objectives: maximizing GPU utilization, maintaining fair request ordering, and meeting latency targets for high-priority requests.

• Model executor: Manages the actual model inference, including memory allocation for KV caches, attention computation, and integration with optimized kernels like FlashAttention. This component coordinates the GPU computation and ensures that the model weights, activations, and cached attention states are efficiently arranged in memory.

• Memory manager: Tracks KV cache allocations, implements paged attention for efficient memory use, and handles memory pressure through eviction policies. Because KV cache memory requirements grow with sequence length and concurrent request count, intelligent memory management is necessary for maintaining high throughput without running out of GPU memory.

Several frameworks have emerged to handle LLM serving at scale, each with different strengths and design philosophies. Understanding the distinctions helps you choose the right tool for your deployment context, because switching frameworks later requires significant re-engineering effort.

vLLM implements PagedAttention for efficient KV cache management, letting high throughput with memory-constrained hardware. By treating KV cache memory like virtual memory pages, vLLM can handle more concurrent requests than naive implementations that allocate contiguous memory blocks. It supports continuous batching and integrates with popular model architectures, which makes it a popular choice for production deployments. Its open-source nature and active community mean that support for new models and hardware typically arrives quickly.

Text Generation Inference (TGI) from Hugging Face provides production-ready serving with tensor parallelism for multi-GPU deployments, quantization support, and optimized attention implementations. Its tight integration with the Hugging Face ecosystem makes it particularly convenient for teams already using Transformers for model development. TGI handles much of the operational complexity automatically, from model loading to request queuing, letting teams to reach production quickly.

TensorRT-LLM from NVIDIA offers highly optimized inference for NVIDIA GPUs, with custom CUDA kernels and support for quantization schemes like FP8 and INT4. When running on NVIDIA hardware, TensorRT-LLM often achieves the highest raw performance, though it requires more effort to set up than framework-agnostic alternatives. The performance gains are most pronounced for latency-necessary applications where every millisecond matters.

Triton Inference Server provides a model-agnostic serving platform that can host multiple models with different backends, useful for ensemble architectures or multi-model deployments. Its flexibility makes it well-suited for complex inference pipelines that combine multiple models or processing steps. For organizations running diverse model portfolios, Triton's unified interface reduces operational overhead.

The configuration balances memory allocation against concurrency in a careful trade-off. Higher gpu_memory_utilization allows more KV cache space for concurrent requests, but leaves less headroom for activation memory during computation. Setting this value too high can cause out-of-memory errors during prefill of long sequences; setting it too low wastes expensive GPU memory that could support additional concurrent requests. Most practitioners find values between 0.85 and 0.92 work well, with lower values giving a safety margin for unexpected memory spikes.

The key parameters for inference server configuration are:

• gpu_memory_utilization: Fraction of GPU memory to reserve for the model weights and KV cache. The remaining fraction is a safety margin for activation memory during computation and other runtime overheads.
• max_num_seqs: Maximum number of concurrent sequences the server can handle. Requests beyond this limit queue until a sequence slot becomes available.
• enable_chunked_prefill: Whether to split the prefill phase into chunks to prevent blocking the decode phase of other requests, letting interleaved progress across multiple active requests.

Understanding the journey of a request through the server helps identify optimization opportunities and debug performance issues. Each stage presents different bottlenecks and optimization levers. Think of the request lifecycle as a relay race: the baton (request) passes through multiple stages, and the slowest leg determines the overall time. Profiling which stage is slowest tells you exactly where to focus optimization effort.

1. Arrival: The request arrives via HTTP POST with prompt text and generation parameters (temperature, max tokens, etc.). The server validates the request format and assigns it a unique identifier for tracking.

2. Preprocessing: The server tokenizes the prompt and validates that the resulting sequence fits within model limits. Long prompts may be rejected or truncated depending on server configuration.

3. Scheduling: The scheduler decides when to begin processing. The request may queue if all batch slots are occupied. Priority and fairness policies determine ordering among waiting requests.

4. Prefill: The model processes all prompt tokens in parallel, populating the KV cache with attention states for all input positions. This phase is compute-bound because it performs dense matrix multiplications across the entire prompt length.

5. Decode Loop: Tokens are generated one at a time, with each new token streamed to the client. This phase is memory-bandwidth-bound because each forward pass must read the entire KV cache from GPU memory while creating only a single output token.

6. Completion: Generation stops when the model produces an end token or reaches the maximum length. The server releases KV cache memory and closes the response stream, making capacity available for new requests.

The key insight is that the prefill and decode phases have completely different performance profiles. Prefill is a matrix multiply problem where throughput scales with hardware compute capability. Decode is a memory-read problem where throughput scales with hardware memory bandwidth. This means that techniques that improve prefill speed (like tensor parallelism) may not help decode speed at all, and optimizations that target one phase can interfere with the other when they share GPU resources.

The metrics distinguish between time-to-first-token (TTFT) and total latency because these capture fundamentally different aspects of user experience. TTFT measures user-perceived responsiveness in streaming applications: when you send a message, how quickly you see the model start to respond? Total latency matters more for batch processing where results are consumed only after complete generation. Optimizing these metrics often involves different trade-offs. For example, aggressive batching improves total throughput but can increase TTFT because requests wait for batch formation.

When multiple model instances serve requests, a routing layer decides which instance handles each request. Effective request routing maximizes throughput while maintaining consistent latency. The routing layer must make rapid decisions with incomplete information, balancing immediate load distribution against longer-term efficiency considerations.

Think of request routing as air traffic control for your model fleet. Just as a controller assigns incoming flights to runways based on runway availability, weather, aircraft size, and priority, the router assigns incoming requests to model instances based on current load, request characteristics, priority level, and session history. A controller who assigns every plane to the nearest runway without checking whether it's occupied creates collisions; a router that sends every request to the same endpoint regardless of load creates a bottleneck just as severe.

The routing layer sees a stream of requests with varying characteristics: different prompt lengths, different expected output lengths, different priority levels, and different session histories. Making good routing decisions requires incorporating all this information into a fast decision that minimizes resource waste. Routing decisions that take longer than a few milliseconds add measurable overhead to every request, so the algorithms involved must be computationally lightweight even as they become semantically richer.

Production routing systems often combine multiple strategies in a hierarchy. A top-level policy might route requests to a specific model family based on capability requirements. A second level might balance load across instances of that model family using token-aware algorithms. A third level might check session affinity to enable KV cache reuse. This layered approach makes each individual routing decision simple while achieving sophisticated aggregate behavior.

Simple deployments route all requests to a single model version, treating the routing problem as pure load balancing. Production systems often need more sophisticated routing that considers request characteristics and business requirements.

Version-based routing directs requests to different model versions based on request metadata. This enables A/B testing by directing a fraction of traffic to a new model version, gradual rollouts that progressively shift traffic to updated models, and fallback to stable versions if new deployments exhibit problems. For example, a system might route 5% of traffic to a new model version while monitoring quality metrics before expanding the rollout.

Capability-based routing matches requests to appropriate models based on task complexity or domain. A lightweight model might handle simple queries like basic factual questions, while complex reasoning tasks route to larger, more capable models. This approach reduces cost without sacrificing quality where it matters: simple questions get fast, cheap answers while challenging problems receive appropriate computational resources.

Priority routing ensures high-priority requests (paying customers, necessary applications) get preferential treatment in terms of both queue position and endpoint selection. Lower-priority requests may queue longer or route to less powerful instances. This enables tiered service levels without requiring completely separate infrastructure for each tier.

The router considers both capacity and priority when making routing decisions. High-priority requests route to the least-loaded endpoint to minimize queuing time and ensure fast response. Lower-priority requests distribute randomly among available endpoints to prevent creating hotspots while still avoiding overloaded instances. This differentiated treatment allows the system to provide better service to priority traffic without completely starving lower-priority requests.

Effective routing depends on accurate health information about each endpoint. If the router sends requests to an unhealthy endpoint, those requests will fail or experience long delays. Servers implement multiple health check levels to detect different kinds of problems.

Think of health checks as the monitoring equivalent of a doctor's examination at different levels of detail. A liveness check is like checking whether a patient is breathing: a minimal test to confirm the system is alive. A readiness check is like checking whether a patient can walk: verifying that the system can do what it's supposed to do. A deep health check is like a full physical: running complete diagnostics to catch subtle problems before they become emergencies. Each check level involves progressively more work and catches progressively more subtle issues.

The latency of health checks matters. Checks that take hundreds of milliseconds delay routing decisions and consume resources on the endpoints being checked. Deep health checks that run test inferences are especially expensive because they consume the same GPU resources needed for real requests. Production systems typically run liveness and readiness checks every few seconds, while deep health checks run every minute or on a triggered basis when shallow checks detect potential problems.

• Liveness checks verify the process is running and responding to basic probes. A hanging process fails liveness checks even if it hasn't technically crashed. These checks detect catastrophic failures quickly.

• Readiness checks verify the server can accept new requests. A server in the process of loading a model passes liveness (the process is running) but fails readiness (it cannot yet serve requests). This distinction prevents routing traffic to instances that are starting up.

• Deep health checks verify end-to-end functionality by running test inferences. These checks catch subtle issues like corrupted model weights, CUDA driver problems, or memory leaks that wouldn't be detected by simpler checks. The trade-off is that deep checks consume GPU resources and take longer to complete.

Health checks enable automatic failover without human intervention. When an endpoint exceeds the failure threshold (typically 3 consecutive failures), the router removes it from rotation and stops sending new requests. This prevents user-facing errors from accumulating while the underlying issue is investigated. After the underlying issue resolves, consecutive successful checks restore the endpoint to service. The recovery threshold requires sustained health before the endpoint receives traffic again, preventing oscillation between healthy and unhealthy states.

Load balancing distributes requests across multiple model instances to maximize throughput and maintain consistent latency. While load balancing is a well-studied problem in distributed systems, LLM serving presents unique challenges that traditional load balancing algorithms don't address well. The variable cost of requests, the long-running nature of generation, and the importance of memory locality all complicate the problem.

Think of load balancing as managing a team of workers who do jobs of wildly varying duration. If you assign jobs by turns (round-robin), a worker who gets a ten-hour job will still be busy long after everyone else has cycled through dozens of short jobs, creating severe imbalance. If you assign by fewest current jobs, you might send the next job to a worker already doing a ten-hour task just because they technically have fewer total jobs than someone with three five-minute tasks. The right approach considers how much work each worker has remaining, not just how many items are on their list.

The basic challenge for LLM load balancing is that "load" is not a single number. An endpoint serving ten short requests that will each complete in 200ms has very different remaining work than an endpoint serving one request that will generate 2000 tokens over 40 seconds. Connection count captures neither the diversity in prompt length nor the diversity in requested output length. Token-aware approaches exist precisely because connection-count metrics mislead the balancer into poor decisions.

Load balancing also interacts with higher-level concerns like session affinity and KV cache efficiency. The best endpoint for a new request from the standpoint of pure load might not be the best endpoint when you factor in that routing to a specific instance would allow the server to reuse cached KV states from a prior request in the same conversation. These multi-factor decisions are a microcosm of the broader production serving challenge: each individual optimization creates interference with others, and achieving good overall performance requires reasoning about the whole system simultaneously.

Standard load balancing approaches provide a foundation for understanding the problem, even if they don't fully address LLM-specific challenges.

Round-robin distributes requests sequentially across endpoints in a fixed order. The first request goes to endpoint 0, the second to endpoint 1, and so on, cycling back to 0 after reaching the last endpoint. This predictable approach is easy to implement and gives each endpoint the same request count. However, it ignores actual load, which matters when requests have varying computational costs. A long generation request and a short one count equally, even though they impose vastly different burdens.

Least connections routes each new request to the endpoint with the fewest active requests. This approach adapts better to variable-duration requests because endpoints processing long requests naturally accumulate fewer connections over time. However, it still treats all requests as equal cost, which fails to capture the difference between generating 10 tokens and generating 1000 tokens.

Weighted distribution assigns requests proportionally to endpoint capacity, which is useful when endpoints have different hardware. An H100 GPU might receive twice as many requests as an A100 because it can process them twice as fast. Weights can be configured manually based on hardware specs or adjusted dynamically based on observed performance.

The results illustrate the behavioral differences between strategies. Round-robin distributes evenly regardless of load, giving each endpoint roughly 33% of requests. Least connections concentrates traffic on the least-loaded endpoint (gpu-0), which may improve its latency but could overload it if requests are long-running. Weighted random accounts for available capacity, directing more requests to endpoints with headroom while maintaining some randomness to avoid perfect synchronization effects.

Traditional metrics like connection count fail to capture LLM workload characteristics because they treat all requests as having equal cost. In reality, a request generating 1000 tokens consumes far more GPU resources and time than one generating 10 tokens, even though both count as single connections. Similarly, a request with a 4000-token prompt requires substantially more prefill computation than one with a 100-token prompt.

The key insight is that LLM load should be measured in tokens, not requests. When we measure load in tokens, we can compute a much more accurate picture of how much remaining work each endpoint has. An endpoint with three requests each generating 20 tokens has roughly the same remaining work as one request generating 60 tokens. An endpoint-count metric would show the first endpoint as having 3x the load when in fact they are equivalent.

Token-aware load balancing accounts for the actual computational cost of requests by tracking token counts rather than just request counts. We also need to distinguish between tokens in the prefill queue (waiting to be processed) and tokens in the decode phase (currently being generated), because these phases have very different costs. The decode phase is the throughput bottleneck in most production systems because memory bandwidth limits how quickly individual decode steps execute, so active decode tokens should be weighted more heavily than queued prefill tokens.

Token-aware balancing makes smarter routing decisions by considering the actual work each endpoint is performing. In this example, gpu-0 has more prompt tokens queued for prefill, but gpu-1 has more active decode tokens. Because decode is the bottleneck phase (each token requires reading the entire KV cache), the long-output request routes to gpu-0, which has less active decoding. This prevents one endpoint from becoming decode-saturated while another sits idle during its compute-bound prefill phase.

Some applications benefit from routing subsequent requests to the same endpoint to enable optimization opportunities. If a conversation spans multiple API calls with the same system prompt or shared context, routing to the same instance enables KV cache reuse through prefix caching, as discussed earlier in this part. Rather than recomputing attention states for the repeated prefix, the server can reuse cached values, significantly accelerating prefill.

However, session affinity conflicts with optimal load balancing because it constrains routing decisions. If a particular session's preferred endpoint becomes overloaded, the system must choose between maintaining affinity (and accepting higher latency) or breaking affinity (and losing cache benefits). A simple compromise routes requests with known session IDs to consistent endpoints using hash-based assignment, while distributing new sessions based on current load.

The trade-off between affinity and balance is most pronounced for long-running conversations with consistent system prompts. For a chatbot with a fixed system prompt that consumes 500 tokens, routing all turns of the same conversation to the same instance could save 500 tokens of prefill computation per turn after the first, a potentially large saving for high-volume deployments. Whether this benefit outweighs the load balancing cost depends on your traffic patterns: if conversations are short and system prompts are short, the benefit is minimal; if conversations are long and system prompts are long, affinity provides meaningful efficiency gains.

Consistent hashing ensures the same session always reaches the same endpoint, assuming the endpoint pool remains stable. This deterministic mapping enables KV cache hits for repeated prompts within a conversation. When endpoints are added or removed, consistent hashing minimizes disruption by changing the mapping for only a fraction of sessions rather than all of them. More sophisticated approaches can combine affinity preferences with load awareness, preferring the affinity endpoint when it has capacity but falling back to other endpoints when necessary.

Static deployments with a fixed number of instances struggle with variable demand. Traffic to inference services often varies dramatically by time of day, day of week, and in response to external events. Auto-scaling adjusts the number of model instances based on load, reducing costs during low-traffic periods while maintaining performance during peaks. This elastic capacity is one of the key advantages of cloud deployments over fixed on-premises infrastructure.

Think of auto-scaling as hiring and laying off staff in real time. A restaurant that always staffs for the dinner rush pays full labor costs all day even when it's empty at 3pm. One that can bring servers in two hours before the rush and send them home after closing minimizes labor cost without sacrificing service quality during peak demand. Auto-scaling does the same for GPU compute: you pay for instances only when you need them, rather than always provisioning for worst-case load. The challenge is that unlike restaurant staff who can show up in minutes, LLM instances take substantial time to become ready, which makes anticipating demand spikes necessary.

The decision logic for auto-scaling involves two separate problems: detecting when to scale (policy) and how fast to scale (mechanism). The policy problem is about choosing the right metrics and thresholds that accurately predict when current capacity will be insufficient. Scaling too eagerly wastes money; scaling too timidly allows user-facing degradation. The mechanism problem is about how quickly new instances become available and how smoothly traffic shifts to them. Cold starts that take two minutes create a significant gap between when you decide to scale and when the capacity is available.

A subtle challenge in auto-scaling LLM services is that the feedback loop between scaling decisions and their effects is delayed. When you add a new instance, it takes time to load the model, warm its caches, and receive traffic. During this time, the original instances continue experiencing high load. This delay means scaling logic must be predictive rather than purely reactive: you need to decide to scale before the situation becomes necessary, not after users are already experiencing degradation.

Choosing the right metrics to trigger scaling is important because different metrics capture different aspects of system health and have different response characteristics.

Queue depth is the number of waiting requests and provides a direct measure of demand exceeding capacity. High queue depth indicates insufficient capacity and directly predicts increased latency for arriving requests. This metric responds quickly to load changes because queues grow immediately when arrivals exceed processing rate. The key insight is that a deep queue is a leading indicator of latency problems: if you wait until P95 latency is already high, requests have already waited too long. Acting on queue depth lets you scale preemptively before users experience degradation.

Latency percentiles, specifically P50, P95, or P99 request latency, directly measure user experience. Scaling based on latency targets addresses user experience directly, which is ultimately what matters. However, latency increases only after the system is already overloaded, making this a lagging indicator. By the time latency rises, users have already experienced degraded service, so latency metrics are better for detecting that scaling has failed than for triggering timely scale-up actions.

GPU utilization is the fraction of GPU compute being used and indicates how efficiently the current capacity is being used. Low utilization suggests over-provisioning and wasted cost. Consistently high utilization suggests under-provisioning and risk of latency degradation as load increases further. However, GPU utilization can be misleading for LLM serving: a server spending most of its time on memory-bound decode operations may show low compute utilization while still being fully saturated in terms of memory bandwidth.

Tokens per second is the aggregate throughput across instances and measures actual work being done. Unlike request count, this metric accounts for variable request sizes. A spike in tokens per second indicates increased load even if request count is stable (perhaps because users are submitting longer prompts).

The simulation illustrates key auto-scaling behaviors. The system scales up quickly when queue depth spikes, adding capacity before latency degrades too severely. It then gradually scales down as load decreases, but with a longer cooldown to avoid oscillation. The asymmetric cooldowns (60 seconds up, 300 seconds down) prevent thrashing while so fast response to traffic increases. Scaling up quickly prevents user-facing degradation, while scaling down slowly ensures that brief dips in traffic don't cause premature capacity reduction that would require expensive cold starts when traffic rebounds.

The key parameters for the auto-scaling logic are:

• max_queue_depth: Threshold for the number of queued requests that triggers a scale-up event. Setting this too low causes unnecessary scaling; setting it too high delays scaling until users are already experiencing degradation.
• max_latency_p95_ms: P95 latency threshold that triggers scaling when exceeded. Because latency is a lagging indicator, this typically fires after queue depth has already signaled the problem.
• scale_up_cooldown_seconds: Minimum time interval between scale-up actions to prevent flapping. Should be long enough that new instances become ready before the next scaling evaluation, but short enough to allow rapid scale-up when needed.

LLM instances take significant time to become ready for serving. Loading a 7B parameter model requires transferring approximately 14GB (in FP16) from storage to GPU memory, then initializing CUDA kernels and compiling any JIT-compiled operations, and allocating KV cache space. This cold start latency, often 30-120 seconds for typical deployments, complicates auto-scaling because new capacity isn't immediately available when scaling decisions are made.

Think of cold start latency as the time it takes a new employee to get their computer set up and learn the login credentials before they can start taking calls. You can hire them the moment the phone lines get busy, but they won't be answering calls for the first hour. For LLM serving, this means you need to start scaling before demand spikes, not in response to demand spikes. Reactive scaling that triggers only when queues are already long will consistently fail to provide capacity in time.

The 30-120 second cold start range spans multiple orders of magnitude depending on deployment configuration. A 7B model stored on local NVMe drives loads much faster than the same model stored in network-attached storage. Models that use eager CUDA kernel compilation (like some quantized models) add seconds of initialization overhead. Models with large context windows allocate more KV cache memory, which takes longer to initialize. Understanding the specific cold start profile of your deployment is needed for configuring appropriate scale-up lead times.

Several strategies help mitigate cold start impact.

Warm pools maintain a pool of pre-initialized but idle instances that have already loaded model weights and initialized their CUDA context. New traffic immediately activates these instances rather than waiting for cold start. The trade-off is paying for idle GPU time, as warm pool instances consume resources even when not serving requests.

Predictive scaling uses historical patterns to anticipate demand and scale preemptively. If traffic typically peaks at 9 AM on weekdays, begin scaling at 8:50 AM so new instances are ready before the rush. Machine learning models can predict traffic patterns more accurately than simple rules, but require historical data and ongoing maintenance.

Gradual traffic shift brings new instances online by ramping traffic gradually rather than immediately directing full load. This prevents overwhelming a fresh instance while its various caches (file system cache, KV prefix cache, CUDA kernel cache) are cold. Gradual ramp-up also allows detecting problems with new instances before they impact a large fraction of traffic.

The warm pool enables near-instant scaling for traffic spikes. When traffic increases, instances activate in seconds (just the time to update routing tables) rather than the minute or more required for cold start. When traffic drops, instances return to the warm pool rather than terminating completely, preserving their warmed state for the next spike. This approach trades some ongoing cost (idle warm instances) for dramatically improved responsiveness to load changes.

Meeting latency targets requires understanding the distinct phases of LLM inference and optimizing each appropriately. Because the prefill and decode phases have fundamentally different computational characteristics, they require different optimization strategies. A one-size-fits-all approach will inevitably leave performance on the table.

Think of optimizing LLM latency as optimizing a two-stage pipeline where each stage has a completely different bottleneck. The prefill stage is like a high-throughput assembly line that can process many units in parallel: the more parallelism you add (more GPUs, larger matrix operations), the faster it goes, and the bottleneck is arithmetic throughput. The decode stage is like a process that must read from a very slow shared library for every single item: the bottleneck is how quickly you can access memory, and adding more compute units helps very little because they all spend most of their time waiting for memory reads to complete. Recognizing these distinct bottlenecks is the first step toward effective optimization.

The practical implication is that profiling your specific workload matters enormously. A chatbot with short system prompts and short responses will spend most of its time in decode, so KV cache compression and speculative decoding are high-value investments. A document summarization service with 3000-token prompts and 200-token summaries will spend a large fraction of its time in prefill, so chunked prefill and tensor parallelism are more valuable. No single optimization profile fits all workloads, and measuring where your time goes prevents you from over-investing in optimizations that target the wrong bottleneck.

As we have seen throughout this part, LLM inference has two distinct phases with different performance characteristics that demand different optimization approaches.

The prefill phase involves processing the input prompt by computing attention across all prompt tokens simultaneously. This phase is compute-bound because it performs dense matrix multiplications with operations proportional to the square of the prompt length (due to attention) times the model dimension. The GPU's compute units are the bottleneck, and memory bandwidth is typically not limiting. Optimization strategies for the prefill phase include:

• Chunked prefill: Split the prefill phase into chunks to avoid blocking the decode phase of other requests, letting interleaved progress on multiple requests
• Tensor parallelism: Distribute compute across GPUs, reducing wall-clock time for long prompts
• Quantized attention: Faster matrix operations that trade some precision for speed

The decode phase generates output tokens one at a time, meaning each forward pass produces only a single token while reading the entire KV cache from memory. This phase is memory-bandwidth-bound because the ratio of computation to memory access is low. Each token requires reading billions of bytes of model weights and cached attention states while performing relatively few arithmetic operations. Optimization strategies for the decode phase include:

• KV cache compression: Reduce memory reads by shrinking the data that must be transferred each step
• Speculative decoding: Generate multiple tokens per forward pass, amortizing memory access cost
• Continuous batching: Share the memory bandwidth cost across multiple requests, improving effective utilization

For a typical prompt of 1000 tokens, prefill completes in roughly 100ms. A 10-token response adds 200ms of decode time, making prefill a significant fraction (33%) of total latency. A 500-token response adds 10 seconds of decode time, making prefill negligible (less than 1%). This dramatic shift means optimization priorities should align with your actual request distribution. If most requests generate short outputs, optimizing prefill matters more. If most requests generate long outputs, decode optimization dominates.

Many applications stream responses to users, making time-to-first-token (TTFT) the primary latency metric that determines perceived responsiveness. When a user sends a message in a chat interface, TTFT measures the time from submission until the first word of the response appears. Users perceive systems with low TTFT as fast and responsive, even if total generation time is similar.

TTFT matters psychologically as much as technically. Research on user experience shows that users form lasting impressions of a system's speed based heavily on how quickly it begins responding, not just how long it takes to complete. A system that starts generating immediately and takes 5 seconds total feels faster than one that waits 4 seconds before generating anything and completes in 3 seconds total, even though the second system is technically faster. This means that even if total token generation speed cannot be improved, reducing TTFT can improve users' perception of system quality.

TTFT measures the time from request arrival until the first generated token is available to the client. This encompasses several distinct stages, each giving optimization opportunities:

• Queue Time: How long the request waits before processing begins. Reducing queue time requires either more capacity (scaling) or smarter scheduling (prioritization).
• Tokenization: Converting the prompt text to token IDs. While typically fast, this step runs on CPU and can bottleneck under high load.
• Prefill Duration: Processing all prompt tokens to populate the KV cache. For long prompts, this dominates TTFT.
• First Decode Step: Generating the first output token. This is typically fast since it's a single forward pass.

The analysis reveals how bottlenecks shift with load. Under low load, prefill dominates TTFT because queue times are minimal and requests begin processing immediately upon arrival. Optimizing prefill speed through better hardware or chunking would improve TTFT in this regime. Under high load, queue time becomes the dominant bottleneck: requests spend most of their time waiting, not processing. In this regime, improving prefill speed won't help much because requests are already waiting; adding capacity or improving scheduling would be more effective. This analysis guides optimization investments toward the actual bottleneck rather than optimizing components that aren't limiting.

Optimizing for latency often conflicts with optimizing for throughput because the techniques that improve one typically harm the other. Larger batches improve GPU utilization and throughput: processing 32 requests in one batch uses GPU compute more efficiently than processing them one at a time. However, larger batches increase latency for individual requests because each request waits for the batch to form and then shares compute time with other requests in the batch.

The key insight is that latency and throughput are not independent dimensions that can be independently optimized; they are coupled through the shared resource of GPU compute. The basic relationship can be stated simply: throughput is rate (requests per second) and latency is time per request. When requests share compute in a batch, each request takes longer (higher latency) but the aggregate rate increases (higher throughput). There is no way to have both maximum throughput and minimum latency simultaneously at high concurrency.

The optimal operating point depends on your SLA requirements and cost constraints:

If your SLA requires P95 latency under 500ms, you're constrained to batch sizes below approximately 32 in this scenario. Larger batches would improve throughput (and thus cost efficiency) but violate the latency requirement. The continuous batching techniques from the previous chapter ease this trade-off by letting different requests enter and exit batches dynamically, improving throughput while limiting how long any individual request must wait.

To make the concepts concrete, let us walk through a realistic capacity planning exercise. Suppose you need to deploy a 7B parameter model serving a chatbot application with the following requirements: 100 requests per second at peak, a P95 TTFT target of 500ms, a P95 total latency target of 5 seconds, and typical prompts of 500 tokens generating 200 tokens of output.

Step 1: Estimate per-request compute requirements.

A 7B model in FP16 requires approximately 14GB for model weights. Each forward pass during decode reads all weights plus the KV cache. With a 500-token prompt and 200 tokens generated, the KV cache grows from 500 to 700 entries during the decode phase. On an H100 GPU with 3.35 TB/s memory bandwidth, each decode step reading 14GB of weights takes roughly $14 \times 10^9 / (3.35 \times 10^{12}) \approx 4.2$ ms per token. For 200 tokens, decode takes approximately 840ms.

Step 2: Estimate prefill time.

Prefill with 500 tokens on an H100 (989 TFLOPS FP16) involves roughly $500^2 \times 4096 \times 2 \approx 2$ billion floating point operations (attention) plus similar cost for FFN layers. The full prefill completes in roughly $14 \times 10^9 / (989 \times 10^{12}) \approx 14$ ms, dominated by memory bandwidth rather than compute at this prompt length.

Step 3: Determine single-instance throughput.

With continuous batching, a single H100 can serve approximately 30-50 concurrent decode sequences while maintaining reasonable token throughput. At 50 concurrent sequences each generating at 4.2ms per token, the single-instance throughput is approximately $50 / 4.2 \times 10^{-3} \approx 12{,}000$ tokens/second or roughly 60 requests completed per second (at 200 tokens each).

Step 4: Calculate required instances for throughput.

At 100 requests per second, you need approximately $100 / 60 \approx 1.7$ instances to handle the throughput. Round up to 2 active instances plus 1 warm spare (for failover and traffic spikes), giving 3 total instances.

Step 5: Verify latency targets.

With 2 active instances and 100 rps arriving, each instance sees 50 rps. Queue depth under steady load with 50 concurrent sequences per instance stays low. Total latency is prefill (14ms) plus decode (840ms) plus queue time (expected near zero at this load), totaling approximately 854ms. The 5-second P95 total latency target has comfortable headroom. TTFT is queue time (near zero) plus prefill (14ms) plus first decode step (4.2ms), totaling approximately 18ms, well within the 500ms target.

Step 6: Configure auto-scaling thresholds.

Set queue depth threshold at 20 requests (signaling sustained load above capacity), scale-up cooldown at 90 seconds (matching model load time on NVMe-attached storage), scale-down cooldown at 300 seconds, warm pool size at 1, and replica bounds between 2 and 8.

This analysis shows that for this workload, the limiting factor is not latency (there is substantial headroom) but throughput: the system can handle the load with 2 instances, but needs careful warm pool configuration to handle demand spikes without cold-start delays.

Production inference systems require complete monitoring to maintain service quality and debug issues. Without visibility into system behavior, operators cannot distinguish between capacity problems, model issues, and infrastructure failures. Good observability enables proactive intervention before users experience degraded service.

Think of monitoring as the nervous system of your serving infrastructure. Without it, the infrastructure is operationally blind, it cannot feel pain when things go wrong, cannot detect when it is being stressed beyond its limits, and cannot signal for help before failures cascade. With good monitoring, operators get early warning of developing problems, data for capacity planning decisions, and the detailed traces needed to diagnose the root cause of any issue that does arise.

Effective monitoring operates at multiple time scales. Real-time dashboards showing current queue depth and latency percentiles let operators detect immediate problems. Hourly trend charts reveal developing capacity issues before they become necessary. Daily and weekly aggregates inform capacity planning and SLA compliance reporting. Each time scale requires different data retention and aggregation strategies, and building all of them into the monitoring infrastructure from the start is much easier than adding them after a production incident reveals their absence.

A common failure mode in monitoring infrastructure is collecting the wrong metrics rather than no metrics at all. GPU utilization sounds informative but can be misleading for LLM serving, as discussed earlier. Request count sounds basic but misses the wide variance in request cost. Building metrics around the actual resource constraints (memory bandwidth, KV cache occupancy, queue depth) rather than generic infrastructure metrics gives operators a much more accurate picture of system health.

Effective monitoring tracks metrics at multiple levels, from individual request performance to aggregate system health to business outcomes.

Request-level metrics capture the experience of individual requests:

• Time to first token (TTFT): Measures perceived responsiveness for streaming applications
• Total latency: Measures end-to-end completion time for batch processing use cases
• Tokens per request (prompt and generated): Indicates request complexity and resource consumption
• Error rates by error type: Distinguishes between timeouts, out-of-memory, and other failure modes

System-level metrics reflect the health of the serving infrastructure:

• GPU utilization and memory usage: Indicates whether hardware is being used efficiently
• Queue depth over time: Reveals demand patterns and capacity mismatches
• Active batch size: Shows how effectively batching is working
• KV cache utilization: Indicates memory pressure and potential for cache eviction

Business-level metrics connect technical performance to business outcomes:

• Requests per second: Measures overall demand and system scale
• Tokens per second: Measures actual work being done, accounting for request size variation
• Cost per 1000 tokens: Combines infrastructure cost with throughput for economic analysis
• SLA compliance rate: The ultimate measure of whether the system meets its commitments

Service Level Objectives (SLOs) define target performance thresholds that the system should maintain. SLOs translate business requirements (users expect responsive service) into measurable technical targets (P95 latency under 5 seconds). Well-defined SLOs serve multiple purposes: they guide engineering priorities during design, provide objective criteria for evaluating incidents, and enable honest communication with customers about service commitments.

The choice of SLO thresholds reflects both technical reality and business priorities. A threshold set too tight creates constant alerts for acceptable behavior and breeds alert fatigue. A threshold set too loose provides false confidence and misses real degradation. Most teams calibrate initial thresholds based on observed performance under good conditions, then tighten them as the system matures and optimization efforts push achievable performance lower.

Common SLOs for inference services include:

• Availability: 99.9% of requests succeed (allows approximately 8.7 hours of downtime per year). This SLO accounts for both planned maintenance and unexpected failures.
• Latency: P95 latency below 5 seconds ensures that even unlucky requests complete in reasonable time.
• TTFT: P95 time-to-first-token below 500ms ensures users perceive the system as responsive.
• Error rate: Less than 0.1% of requests return errors. This keeps high reliability for applications built on the inference service.

Alerting should trigger when metrics approach SLO thresholds, not just when they're breached. Alerting only on violations means users have already experienced degraded service by the time operators learn of the problem:

The monitor verifies that all performance metrics currently satisfy their service level objectives. By implementing both warning thresholds (at 80% of the SLO) and necessary thresholds (at 100%), the system provides early indications of degrading performance before actual SLA violations occur. Warning alerts give operators time to investigate and potentially scale up or address issues before users experience violations. Critical alerts indicate that the SLA has already been breached and immediate action is required.

Let's implement a simplified inference serving pipeline that combines the concepts from this chapter. This example demonstrates how request handling, streaming responses, concurrency limits, and metrics collection work together in a complete system:

This simulation demonstrates the key components working together in a realistic pattern: requests arrive at varying rates following a Poisson process, queue when the server is at capacity (limited to 5 concurrent requests), stream responses as tokens generate, and metrics accumulate for monitoring. The async design allows concurrent request processing while respecting capacity limits, and the streaming response pattern provides tokens to clients as soon as they're available rather than waiting for complete generation.

The key parameters for the inference server simulation are:

• max_concurrent: Maximum number of requests the server processes simultaneously. Requests beyond this limit are queued, which increases their TTFT but prevents overwhelming the model or running out of GPU memory.
• tokens_per_second: The rate at which the model generates tokens during the decode phase. Higher values indicate faster hardware or more efficient implementation.
• arrival_rate: The average number of requests arriving per second. When arrival rate exceeds processing rate, queues build up and latency increases.

Inference serving for LLMs remains challenging despite sophisticated techniques. Operators must manage several basic tensions according to their requirements. Understanding these limitations is as important as understanding the capabilities, because production incidents almost always stem from one of these tensions being poorly managed rather than from bugs in the serving logic.

The cost-versus-latency tension is perhaps the most universal challenge. GPU instances are expensive, often costing several dollars per hour per GPU, but maintaining low latency requires spare capacity. Running at high utilization saves money but increases queue times and latency variance because there is no buffer for traffic spikes. Organizations must balance these competing priorities based on their SLAs and budget constraints. A consumer application with millions of users may tolerate higher latency for free tier users to reduce cost, while an enterprise application may pay for reserved capacity to guarantee consistent latency. Auto-scaling helps but introduces its own costs: warm pool instances consume resources even when idle, and cold starts delay capacity increases during demand spikes. There is no configuration that makes this tension disappear; operators must consciously choose where on the cost-latency tradeoff curve their specific application should sit.

The batching complexity tension affects both performance and operational simplicity. Continuous batching improves throughput significantly by amortizing GPU memory bandwidth across multiple requests, but it complicates request prioritization and debugging. When requests share batches, attributing latency to individual requests becomes harder because each request's timing depends on what else is in the batch. If a high-priority request happens to be batched with many long-generating requests, it experiences the same decode pace as the long requests even though it should theoretically complete faster. Strict SLA requirements may force smaller batches, sacrificing efficiency for predictability. The system that maximizes average throughput often does so at the cost of tail latency predictability, which is precisely what matters most for SLA compliance.

Tail latency behavior in LLM serving systems is systematically worse than in traditional stateless APIs, and this deserves explicit acknowledgment. While median latency may be well within SLA targets, P99 latency can be dramatically higher due to several compounding effects: occasional garbage collection pauses in the serving process, rare but long requests that occupy batch slots and delay others, memory pressure events that trigger KV cache eviction and force expensive recomputation, and network jitter on the necessary path for streaming responses. Achieving consistently low tail latency requires overprovisioning headroom above what median workloads require, which directly conflicts with cost efficiency goals. Operators who look only at P50 and P95 in their dashboards will be surprised by customer complaints about occasional very slow responses that are real but invisible in the aggregate statistics.

Model updates add an entire dimension of operational complexity that does not exist for stateless services. Deploying a new model version without downtime requires careful orchestration: blue-green deployments maintain two complete environments and switch traffic between them, requiring double the GPU infrastructure during transitions. Canary releases gradually shift traffic to new versions while monitoring for regressions in quality metrics that cannot be measured from latency alone. An updated model might be faster but produce different or worse outputs, and catching such regressions requires either human evaluation or automated quality metrics that run alongside performance monitoring. Rollbacks must be fast when new versions underperform, but rolling back means the deployment pipeline's time and cost were wasted. Both approaches add operational complexity and require infrastructure support that must be built and maintained alongside the serving system itself.

Multi-tenancy introduces isolation challenges that compound over time. Shared infrastructure serving multiple customers means that one customer's burst of long requests (perhaps an automated script generating bulk content) can degrade service for others who are paying for the same SLAs. Token-aware load balancing and request queuing help distribute this impact, but perfect isolation requires separate instances, which increases cost proportionally to the number of tenants. The compromise most systems reach is partial isolation: separate queues per tenant with shared model instances, which provides fairness in scheduling but not full isolation of latency characteristics. When isolation guarantees matter, as they do for enterprise customers, this partial isolation is often insufficient and dedicated instances are required.

The techniques covered in this part (KV caching, quantization, speculative decoding, continuous batching) address many of these challenges, but their interaction with serving infrastructure requires careful tuning. Optimizing a single request's inference speed differs from optimizing system-wide throughput under concurrent load. A technique that improves single-request latency might harm throughput under load, or vice versa. The only reliable path to a well-tuned system is systematic measurement under realistic load, followed by iterative adjustment guided by the specific metrics that matter for your application and customers.

This chapter covered the infrastructure layer that turns trained language models into production services.

Inference server architecture comprises specialized components including request handlers, schedulers, model executors, and memory managers. Modern servers like vLLM and TGI implement sophisticated features like PagedAttention and continuous batching that make them dramatically more efficient than simple model serving. The request lifecycle spans queuing, tokenization, prefill, decode, and completion, with each stage presenting distinct optimization opportunities.

Request routing determines which model instance handles each request, supporting version-based routing for A/B testing, capability-based routing for cost optimization, and priority routing for SLA differentiation. Health checks ensure routing decisions reflect actual endpoint availability and enable automatic failover without human intervention.

Load balancing for LLMs requires token-aware approaches that account for variable request costs. Traditional least-connections balancing fails to capture the difference between short and long generations. Session affinity enables KV cache reuse but conflicts with optimal load distribution, and real systems balance these two objectives based on traffic patterns and cache efficiency gains.

Auto-scaling adjusts capacity based on metrics like queue depth, latency percentiles, and GPU utilization. Cold start latency for LLM instances (often 30-120 seconds) necessitates warm pools and predictive scaling to maintain responsiveness during demand spikes. The asymmetric cooldowns for scale-up and scale-down prevent capacity oscillation while allowing a rapid response to demand increases.

Latency optimization requires understanding the distinct prefill (compute-bound) and decode (memory-bound) phases. Time-to-first-token matters for interactive applications, while total latency matters for batch processing. Bottleneck analysis reveals whether optimization should target prefill speed, decode speed, or queue reduction, and investing in the wrong dimension wastes effort.

Monitoring tracks request-level metrics (TTFT, latency), system-level metrics (GPU utilization, queue depth), and business metrics (tokens per second, cost). SLO-based alerting with warning thresholds provides early warning before service degradation affects users.

These serving concepts build on the inference optimization techniques from earlier chapters in this part. Together, they enable deploying language models that scale from prototype to production, handling thousands of concurrent users while meeting latency and cost targets.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about Inference Serving.