LLM Inference Serving: Architecture, Routing & Auto-Scaling › Limitations and Practical Considerations

Inference serving for LLMs remains challenging despite sophisticated techniques. Operators must manage several basic tensions according to their requirements. Understanding these limitations is as important as understanding the capabilities, because production incidents almost always stem from one of these tensions being poorly managed rather than from bugs in the serving logic.

The cost-versus-latency tension is perhaps the most universal challenge. GPU instances are expensive, often costing several dollars per hour per GPU, but maintaining low latency requires spare capacity. Running at high utilization saves money but increases queue times and latency variance because there is no buffer for traffic spikes. Organizations must balance these competing priorities based on their SLAs and budget constraints. A consumer application with millions of users may tolerate higher latency for free tier users to reduce cost, while an enterprise application may pay for reserved capacity to guarantee consistent latency. Auto-scaling helps but introduces its own costs: warm pool instances consume resources even when idle, and cold starts delay capacity increases during demand spikes. There is no configuration that makes this tension disappear; operators must consciously choose where on the cost-latency tradeoff curve their specific application should sit.

The batching complexity tension affects both performance and operational simplicity. Continuous batching improves throughput significantly by amortizing GPU memory bandwidth across multiple requests, but it complicates request prioritization and debugging. When requests share batches, attributing latency to individual requests becomes harder because each request's timing depends on what else is in the batch. If a high-priority request happens to be batched with many long-generating requests, it experiences the same decode pace as the long requests even though it should theoretically complete faster. Strict SLA requirements may force smaller batches, sacrificing efficiency for predictability. The system that maximizes average throughput often does so at the cost of tail latency predictability, which is precisely what matters most for SLA compliance.

Tail latency behavior in LLM serving systems is systematically worse than in traditional stateless APIs, and this deserves explicit acknowledgment. While median latency may be well within SLA targets, P99 latency can be dramatically higher due to several compounding effects: occasional garbage collection pauses in the serving process, rare but long requests that occupy batch slots and delay others, memory pressure events that trigger KV cache eviction and force expensive recomputation, and network jitter on the necessary path for streaming responses. Achieving consistently low tail latency requires overprovisioning headroom above what median workloads require, which directly conflicts with cost efficiency goals. Operators who look only at P50 and P95 in their dashboards will be surprised by customer complaints about occasional very slow responses that are real but invisible in the aggregate statistics.

Model updates add an entire dimension of operational complexity that does not exist for stateless services. Deploying a new model version without downtime requires careful orchestration: blue-green deployments maintain two complete environments and switch traffic between them, requiring double the GPU infrastructure during transitions. Canary releases gradually shift traffic to new versions while monitoring for regressions in quality metrics that cannot be measured from latency alone. An updated model might be faster but produce different or worse outputs, and catching such regressions requires either human evaluation or automated quality metrics that run alongside performance monitoring. Rollbacks must be fast when new versions underperform, but rolling back means the deployment pipeline's time and cost were wasted. Both approaches add operational complexity and require infrastructure support that must be built and maintained alongside the serving system itself.

Multi-tenancy introduces isolation challenges that compound over time. Shared infrastructure serving multiple customers means that one customer's burst of long requests (perhaps an automated script generating bulk content) can degrade service for others who are paying for the same SLAs. Token-aware load balancing and request queuing help distribute this impact, but perfect isolation requires separate instances, which increases cost proportionally to the number of tenants. The compromise most systems reach is partial isolation: separate queues per tenant with shared model instances, which provides fairness in scheduling but not full isolation of latency characteristics. When isolation guarantees matter, as they do for enterprise customers, this partial isolation is often insufficient and dedicated instances are required.

The techniques covered in this part (KV caching, quantization, speculative decoding, continuous batching) address many of these challenges, but their interaction with serving infrastructure requires careful tuning. Optimizing a single request's inference speed differs from optimizing system-wide throughput under concurrent load. A technique that improves single-request latency might harm throughput under load, or vice versa. The only reliable path to a well-tuned system is systematic measurement under realistic load, followed by iterative adjustment guided by the specific metrics that matter for your application and customers.