Model Serving › Limitations and Practical Considerations

Model serving combines software engineering, systems programming, and machine learning. Each domain contributes a distinct category of challenges.

LLM serving is tightly coupled to GPU hardware in ways that create significant operational constraints. A model compiled with TensorRT-LLM for one GPU generation may not run on another without recompilation. Serving frameworks that use custom CUDA kernels require specific driver and toolkit versions. Upgrading hardware often means requalifying the entire serving stack. This hardware specificity contrasts with traditional web services that can run on commodity hardware with minimal configuration, and it means that organizations running LLM services must invest substantially in hardware standardization and upgrade planning.

The dependency extends to firmware and driver versions. A serving stack that works perfectly with CUDA 12.2 and driver 535 may silently produce wrong outputs with CUDA 12.4 due to kernel-level changes. Integration tests must validate model outputs against reference outputs before any infrastructure update is adopted; checking only that the server starts and responds is insufficient.

The time required to load a large model into GPU memory, allocate KV cache structures, and complete warmup requests before the server can handle production traffic is substantial. A 70B-parameter model may require several minutes from process start to first request. This cold start latency complicates autoscaling strategies: spinning up new instances in response to a traffic spike takes too long to help with the spike itself. Serving systems must maintain warm instances proactively, accepting the cost of idle capacity to ensure rapid response to demand increases.

One mitigation is checkpoint pre-loading: caching model weights in CPU memory or on a local NVMe drive on the serving host, so that when the GPU process restarts, it can load weights from fast local storage rather than from network-attached storage. A local NVMe drive typically provides 5-7 GB/s read speed, compared to 1-2 GB/s for network storage, cutting disk read time by 3-5x for large models. When combined with GPU memory pre-allocation (reserving GPU memory before the model is fully loaded), a well-optimized cold start can bring a 70B model to ready state in under two minutes instead of five or more.

GPU costs dominate the economics of LLM serving. A single H100 GPU costs thousands of dollars per month on cloud platforms, and serving large models requires many such GPUs. The key metric is tokens served per dollar, which depends on both raw inference speed and GPU utilization. An underutilized GPU generating few tokens is expensive; a fully utilized GPU generating many tokens is efficient.

Achieving high utilization requires workload shaping, good batching, and sometimes mixed workloads: running batch inference jobs during off-peak hours on the same infrastructure that handles interactive traffic during peak hours. Token-based pricing, where users pay per thousand tokens rather than per request, aligns incentives with efficient utilization by pricing long requests proportionally to their actual resource consumption.

The cost calculation changes significantly with model size and optimization choices. A 7B model with int8 quantization running at 80% utilization can serve far more tokens per dollar than a 70B model in float16 running at 40% utilization, even if the 70B model produces better output quality. Understanding the quality-per-dollar trade-off, not just quality-per-output, is essential for building economically sustainable LLM services.

Spot or preemptible instances from cloud providers offer discounts of 50-80% compared to on-demand pricing at the cost of potential instance interruption. For stateless inference servers, spot instances work well because any in-flight requests at interruption time can simply be retried against other instances. Systems that route around interrupted instances automatically (with proper health checking and load balancing) can run large portions of their serving infrastructure on spot, dramatically reducing costs.

Production serving systems must handle partial failures gracefully. Individual GPUs can fail, model instances can crash, and network connectivity between components can be interrupted. A well-designed serving stack isolates these failures: a crashed instance should be removed from the load balancer pool and restarted automatically, while other instances continue serving traffic. Health check systems detect failures within seconds, and circuit breakers prevent the load balancer from sending requests to known-bad instances.

For tensor-parallel deployments where a single model spans multiple GPUs, a failure of any one GPU in the group takes down the entire instance. This is one reason teams prefer wider replication (more single-GPU or dual-GPU instances) over deeper tensor parallelism (fewer instances each using many GPUs) when reliability matters. A 4-way tensor-parallel deployment has a higher probability of GPU failure at any point in time than a 2-way deployment with twice as many replicas.

Deeper failures, such as model weight corruption or systematic errors introduced by a model update, require application-level quality monitoring rather than just infrastructure health checks. Tracking output quality metrics (answer relevance, refusal rate, format compliance) alongside infrastructure metrics allows you to detect model-level regressions that infrastructure health checks would miss. The monitoring and quality assurance capabilities covered in later chapters of this part are essential for catching these more subtle failure modes before they significantly impact users.

Effective model serving does not stand alone: it is most powerful when combined with the inference optimization techniques covered in Part XXXI. KV caching, quantization, speculative decoding, and continuous batching all operate at the level of individual requests. Model serving adds the orchestration layer above them, making sure that requests are routed efficiently, hardware is utilized fully, and the system remains stable under load. A serving system built without these underlying optimizations will achieve a fraction of the throughput possible with them.

Speculative decoding in particular interacts with serving in non-obvious ways. The draft model used for speculation adds its own memory requirement and compute overhead. For a single-request workload, speculative decoding can reduce latency by 2-3x. For a high-throughput batched workload, the benefit shrinks because batched decode is already efficient: the KV cache reads that make decode memory-bandwidth-bound are partially offset by the larger batch. Teams deploying speculative decoding in production must validate improvements in their specific workload metrics rather than assuming the laboratory results transfer directly.