Model Serving › Deployment Patterns

How you deploy a model serving system shapes its operational characteristics, including cost, reliability, and ease of management.

The simplest deployment runs one model instance on one or more GPUs, serving all traffic from that single process. This is appropriate for development, low-traffic applications, and models too large to replicate economically. It eliminates load balancing complexity and makes debugging straightforward.

The obvious limitation is the lack of redundancy: if the serving process crashes or the GPU experiences a hardware fault, all service is interrupted until the instance recovers. Single-instance deployments also cannot scale horizontally to handle traffic beyond what one instance can process, though vertical scaling (adding GPUs for tensor parallelism) extends this ceiling substantially.

For development and experimentation, single-instance deployment is nearly always the right choice. The configuration simplicity means you can quickly test different models, configurations, and optimization settings without managing distributed infrastructure. Problems are easy to diagnose because there is only one log stream and one process to inspect. Move to replicated deployment when your traffic or reliability requirements demand it, not before.

Running multiple identical instances of the model behind a load balancer provides both redundancy and horizontal scaling. Each instance handles a portion of the traffic, and a load balancer distributes requests across healthy instances. If one instance fails, its traffic routes to the remaining instances.

Stateless inference servers are well-suited to replication because there is no shared state between requests beyond the loaded model weights. Load balancers can use simple round-robin or least-connections policies because any instance can handle any request equally well. This is in contrast to stateful services where requests that belong to the same "session" must be routed to the same server.

The challenge with replicated LLM deployments is memory cost: each replica requires a full copy of the model weights in GPU memory. For large models, this makes wide replication expensive. Teams often balance the number of replicas against the tensor parallelism degree within each replica, choosing configurations that maximize served requests per dollar of GPU cost.

Consider a 70B model that requires 8 GPUs for tensor parallelism. Running 2 replicas costs 16 GPUs and provides twice the throughput with redundancy. Running a single replica with 8 GPUs costs half as much but has no redundancy. The right choice depends on your traffic volume and availability requirements. For high-availability production deployments, two replicas is a reasonable minimum: it provides both scaling headroom and tolerance for a single-instance failure.

Prefix caching across replicas adds a nuance to load balancing. If replica A has cached the KV state for a popular system prompt and replica B has not, routing a request with that system prompt to replica A avoids a prefill computation. Consistent hashing on the first N tokens of a request can direct requests with the same prefix to the same replica, preserving prefix cache hits. This is more complex than round-robin but can substantially reduce compute on workloads with common prefixes.

Zero-downtime deployments use deployment patterns borrowed from general software operations.

Blue-green deployment maintains two complete environments, "blue" (current production) and "green" (new version). When deploying a model update, the green environment is spun up with the new version, validated, and then traffic is atomically switched from blue to green. If issues emerge, traffic instantly rolls back to blue. The trade-off is cost: maintaining two full environments doubles GPU expenses during the deployment window. For large models requiring many GPUs, this doubling can be substantial, making blue-green deployment economically challenging except for brief deployment windows.

Canary deployment routes a small fraction (such as 5%) of traffic to the new version while the remainder continues on the old version. Quality metrics, latency, and error rates are monitored for the canary slice. If metrics look good over a defined evaluation period, the traffic fraction increases progressively until the new version handles all traffic. This provides early failure detection with gradual exposure, balancing safety and deployment speed.

The evaluation period between traffic shifts is critical. You need enough traffic through the canary to detect regressions with statistical confidence. A 5% canary receiving 1,000 requests in its first hour gives you reasonable confidence for common failure modes, but may not expose rare edge cases. Defining clear rollout criteria before deployment (for example, "advance when p99 latency is within 10% of baseline and error rate is below 0.1%") prevents the evaluation from being subject to subjective judgment under time pressure.

Static deployments with a fixed number of replicas waste capacity during off-peak hours and may be insufficient during peak hours. Autoscaling dynamically adjusts the number of running instances in response to traffic.

Horizontal autoscaling (adding or removing replicas) is the standard approach for web services. For LLM serving, autoscaling has one critical challenge: the cold start problem. Adding a new replica requires loading model weights, which takes minutes for large models. By the time the new replica is ready to serve traffic, the traffic spike that triggered the scale-out may already be over. This means autoscaling for LLMs must be proactive: anticipate traffic increases before they arrive (using historical patterns or predictive signals) and scale up in advance rather than in reaction.

Minimum replica counts ensure that at least one warm instance is always ready to serve traffic. A common pattern is maintaining two replicas as a minimum even during very low traffic, so that the system can always absorb a sudden burst with one instance while the autoscaler provisions additional capacity.

Vertical scaling within a single instance (changing the tensor parallelism degree) is not typically done dynamically because it requires restarting the serving process. Horizontal scaling is therefore the primary autoscaling mechanism.