Auto-scaling › Scale-Up vs. Scale-Out

Once you have decided that more capacity is needed, there are two fundamentally different ways to provide it: vertical scaling (scale-up) and horizontal scaling (scale-out). Understanding their tradeoffs is essential for designing a cost-effective and responsive LLM infrastructure.

Vertical scaling adds resources to an existing instance: a larger GPU, more VRAM, more CPU cores, more system RAM. For LLM inference, the most common form is moving from a smaller GPU (say, an A10G with 24 GB VRAM) to a larger one (an A100 with 80 GB VRAM). Alternatively, you might add a second GPU to a single node and run the model in tensor-parallel mode across both.

Vertical scaling is attractive when your bottleneck is model capacity rather than throughput. If a model does not fit in a single GPU's VRAM, scaling up is not optional. A 70B parameter model in FP16 requires approximately 140 GB of VRAM. A single A100 provides 80 GB, which is insufficient without quantization. To run the model on a single node, you need at least two A100s in tensor-parallel configuration, or one H100 with 80 GB (though even then quantization is typically required). This is a hard constraint that the scaling policy cannot work around: you must have enough VRAM per model replica before any scaling discussion is relevant.

Vertical scaling is also appealing for latency-sensitive workloads. A single A100 will generally produce tokens faster than a single A10G because its wider memory bandwidth lets the attention and feedforward computations execute more quickly. This matters for services where you care about individual request performance, not just aggregate throughput. Moving up from an A10G (600 GB/s memory bandwidth) to an H100 (3.35 TB/s) can reduce per-token generation time by a factor of five or more for bandwidth-bound workloads.

The drawbacks of vertical scaling are significant in an auto-scaling context. Most cloud providers do not support dynamic vertical scaling: you cannot add a GPU to a running instance without stopping and restarting it. This makes vertical scaling a pre-deployment decision rather than a runtime response to load changes. You choose the right instance type when you design the deployment, and you stick with it. Changing instance types in response to a live traffic spike is simply not feasible in most cloud environments.

Even when vertical scaling is technically possible (for example, by migrating workloads between instance types in a managed Kubernetes cluster), it is slow and disruptive. The economics are also often unfavorable: a single large instance is more expensive than two smaller instances with equivalent aggregate compute, partly due to hardware rarity and partly due to cloud provider pricing models that charge a premium for the largest instance types.

Horizontal scaling adds more instances running the same model. Each instance is a complete, independent copy of the serving process. A load balancer distributes incoming requests across all running instances.

Scale-out is the dominant strategy for LLM auto-scaling in production for several reasons. First, it is incremental: you can add one instance at a time, progressively increasing capacity without service interruption. Second, it is resilient: if one instance crashes or becomes unhealthy, the load balancer routes around it and the fleet continues serving. Third, it naturally supports geographic distribution: you can scale out in multiple regions independently, keeping each region's capacity proportional to its local traffic. Fourth, the economics are favorable: adding two A10G instances is typically cheaper than upgrading to one A100 for the same aggregate throughput.

The cost of scale-out is model startup time. Loading a large LLM from disk (or from object storage) into GPU memory is not instantaneous. A 70-billion parameter model in FP16 requires roughly 140 GB of data to transfer, which at 10 GB/s storage throughput takes around 14 seconds before the first request can be served. On cloud platforms that provision bare-metal GPU instances on demand, there is also VM startup time and container pull time on top of model loading. The combined cold start latency for a 70B parameter model can easily exceed three minutes in a worst-case cloud environment.

This startup latency is the central constraint that differentiates LLM auto-scaling from conventional web service auto-scaling. A conventional web application container starts in 2-5 seconds. An LLM serving instance starts in 30 seconds to 5 minutes. This difference changes everything about how policies must be designed, how far in advance proactive scaling must act, and how much idle capacity you must maintain as a buffer against sudden spikes.

Understanding this constraint also motivates a family of engineering optimizations aimed at reducing startup time. Container image caching ensures that the serving binary and library dependencies are already present on the host when a new instance is scheduled. Model weight caching on local NVMe storage reduces model loading from minutes (for a network download from object storage) to seconds (for a local disk read). Some systems maintain a pool of pre-initialized but idle serving processes, ready to accept traffic within seconds once they receive a routing signal.

The practical rule is to use scale-out as the primary scaling mechanism and treat scale-up as a pre-deployment decision. You choose the right instance type for your model's requirements when designing the deployment, then auto-scale by adding more of those instances in response to load. Vertical scaling during a live incident is rarely feasible given the disruption it causes.

This rule has an important corollary: instance type selection is a capacity planning decision, not a scaling decision. When you choose between running a 7B model on A10Gs or A100s, you are making a tradeoff between per-instance cost, per-instance throughput, and VRAM capacity. That decision should be made at deployment time based on your expected workload characteristics, not in response to live traffic.

The one exception is multi-GPU tensor parallelism: if a model requires 4 GPUs per instance (common for very large models), then your "instance" is a 4-GPU node, and horizontal scaling means adding more 4-GPU nodes. The principle remains the same. The unit of horizontal scaling is whatever constitutes a complete, self-contained serving replica, whether that is a single GPU, a multi-GPU node, or a multi-node cluster.

There is also a hybrid strategy that some large-scale deployments use: maintain a base fleet of smaller, cheaper instances for normal load, and use larger, more powerful instances as overflow capacity during peaks. A request router directs traffic to the appropriate tier based on request characteristics (for example, long-context requests might always go to the high-VRAM tier) or based on queue depth (overflow to the high-performance tier only when the base fleet is saturated). This requires more sophisticated routing logic but can be more cost-efficient than a homogeneous fleet.