LLM Inference Serving: Architecture, Routing & Auto-Scaling › Worked Example: Sizing a Serving Deployment

To make the concepts concrete, let us walk through a realistic capacity planning exercise. Suppose you need to deploy a 7B parameter model serving a chatbot application with the following requirements: 100 requests per second at peak, a P95 TTFT target of 500ms, a P95 total latency target of 5 seconds, and typical prompts of 500 tokens generating 200 tokens of output.

Step 1: Estimate per-request compute requirements.

A 7B model in FP16 requires approximately 14GB for model weights. Each forward pass during decode reads all weights plus the KV cache. With a 500-token prompt and 200 tokens generated, the KV cache grows from 500 to 700 entries during the decode phase. On an H100 GPU with 3.35 TB/s memory bandwidth, each decode step reading 14GB of weights takes roughly $14 \times 10^9 / (3.35 \times 10^{12}) \approx 4.2$ ms per token. For 200 tokens, decode takes approximately 840ms.

Step 2: Estimate prefill time.

Prefill with 500 tokens on an H100 (989 TFLOPS FP16) involves roughly $500^2 \times 4096 \times 2 \approx 2$ billion floating point operations (attention) plus similar cost for FFN layers. The full prefill completes in roughly $14 \times 10^9 / (989 \times 10^{12}) \approx 14$ ms, dominated by memory bandwidth rather than compute at this prompt length.

Step 3: Determine single-instance throughput.

With continuous batching, a single H100 can serve approximately 30-50 concurrent decode sequences while maintaining reasonable token throughput. At 50 concurrent sequences each generating at 4.2ms per token, the single-instance throughput is approximately $50 / 4.2 \times 10^{-3} \approx 12{,}000$ tokens/second or roughly 60 requests completed per second (at 200 tokens each).

Step 4: Calculate required instances for throughput.

At 100 requests per second, you need approximately $100 / 60 \approx 1.7$ instances to handle the throughput. Round up to 2 active instances plus 1 warm spare (for failover and traffic spikes), giving 3 total instances.

Step 5: Verify latency targets.

With 2 active instances and 100 rps arriving, each instance sees 50 rps. Queue depth under steady load with 50 concurrent sequences per instance stays low. Total latency is prefill (14ms) plus decode (840ms) plus queue time (expected near zero at this load), totaling approximately 854ms. The 5-second P95 total latency target has comfortable headroom. TTFT is queue time (near zero) plus prefill (14ms) plus first decode step (4.2ms), totaling approximately 18ms, well within the 500ms target.

Step 6: Configure auto-scaling thresholds.

Set queue depth threshold at 20 requests (signaling sustained load above capacity), scale-up cooldown at 90 seconds (matching model load time on NVMe-attached storage), scale-down cooldown at 300 seconds, warm pool size at 1, and replica bounds between 2 and 8.

This analysis shows that for this workload, the limiting factor is not latency (there is substantial headroom) but throughput: the system can handle the load with 2 instances, but needs careful warm pool configuration to handle demand spikes without cold-start delays.