Model Serving › Serving Configuration
Language AI HandbookAll chapters Model Serving › Serving ConfigurationOne section · Sections in this chapter5 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The performance of a serving system depends heavily on how it is configured. The key parameters control the trade-off between throughput, latency, memory usage, and resource cost.
Throughput vs. Latency Trade-offs
Serving configuration almost always involves trading throughput for latency or vice versa. Understanding which direction to optimize requires understanding your workload and user requirements.
Batch size is the most direct lever. Larger batches amortize GPU overhead across more requests, improving tokens-per-second throughput but increasing the time each individual request waits for other requests in the batch to complete. If your application cares about response time (interactive chatbots, real-time translation), smaller batch sizes with lower latency are preferable. If you are running overnight batch processing where per-request latency matters less than total throughput, larger batches maximize efficiency.
Queue depth controls how many requests accumulate before the system starts applying backpressure to callers. A deep queue absorbs bursty traffic by buffering requests, but at the cost of increased tail latency when the queue fills. A shallow queue forces callers to retry quickly, which works well when average load is below capacity but provides less smoothing for traffic spikes.
Max sequence length limits how long prompts and outputs can be. Shorter limits conserve KV cache memory and allow more concurrent requests, but reject or truncate long inputs. Setting this appropriately for your use case avoids wasting memory on pathologically long sequences while not frustrating users with legitimate long-context needs.
It is worth understanding why these three knobs interact. Increasing max sequence length increases KV cache consumption per request. This forces a reduction in max concurrent requests to stay within memory limits, which reduces effective batch size, which reduces throughput. Everything connects. A change to one parameter ripples through the others, which is why capacity planning requires thinking about the full system rather than individual knobs in isolation.
def estimate_throughput_latency(
batch_size: int,
tokens_per_step_ms: float = 2.0,
overhead_ms: float = 5.0,
avg_output_tokens: int = 200,
) -> dict:
"""
Simplified throughput and latency model for a serving configuration.
tokens_per_step_ms: Time to generate one token for a single sequence (ms)
overhead_ms: Fixed overhead per batch (scheduling, data movement)
"""
# Total decode time per batch (serialized token steps)
decode_time_ms = avg_output_tokens * tokens_per_step_ms
total_batch_time_ms = overhead_ms + decode_time_ms
# Average latency each request experiences (FCFS: waits for entire batch)
avg_latency_ms = total_batch_time_ms
# Throughput: requests completed per second
throughput_rps = (batch_size / total_batch_time_ms) * 1000
return {
"batch_size": batch_size,
"avg_latency_ms": avg_latency_ms,
"throughput_rps": throughput_rps,
}
batch_sizes = [1, 4, 8, 16, 32, 64]
configs = [estimate_throughput_latency(bs) for bs in batch_sizes]Batch Size | Avg Latency (ms) | Throughput (req/s)
-------------------------------------------------------
1 | 405 | 2.5
4 | 405 | 9.9
8 | 405 | 19.8
16 | 405 | 39.5
32 | 405 | 79.0
64 | 405 | 158.0The table isolates one part of the serving trade-off. Its simplified calculation holds per-batch service time at 405 ms, so throughput rises with batch size while service latency stays constant. It does not model the time spent waiting for enough requests to form a batch or waiting in a queue. Those omitted costs grow with batching pressure in a real deployment, which is why practical systems tune batch size against measured end-to-end latency rather than relying on service-time arithmetic alone.
Serving Parameters in Detail
The parameters exposed by modern LLM serving frameworks cluster into several categories, each controlling a different aspect of serving behavior.
Memory configuration parameters control how much GPU memory the server reserves and how it is divided between weights, activations, and KV cache:
- gpu_memory_utilization: The fraction of total GPU memory the server claims. Setting this to 0.90 reserves 90% for the model and KV cache, leaving 10% as headroom for PyTorch internal allocations and safety margin. Values above 0.95 risk out-of-memory errors under peak load.
- max_model_len: The maximum total sequence length (prompt plus generated tokens) the server will accept. Reducing this limit frees KV cache memory and allows more concurrent requests.
- max_num_seqs: The maximum number of requests the server will process concurrently. This caps KV cache usage and prevents memory exhaustion when many long requests arrive simultaneously.
Batching parameters control how requests are grouped for GPU execution:
- max_num_batched_tokens: The total number of tokens across all requests processed in a single forward pass. This controls the trade-off between batch efficiency and the latency impact of processing large prefill requests.
- enable_chunked_prefill: When enabled, long prefill phases are split into smaller chunks. Each chunk interleaves with ongoing decode steps for other requests, preventing long prefills from blocking decode latency.
Scheduling parameters determine how the server handles resource contention:
- scheduling_policy: Whether to process requests strictly in arrival order (FCFS) or by priority score.
- preemption_mode: When memory pressure requires freeing a request's KV cache, "recompute" discards and later regenerates it, while "swap" moves it to CPU memory. Recompute is simpler but wastes compute; swap preserves work but requires CPU memory bandwidth.
Understanding when to adjust these parameters requires thinking about your workload characteristics. For a customer service chatbot handling short conversations from many simultaneous users, you want high max_num_seqs and modest max_model_len. For a document summarization pipeline processing long articles one at a time, you want high max_model_len and accept lower concurrency. For a mixed workload, chunked prefill becomes essential to prevent long document requests from disrupting the latency of concurrent short conversational requests.
Latency Metrics and SLOs
Production serving systems track multiple latency metrics because different aspects of latency matter for different use cases.
Time-to-first-token (TTFT) measures how long a user waits before seeing any output. For streaming applications where tokens appear progressively in the UI, TTFT determines how long the screen sits blank after a user submits a request. High TTFT makes applications feel unresponsive even if tokens arrive quickly after the first one. TTFT is dominated by prefill time for long prompts and by queue wait time when the server is busy.
Time-per-output-token (TPOT) measures the average delay between consecutive tokens during streaming generation. This determines the speed at which text appears on screen. Users perceive fast token generation (low TPOT) as a smooth, fluid experience. TPOT is dominated by the KV cache read bandwidth during the decode phase.
End-to-end latency is the total time from request submission to final token received. This matters for non-streaming applications that wait for the complete response before displaying anything. It equals TTFT plus (output tokens TPOT).
Throughput is tokens generated per second across all concurrent requests. This is the primary efficiency metric for batch workloads where latency is secondary to cost.
Service level objectives (SLOs) define the acceptable range for each metric. A typical interactive application might require p50 TTFT under 300ms, p99 TTFT under 1000ms, and p50 TPOT under 50ms. Setting SLOs requires understanding user expectations: research suggests users begin to perceive applications as "slow" when TTFT exceeds 500ms, but smooth streaming at 20+ tokens per second feels fast even for long responses.
Tracking the full distribution of latency (p50, p90, p95, p99) rather than just averages is essential. The tail of the latency distribution represents your worst-performing requests, which are often the most memorable for users. A system with a good average latency but a long tail leaves a fraction of users with a terrible experience on every visit.
Rate Limiting and Backpressure
A production serving system must protect itself from being overwhelmed by more requests than it can handle. Without rate limiting, a traffic spike can fill the request queue, causing all in-flight requests to experience extreme latency while new requests continue to pile up, making the system slow for everyone.
Rate limiting enforces an upper bound on how many requests a client or tenant can submit per unit time. Token-bucket algorithms are commonly used: each client has a token bucket that refills at a fixed rate. Submitting a request costs one token. When the bucket is empty, further requests are rejected with a 429 (Too Many Requests) HTTP response, signaling to the caller that it should back off and retry later.
Backpressure propagates this signal upstream through the entire call chain. Instead of queueing indefinitely, a saturated server rejects new requests immediately rather than accepting them into a long queue where they would wait and eventually time out. This fail-fast behavior gives clients accurate information about server state and enables intelligent retry logic, such as exponential backoff.
import time
from dataclasses import dataclass
@dataclass
class TokenBucket:
"""
Token bucket rate limiter.
capacity: max burst size in tokens
refill_rate: tokens replenished per second
"""
capacity: float
refill_rate: float
_tokens: float = 0.0
_last_refill: float = 0.0
def __post_init__(self):
self._tokens = self.capacity
self._last_refill = time.time()
def _refill(self) -> None:
now = time.time()
elapsed = now - self._last_refill
self._tokens = min(
self.capacity, self._tokens + elapsed * self.refill_rate
)
self._last_refill = now
def try_consume(self, tokens: float = 1.0) -> bool:
"""Returns True if the request is allowed, False if rate limited."""
self._refill()
if self._tokens >= tokens:
self._tokens -= tokens
return True
return False
@property
def available_tokens(self) -> float:
self._refill()
return self._tokens# Simulate a burst of 20 requests against a bucket with capacity=10, rate=5/sec
bucket = TokenBucket(capacity=10.0, refill_rate=5.0)
results = []
for i in range(20):
allowed = bucket.try_consume()
results.append((i + 1, allowed, round(bucket.available_tokens, 2)))
allowed_count = sum(1 for _, allowed, _ in results if allowed)
rejected_count = sum(1 for _, allowed, _ in results if not allowed)Token Bucket Rate Limiting (capacity=10, rate=5/sec)
Burst of 20 simultaneous requests:
Request | Allowed | Tokens Left
-----------------------------------
1 | YES | 9.00
2 | YES | 8.00
3 | YES | 7.00
4 | YES | 6.00
5 | YES | 5.00
6 | YES | 4.00
7 | YES | 3.00
8 | YES | 2.00
9 | YES | 1.00
10 | YES | 0.00
11 | NO | 0.00
12 | NO | 0.00
13 | NO | 0.00
14 | NO | 0.00
15 | NO | 0.00
16 | NO | 0.00
17 | NO | 0.00
18 | NO | 0.00
19 | NO | 0.00
20 | NO | 0.00
Summary: 10 allowed, 10 rejectedThe first 10 requests drain the bucket immediately. Requests 11 through 20 arrive before the bucket can replenish enough tokens and are rejected. In a real system, rejected callers receive a 429 response and apply exponential backoff before retrying, allowing the bucket to refill. This prevents a single bursty client from monopolizing server capacity.
Rate limiting can operate at multiple granularities. Per-user limits prevent a single account from crowding out others. Per-API-key limits allow different products within an organization to have independent quotas. Token-based limits (counting input plus output tokens rather than requests) more accurately reflect actual resource consumption, because a 5000-token request consumes roughly 50 times as much compute as a 100-token request.
For multi-tenant deployments where different tenants have different SLAs, rate limiting alone is insufficient. You also need admission control: the ability to reject new requests when the server is at capacity, rather than accepting them into an ever-growing queue. A queue that grows without bound eventually causes every request to time out from the caller's perspective, even though the server is technically "accepting" the requests. Effective admission control keeps queue depth bounded by returning 503 (Service Unavailable) responses when the server is saturated, allowing callers to route requests to another instance or degrade gracefully.