Model Serving › Request Handling

Once a model is loaded, the serving system must accept requests from clients, schedule them for execution, and return results. This layer handles the coordination between user-facing APIs and the GPU-side computation.

Before discussing scheduling and batching, it helps to understand the two-phase structure of LLM inference, because virtually every serving optimization exploits the difference between these phases.

The prefill phase processes the entire prompt in parallel. All input tokens are fed through the model simultaneously, producing the initial KV cache and the logits for the first output token. Prefill is compute-bound: it saturates the GPU's matrix multiplication units because the batch of token representations is large. A 2048-token prompt produces a matrix multiply involving a $[2048, d\_model]$ activation tensor, which fully occupies tensor cores.

The decode phase generates output tokens one by one. At each step, the model processes only the single most recently generated token, using the stored KV cache to attend over the full context. Decode is memory-bandwidth-bound: the computational work per step is tiny (just one token's worth of matrix multiplies), but each step must read the entire KV cache from GPU memory. The GPU's compute units sit mostly idle while waiting for memory fetches.

This asymmetry has far-reaching consequences for serving design. Prefill and decode compete for the same GPU resources but have different bottlenecks. A long prefill request arriving while many decode steps are in progress will delay those decode steps, spiking their time-to-first-token. Serving architectures that ignore this asymmetry end up with poor interactive latency even when overall throughput looks fine.

Requests arrive asynchronously and at variable rates. When demand exceeds current processing capacity, requests must wait. How they wait, and how the system decides which request to process next, has significant effects on both latency and throughput.

First-come, first-served (FCFS) scheduling is the simplest policy: process requests in the order they arrive. It is fair and easy to implement, but ignores request characteristics that could improve overall efficiency. A single very long request can block many short requests that would complete quickly, wasting GPU capacity. FCFS is appropriate when fairness across all users has priority and all requests have roughly similar cost.

Shortest-job-first (SJF) scheduling prioritizes requests expected to complete quickly, minimizing average waiting time across all requests. The challenge is estimating job length without executing it. For language models, the prompt length provides a rough estimate of prefill time, but the decode length is typically unknown at scheduling time (the model generates until it decides to stop). Heuristics like maximum generation length or user-supplied length hints help, but length estimates are inherently uncertain. In pathological cases, SJF starves long requests indefinitely: if short requests keep arriving, a long request may never run.

Priority-based scheduling introduces service tiers where premium users or critical applications receive preferential queue position. A production system might define priority levels where API customers always receive faster service than background batch jobs running on the same infrastructure. Priority scheduling requires carefully calibrating the scheduling function to prevent low-priority work from perpetual starvation while still guaranteeing good service for high-priority requests.

Preemptive scheduling goes one step further: if a high-priority request arrives while a lower-priority request is being processed, the server can pause the lower-priority request (swapping its KV cache to CPU memory or discarding and recomputing it later) to immediately serve the high-priority one. This adds complexity but provides much stronger latency guarantees for critical traffic.

The three policies produce different orderings with distinct trade-offs. FCFS preserves arrival order and is maximally fair, but req-002 (a bulky request) holds up shorter jobs behind it. SJF reorders so small jobs run first, potentially improving average latency, but risks starving large requests indefinitely if small requests keep arriving. Priority-based ordering ensures high-priority users receive fast service regardless of request size, but requires careful calibration to prevent low-priority work from never executing.

In practice, most production systems use FCFS as their default policy because fairness and simplicity are easier to reason about than the subtle bugs that poorly tuned SJF or priority schedulers can introduce. Priority-based scheduling is layered on top of FCFS for systems with clearly differentiated service tiers, such as when a team needs to guarantee latency for interactive users while also running background batch jobs on the same infrastructure.

Processing multiple requests together in a single forward pass improves GPU utilization by amortizing the fixed overhead of each forward pass across many requests. This is essential for cost-efficient serving.

Static batching groups requests into fixed-size batches that are submitted together. All requests in a batch are processed in parallel during prefill, and all continue decoding together until every request in the batch completes. The problem is the "straggler" effect: if one request in a batch requires 500 tokens while others need only 50, the GPU sits partially idle during the extra 450 decode steps as the short requests have already finished. The partially complete batch cannot accept new requests to fill the vacated slots. Static batching is simple but wasteful.

Consider a concrete example. A batch of 8 requests is submitted together. Six requests finish after 60 tokens each. Two requests continue generating for 300 more tokens. During those 300 extra decode steps, the batch is operating at 25% capacity rather than 100%. The GPU is doing real work for 2 sequences but could be doing work for 8. The missed throughput is unrecoverable: those cycles are gone.

Dynamic batching relaxes the batch-size constraint and collects requests over a short time window before submitting them as a batch. This improves batch fullness compared to processing requests individually but introduces a small amount of additional latency as requests wait for the window to close. A 20-millisecond collection window, for instance, allows a burst of requests to accumulate into a larger batch, improving amortization. The window length creates a direct latency floor: the fastest a request can be processed is at least as long as the window.

Continuous batching (also called in-flight batching) is the most sophisticated approach. Rather than waiting for all requests in a batch to finish before accepting new ones, the server continuously adds new requests to the active batch as completed requests free up their slots. Each decode step potentially includes different requests, with arrivals inserting after their prefill completes and completions freeing space for waiting requests. As we covered in Part XXXI, continuous batching dramatically improves GPU utilization for workloads with variable-length outputs.

The key insight in continuous batching is that the GPU does not care whether the 8 sequences in the current decode step are the same 8 sequences that were in the previous step. As long as the batch is full of valid requests, the GPU does productive work. Continuous batching keeps the batch full by treating request boundaries as opportunities to swap in waiting requests rather than as synchronization points for the entire batch.

A more recent architectural pattern separates the prefill and decode phases of inference onto different hardware. This approach, sometimes called disaggregated serving or chunked prefill, recognizes that prefill and decode have different computational profiles and can benefit from different hardware or scheduling priorities.

Prefill is compute-bound: it processes all prompt tokens in parallel using dense matrix operations, fully utilizing GPU compute units. Decode is memory-bandwidth-bound: each step reads the entire KV cache from GPU memory to generate a single token. Mixing prefill and decode in the same batch means that long prefill phases for new requests delay the ongoing decoding for existing requests, spiking their latency.

In a fully disaggregated architecture, a pool of prefill instances handles incoming requests and computes their initial KV states, then transfers those KV tensors over a high-bandwidth network to a pool of decode instances that continue token generation. This separation allows each pool to be optimized for its task: prefill instances might use smaller GPUs with more compute throughput per dollar, while decode instances benefit from GPUs with high memory bandwidth. The trade-off is the KV cache transfer overhead: moving potentially gigabytes of KV state between machines requires high-bandwidth interconnects and adds a fixed delay between prefill completion and decode start.

Chunked prefill is a lighter-weight middle ground that keeps prefill and decode on the same GPU but limits how many prefill tokens are processed per step. Rather than completing an entire 2048-token prefill before resuming decode, the server processes 512 tokens of prefill per step, interleaving with ongoing decode steps. This prevents any single prefill request from monopolizing the GPU for an extended period, keeping time-to-first-token stable for concurrent decode requests.