Continuous Batching: Optimizing LLM Inference Throughput › Request Completion Handling
Language AI HandbookAll chapters Continuous Batching: Optimizing LLM Inference Throughput › Request Completion HandlingOne section · Sections in this chapter6 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Handling completed requests efficiently is important for continuous batching performance. The system must quickly detect completions, release resources, and fill the vacated slots. Any delay in this process reduces the benefits of continuous batching by leaving slots empty when work is available.
The completion handling pipeline runs at the boundary between every pair of iterations. It must execute fast enough to avoid becoming a bottleneck: if the completion check takes 5ms and the decode step takes 10ms, you have added 50% overhead. Production implementations use highly optimized routines, often scanning completion conditions directly on GPU tensors without transferring data to CPU.
Completion Detection
A request completes when it generates a special end-of-sequence token or reaches a maximum length limit. The completion check happens after each decode step and must be fast enough to avoid becoming a bottleneck. In practice, this means simple integer comparisons that execute in nanoseconds.
The end-of-sequence token is a special marker that the model has learned to generate when it believes the response is complete. Different models use different token IDs for this purpose. GPT-family models typically use token ID 50256, while LLaMA models use token ID 2. The maximum length limit is a safety valve, preventing runaway generation that would consume resources to long sequences. Setting this limit requires balancing protection against runaway costs with flexibility for legitimately long responses.
def check_completion(
generated_token: int,
tokens_generated: int,
max_length: int,
eos_token_id: int = 2,
) -> tuple[bool, str]:
"""
Check if a request has completed.
Returns:
(is_complete, reason)
"""
if generated_token == eos_token_id:
return True, "eos_token"
if tokens_generated >= max_length:
return True, "max_length"
return False, "ongoing"Resource Release
When a request completes, the system must release several resources promptly to make room for new work. Each resource type has its own release mechanism and timing considerations.
- KV cache memory. The cached key-value tensors for all layers must be freed. With paged attention, as discussed in the Paged Attention chapter, this means returning memory blocks to the free pool. The blocks become immediately available for allocation to new requests, maximizing memory utilization.
- Batch slot. The position in the batch tensor becomes available for a new request. This slot is one concurrent generation stream, and freeing it allows the waiting queue to advance.
- Request metadata. Tracking structures for the completed request can be cleaned up. This includes timing information, token histories, and any other per-request state maintained by the scheduler.
The order of these release operations matters. KV cache memory should be freed before the new request attempts to allocate it. The batch slot assignment should complete before the next iteration begins. Careful sequencing ensures that resource counters remain consistent and that we never accidentally double-allocate memory.
@dataclass
class SlotState:
"""Tracks the state of a batch slot."""
slot_id: int
request_id: int | None = None
is_free: bool = True
kv_blocks_allocated: int = 0
class SlotManager:
"""Manages batch slots for continuous batching."""
def __init__(self, num_slots: int, blocks_per_slot: int = 16):
self.slots = [SlotState(slot_id=i) for i in range(num_slots)]
self.blocks_per_slot = blocks_per_slot
self.total_blocks = num_slots * blocks_per_slot
self.free_blocks = self.total_blocks
def allocate_slot(self, request_id: int, num_blocks: int) -> int | None:
"""Allocate a slot for a new request."""
if num_blocks > self.free_blocks:
return None # Not enough memory
for slot in self.slots:
if slot.is_free:
slot.is_free = False
slot.request_id = request_id
slot.kv_blocks_allocated = num_blocks
self.free_blocks -= num_blocks
return slot.slot_id
return None # No free slots
def release_slot(self, slot_id: int) -> int:
"""Release a slot and return freed blocks."""
slot = self.slots[slot_id]
freed_blocks = slot.kv_blocks_allocated
slot.is_free = True
slot.request_id = None
slot.kv_blocks_allocated = 0
self.free_blocks += freed_blocks
return freed_blocks
def get_active_count(self) -> int:
return sum(1 for s in self.slots if not s.is_free)# Demonstrate slot management
manager = SlotManager(num_slots=4, blocks_per_slot=16)
# Define request requirements
req1_id, req1_blocks = 100, 8
req2_id, req2_blocks = 101, 12
req3_id, req3_blocks = 102, 6
# Allocate some slots
slot1 = manager.allocate_slot(request_id=req1_id, num_blocks=req1_blocks)
slot2 = manager.allocate_slot(request_id=req2_id, num_blocks=req2_blocks)
slot3 = manager.allocate_slot(request_id=req3_id, num_blocks=req3_blocks)Slot Management Demo: Initial free blocks: 64 Allocated slot 0 for request 100 (8 blocks) Allocated slot 1 for request 101 (12 blocks) Allocated slot 2 for request 102 (6 blocks) Free blocks remaining: 38 Active requests: 3
The manager successfully allocates blocks for three requests, decrementing the free pool accordingly. This tracking ensures we never overcommit the available KV cache memory, which would lead to out-of-memory errors during generation.
# Complete request 101 and free its resources
freed = manager.release_slot(slot2)Released slot 1 (request 101) Freed 12 blocks Free blocks now: 50 Active requests: 2
Releasing the slot immediately returns blocks to the free pool, making them available for waiting requests in the very next iteration. This rapid turnover is what lets continuous batching to maintain high utilization. Notice that the freed blocks become available synchronously, so the very next call to _fill_batch can allocate them to a new request without any delay.
Immediate Response Delivery
Unlike static batching, continuous batching can return completed responses immediately. Users don't wait for other requests in the batch, receiving results the moment generation finishes. This eliminates the latency inflation common in static batching and gives a more responsive experience.
The immediate delivery mechanism requires careful coordination between the generation loop and the response handling system. Completed results must be extracted and transmitted without blocking the main processing loop, often using asynchronous I/O or separate response threads. A naive implementation might pause the generation loop to send each response, but this would introduce jitter that accumulates across many requests. Production systems decouple response delivery from generation, queuing completed results for transmission by a separate thread or event loop.
@dataclass
class TimedRequest:
"""Request with timing information."""
id: int
output_length: int
submit_time: float = 0.0
complete_time: float = 0.0
@property
def latency(self) -> float:
return self.complete_time - self.submit_time
class LatencyTracker:
"""Tracks per-request latencies."""
def __init__(self):
self.requests: dict[int, TimedRequest] = {}
def submit(self, request: TimedRequest, current_time: float):
request.submit_time = current_time
self.requests[request.id] = request
def complete(self, request_id: int, current_time: float):
if request_id in self.requests:
self.requests[request_id].complete_time = current_time
def get_latencies(self) -> List[float]:
return [
r.latency for r in self.requests.values() if r.complete_time > 0
]