Continuous Batching: Optimizing LLM Inference Throughput › Key Parameters

Tuning a continuous batching system requires understanding how several parameters interact. Higher throughput may increase latency or memory use, and reducing memory pressure can affect the other two measures. Each parameter affects multiple aspects of system behavior, so changes often have non-obvious second-order effects.

The key parameters for the continuous batching implementation are:

• max_batch_size: Maximum number of concurrent requests. Higher values improve throughput but require more memory for KV caches. The optimal value depends on model size, available GPU memory, and expected request characteristics. Setting this too high leads to out-of-memory errors; too low leaves GPU compute underutilized.
• block_size: Number of tokens per memory block in paged attention. Typically 16 or 32. Smaller blocks reduce internal fragmentation (wasted space within a block) but increase bookkeeping overhead. Larger blocks are simpler to manage but waste more memory for short sequences.
• chunk_size: Number of tokens processed per iteration during prefill. Helps balance latency and throughput. Smaller chunks give more responsive decode (less preemption of existing requests) but may reduce prefill efficiency due to increased per-iteration overhead.
• scheduler_policy: The ordering policy for the waiting queue (FIFO, SJF, priority). This affects fairness and average latency more than raw throughput.
• preemption_threshold: How full the memory pool must be before considering preemption. Setting this too low causes premature preemption; too high risks out-of-memory errors when a burst of long requests arrives.