Paged Attention › Key Parameters

Understanding the key parameters that control PagedAttention behavior helps you tune vLLM for your specific workload. The right settings depend heavily on your request distribution, target latency, and available hardware.

The key parameters for vLLM related to PagedAttention are:

• block_size: Number of tokens per physical block. Smaller blocks (8 or less) reduce internal fragmentation but increase block table size and lookup overhead. Larger blocks (32 or more) are more cache-friendly but waste more memory on partially filled final blocks. The default of 16 is a well-tuned starting point for most workloads.
• gpu_memory_utilization: Fraction of GPU memory reserved for KV cache. Higher values enable more concurrent sequences but leave less memory for activations and model weights. Setting this too high can cause out-of-memory errors during forward passes.
• max_num_seqs: Maximum concurrent sequences. PagedAttention enables much higher limits than traditional approaches because it eliminates fragmentation. In practice, this is often limited by the number of free blocks rather than any artificial cap.
• swap_space: CPU memory for block swapping when GPU memory is exhausted. vLLM can swap the KV cache blocks for lower-priority requests to CPU memory, letting the GPU to serve higher-priority requests. This adds latency for swapped requests but prevents starvation.

The interplay between these parameters is important. A large gpu_memory_utilization with a small block_size maximizes the number of sequences you can serve but increases the overhead per attention step. For latency-sensitive workloads, prefer slightly larger blocks and slightly lower utilization to reduce per-step overhead. For throughput-optimized batch serving, push utilization higher and accept more block table overhead.