Paged Attention › Working with vLLM in Practice
Michael BrenndoerferJanuary 8, 20261 min read
Language AI HandbookAll chapters Part 42 of 67Inference Optimization
Paged Attention › Working with vLLM in PracticeOne section · Sections in this chapter7 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's see how PagedAttention benefits manifest when using vLLM for inference. From a user perspective, vLLM exposes a simple API that feels like any other inference library. The PagedAttention machinery is entirely internal: you do not manage block tables or physical blocks directly. You simply submit requests, and vLLM handles all the memory management transparently.
In[2]:
Code# This code shows vLLM usage patterns (requires GPU and vllm installed)
from vllm import LLM, SamplingParams
# vLLM automatically uses PagedAttention
llm = LLM(
model="meta-llama/Llama-2-7b-hf",
# Block size can be configured (default is typically 16)
block_size=16,
# GPU memory utilization for KV cache
gpu_memory_utilization=0.9,
# Maximum number of sequences to batch together
max_num_seqs=256,
# CPU swap space in GB (for swapping out preempted requests)
swap_space=4,
)
# Multiple requests can be batched efficiently
prompts = [
"Explain quantum computing in simple terms:",
"Write a haiku about programming:",
"What are the benefits of exercise?",
# ... many more prompts
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=256)
# vLLM handles dynamic batching and memory management automatically
outputs = llm.generate(prompts, sampling_params)