Paged Attention › Benefits and Performance Impact
Language AI HandbookAll chapters Paged Attention › Benefits and Performance ImpactOne section · Sections in this chapter10 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
PagedAttention provides several interconnected benefits that compound to deliver substantial throughput improvements. Understanding how these benefits interact helps you predict the gains you will see in your own deployment and identify which workload characteristics make PagedAttention most effective.
The most direct benefit is higher batch sizes, which translates to higher GPU utilization. Modern GPUs are designed to process large batches efficiently; the hardware has enough parallelism to handle hundreds of requests simultaneously. Traditional KV cache allocation prevented systems from filling the GPU because memory was being wasted on fragmentation. PagedAttention removes that constraint. When batch size doubles, throughput typically more than doubles, because larger batches improve the arithmetic intensity of the computation and allow the GPU's memory bandwidth to be used more efficiently.
A secondary but important benefit is reduced latency for queued requests. With traditional allocation, a server that is nominally at capacity may have most of its GPU memory sitting empty. New requests queue up not because the GPU is busy computing but because there is nowhere to put their KV caches. PagedAttention eliminates this artificial queuing, allowing the server to accept and begin processing more requests immediately.
Higher Batch Sizes
The most direct benefit is the ability to serve more concurrent sequences. With 5x better memory utilization, a serving system can batch 5x more requests simultaneously. Since transformer inference is typically compute-bound for large batches, higher batch sizes improve GPU utilization:
import numpy as np
def estimate_throughput_improvement(
traditional_batch_size: int = 8,
memory_improvement: float = 5.0,
compute_efficiency_curve: callable = None,
):
"""
Estimate throughput improvement from better memory utilization.
"""
if compute_efficiency_curve is None:
# Typical curve: efficiency improves with batch size up to a point
def compute_efficiency_curve(batch_size):
# Logarithmic improvement, saturating around batch 64
return min(1.0, 0.3 + 0.7 * np.log2(batch_size + 1) / np.log2(65))
paged_batch_size = int(traditional_batch_size * memory_improvement)
trad_efficiency = compute_efficiency_curve(traditional_batch_size)
paged_efficiency = compute_efficiency_curve(paged_batch_size)
# Throughput = batch_size x efficiency
trad_throughput = traditional_batch_size * trad_efficiency
paged_throughput = paged_batch_size * paged_efficiency
return {
"traditional": {
"batch_size": traditional_batch_size,
"compute_efficiency": trad_efficiency,
"relative_throughput": trad_throughput,
},
"paged": {
"batch_size": paged_batch_size,
"compute_efficiency": paged_efficiency,
"relative_throughput": paged_throughput,
},
"throughput_multiplier": paged_throughput / trad_throughput,
}
throughput_results = estimate_throughput_improvement()Throughput Improvement Estimate ============================================= Traditional allocation: Batch size: 8 GPU compute efficiency: 67% Relative throughput: 5.3 Paged allocation: Batch size: 40 GPU compute efficiency: 92% Relative throughput: 36.9 Throughput improvement: 6.9x
The throughput improvement demonstrates that memory efficiency directly translates to performance. By fitting more sequences into the GPU simultaneously, we operate in a regime of higher compute intensity, extracting more value from the hardware with each forward pass.
Dynamic Batching Support
Paged memory enables continuous batching (which we will explore in an upcoming chapter), where new requests can join an ongoing batch as old ones complete. With traditional contiguous allocation, adding a new request requires finding a large enough contiguous memory region. With paging, new requests simply acquire free blocks as needed, and those free blocks can come from anywhere in the pool, including from requests that just completed.
This property is what makes continuous batching viable at scale. In a continuous batching system, the scheduler maintains a running batch and swaps completed sequences out while adding newly arrived sequences in. Each swap requires only returning the old sequence's blocks to the free pool and allocating new blocks for the new sequence. Both operations are constant time with paged allocation, making continuous batching tractable even at very high request rates.
import numpy as np
def simulate_dynamic_batching(
arrival_rate: float = 2.0, # requests per time unit
service_rate: float = 1.5, # completions per time unit
duration: int = 100,
max_concurrent_traditional: int = 8,
max_concurrent_paged: int = 40,
):
"""
Simulate request handling with dynamic batching.
"""
results = {"traditional": [], "paged": []}
for strategy, max_concurrent in [
("traditional", max_concurrent_traditional),
("paged", max_concurrent_paged),
]:
active_requests = 0
queued = 0
total_processed = 0
queue_times = []
queue_depths = []
for t in range(duration):
# Arrivals (Poisson process)
arrivals = np.random.poisson(arrival_rate)
# Process completions
if active_requests > 0:
completions = min(
np.random.poisson(
service_rate * active_requests / max_concurrent
),
active_requests,
)
active_requests -= completions
total_processed += completions
# Handle new arrivals
for _ in range(arrivals):
if active_requests < max_concurrent:
active_requests += 1
else:
queued += 1
# Process queue
while queued > 0 and active_requests < max_concurrent:
queued -= 1
active_requests += 1
queue_times.append(t)
queue_depths.append(queued)
results[strategy] = {
"arrival_rate": arrival_rate,
"service_rate": service_rate,
"total_processed": total_processed,
"avg_queue_depth": np.mean(queue_depths) if queue_depths else 0,
"max_concurrent": max_concurrent,
}
return results
batching_results = simulate_dynamic_batching()Dynamic Batching Simulation ============================================= Request arrival rate: 2.0/time unit Service rate: 1.5/time unit Traditional: Max concurrent: 8 Total processed: 139 Paged: Max concurrent: 40 Total processed: 110
Paged allocation significantly outperforms the traditional approach in dynamic settings. While the traditional allocator quickly becomes constrained by its small batch capacity, the paged allocator continuously reuses freed blocks, maintaining high throughput across the entire simulation. The difference in total processed requests reflects the basic capacity advantage that paging provides in dynamic workloads.
Preemption and Priority Scheduling
With paged memory, preempting a low-priority request to serve a high-priority one becomes straightforward and efficient. The preempted request's blocks can be either swapped to CPU memory or simply marked as reusable if the request can be restarted from the prompt. When the request resumes, it reloads its KV cache from the saved state and continues generation from where it left off.
This preemption capability enables sophisticated scheduling policies that traditional contiguous allocation cannot support. A system with preemption can guarantee quality-of-service for high-priority users, implement latency-sensitive queuing policies like SRPT (shortest remaining processing time), and avoid head-of-line blocking where one long request prevents shorter requests from being served. These scheduling capabilities are not specific to PagedAttention, but PagedAttention makes them practical by so that preemption and resumption are efficient operations rather than expensive memory copies.