Autoregressive Generation › Generation Speed Optimization
Language AI HandbookAll chapters Autoregressive Generation › Generation Speed OptimizationOne section · Sections in this chapter6 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Beyond KV caching, several techniques can accelerate generation. These optimizations are important for production deployments where latency directly impacts user experience. A user waiting for an AI assistant expects responses within seconds; a code completion tool must respond within milliseconds to avoid disrupting the programmer's flow. Meeting these latency requirements while serving thousands of concurrent users demands careful engineering at every layer of the stack.
The optimizations we explore in this section operate at different levels. Batch processing amortizes fixed costs by sharing them across multiple users. Mixed precision reduces the memory bandwidth required per operation, directly translating to faster computation on GPU hardware. Speculative decoding exploits the observation that a cheap model can correctly predict what an expensive model would have generated most of the time. Quantization goes further than half-precision, reducing weights to 8 bits or fewer. Continuous batching addresses the throughput problem of serving many users with different output lengths. These techniques are not mutually exclusive: production serving systems typically combine all of them.
The key mental model is to think about where time is being spent. For small batch sizes (a single user), generation is typically memory-bandwidth-bound: the bottleneck is loading model weights from GPU memory to the compute units, not the mathematical operations themselves. For large batch sizes, it becomes compute-bound: the GPU is fully utilized doing matrix multiplications. Mixed precision and quantization help in the memory-bandwidth-bound regime by shrinking the weights. Batching helps by moving into the compute-bound regime where the GPU is more fully utilized.
Batch Processing
Processing multiple prompts simultaneously amortizes the overhead of loading model weights. The attention computation is parallelized across the batch dimension:
def batch_generate(
model,
tokenizer,
prompts: list[str],
max_new_tokens: int = 30,
):
"""Generate completions for multiple prompts by processing each sequentially."""
results = []
for prompt in prompts:
input_ids = tokenizer.encode(prompt, return_tensors="pt")
generated_ids = input_ids.clone()
for _ in range(max_new_tokens):
with torch.no_grad():
outputs = model(generated_ids)
next_token = torch.argmax(
outputs.logits[:, -1, :], dim=-1, keepdim=True
)
generated_ids = torch.cat([generated_ids, next_token], dim=1)
if next_token.item() == tokenizer.eos_token_id:
break
results.append(
tokenizer.decode(generated_ids[0], skip_special_tokens=True)
)
return resultsBatch generation (4 prompts): 1.208s Sequential generation: 1.183s Speedup: 1.0×
Batch generation processes all four prompts faster than running them one at a time. The speedup comes from amortizing the model loading overhead and parallelizing the attention computations across the batch dimension. This makes batching needed for high-throughput serving scenarios.
The efficiency gain from batching is most pronounced when the batch is large enough to keep the GPU fully occupied. A modern GPU has thousands of CUDA cores designed to work in parallel; a batch size of 1 typically leaves most of them idle during the matrix multiplications that dominate model computation. Increasing batch size from 1 to 16 might give a 10-15 times throughput improvement with only a modest latency increase, because the per-token cost is dominated by loading weights from memory, which is amortized across the batch. Beyond some point, increasing batch size further provides diminishing returns as the GPU becomes compute-bound rather than memory-bandwidth-bound.
Mixed Precision
Using half-precision (float16) or brain floating point (bfloat16) reduces memory bandwidth and enables faster computation on modern GPUs:
# Load model in half precision
model_fp16 = GPT2LMHeadModel.from_pretrained(
"gpt2",
torch_dtype=torch.float16,
device_map="auto" if torch.cuda.is_available() else None,
)The memory savings from half precision are substantial:
Model size comparison: Float32: 474.7 MB Float16: 237.4 MB (theoretical) Memory reduction: 50%
Cutting the model precision in half directly halves the memory footprint. This allows larger batch sizes or longer sequences to fit in the same GPU memory. On modern GPUs with tensor cores, float16 operations are also faster than float32. This provides both memory and speed benefits.
Speculative Decoding
Speculative decoding uses a smaller "draft" model to propose multiple tokens, then verifies them with the larger model in a single forward pass. When predictions align, multiple tokens are accepted at once:
The effectiveness of speculative decoding depends on how well the draft model predicts the target model's outputs. When they agree frequently, significant speedups are possible, sometimes 2-4 times faster than standard generation. When they disagree often, the overhead of running two models provides little benefit, and it may be slower than just running the target model alone. The acceptance rate depends on the task: generating common phrases, code with predictable syntax, or formulaic responses tends to see high acceptance rates, while highly creative or unpredictable generation sees lower acceptance.
The main point behind why speculative decoding preserves output quality is that the rejection mechanism is mathematically calibrated to produce samples from the target model's distribution exactly, not approximately. When the draft token is rejected, the target model's distribution at that position is used to resample. This keeps the final output is statistically identical to what the target model would have produced without the draft. You get speed without sacrificing quality, though only when the draft model is accurate enough to provide a meaningful acceptance rate.
Quantization
Reducing model precision further with quantization (8-bit, 4-bit, or even lower) enables faster inference with reduced memory:
Quantization trade-offs: ------------------------------------------------------------ Precision Memory Speed Quality ------------------------------------------------------------ Float32 100% 1× Baseline Float16 50% ~1.5× Minimal loss Int8 25% ~2× Small loss Int4 12.5% ~3× Noticeable loss
The trade-off between precision and quality is application-dependent. For many tasks, float16 provides virtually identical results to float32. Int8 quantization typically works well for inference with minimal degradation. Int4 can cause noticeable quality loss but enables running much larger models on limited hardware.
The quality impact of quantization depends on the model architecture, the training procedure, and the specific task. Some models are trained with quantization awareness (QAT, quantization-aware training) and degrade gracefully to 4-bit; others that were trained purely in float32 may show more significant quality loss at low precision. In general, larger models are more tolerant of quantization because they have more "redundant" capacity that can absorb the noise introduced by rounding weights to fewer bits. A 70B parameter model quantized to 4-bit often outperforms a 7B parameter model in float16, making quantization a powerful tool for running capable models on consumer hardware.
Continuous Batching
For serving multiple users, continuous batching (also called in-flight batching) dynamically adds and removes sequences from a batch as they complete. This maximizes GPU utilization compared to static batching where all sequences must wait for the longest one. The difference is significant: with static batching, a batch containing one very long response and three short responses wastes three-quarters of its capacity while the long response is still being generated. With continuous batching, those three slots are freed up as soon as the short responses complete and immediately filled with new requests.
Think of continuous batching as analogous to a busy restaurant kitchen. A static batching kitchen would seat all tables only when all tables are empty, causing most tables to sit idle while a few slow diners linger. A continuous batching kitchen seats new parties as soon as any individual table clears, maximizing the number of customers served per hour. The analogy breaks down only in that the kitchen must serve all meals in a single GPU step: every sequence in the batch advances by exactly one token per iteration, so "clearing a table" and "seating a new party" both happen between iterations.
Autoregressive language models predate transformers by several decades. N-gram language models, used throughout the 1990s and 2000s, estimated from corpus co-occurrence counts, but could only condition on a short fixed window of prior context (typically 2-5 tokens) due to the combinatorial explosion of longer n-gram statistics. Recurrent neural networks (RNNs) introduced in the 2010s offered unbounded context in principle, but their compressed hidden state struggled to retain information across long distances. The 2017 "Attention Is All You Need" paper by Vaswani et al. showed that a transformer architecture using only attention mechanisms, with no recurrence, could outperform RNNs on sequence-to-sequence tasks. OpenAI's GPT (2018) applied this architecture to language modeling in the decoder-only configuration that became the dominant paradigm for large language models: train a transformer to predict the next token autoregressively, at massive scale, and emergent capabilities appear. The KV cache optimization, now so basic that it is built into every production serving framework, was implicit in the original attention formulation but became explicitly engineered as context lengths grew from 512 tokens (GPT-1) to 128K tokens and beyond in modern models.