Autoregressive Generation › Summary

Autoregressive generation is the basic process by which decoder-only transformers produce text. This chapter traced the mechanism from first principles, through its mathematical foundation in the chain rule of probability, through its computational implementation, and through the optimizations that make it practical at scale. This chapter covers:

• Generation loop: At each step, the model predicts a probability distribution over the vocabulary, samples or selects the next token, appends it to the sequence, and repeats until a stopping criterion is met.

• KV caching: By caching the key and value projections from previous positions, we avoid redundant computation during generation. This transforms the computational cost from quadratic to linear in sequence length, making practical generation possible.

• Stopping criteria: Generation can terminate via EOS tokens for natural endings, maximum length limits for bounded outputs, or custom stop sequences for structured outputs like chat turns.

• Speed optimizations: Batch processing and mixed precision reduce overhead. Speculative decoding plus quantization and continuous batching also make generation practical for production deployments.

• Memory trade-offs: KV caching trades memory for computation. For long sequences or large batch sizes, cache memory can become a significant constraint.

The generation procedure we've explored here forms the foundation for all the decoding strategies covered in subsequent chapters. Temperature scaling, top-k sampling, nucleus sampling, and repetition penalties all operate within this basic framework, modifying how we select the next token from the model's predicted distribution. The KV cache, the stopping criteria, and the optimization techniques are all orthogonal to these sampling choices: you can combine any decoding strategy with KV caching, any stopping criterion, and any precision level. This modularity is one of the strengths of the autoregressive framework: the core loop is simple, and each component can be optimized independently.

Two mental models are worth carrying forward. First, think of generation as a probability chain: every generated sequence is the product of a long chain of conditional probabilities, and the quality of the output depends on how well each link in the chain was estimated. Second, think of the KV cache as earned credit: every token the model has processed represents work that need never be repeated, and the cache is the mechanism by which that earned credit is stored and reused. Together, these two ideas capture both why autoregressive generation works, grounded probability theory applied at each step, and why it is efficient in practice, reuse of prior computation via caching.